Cisco Blogs

Architecture of a Platform

April 3, 2009 - 3 Comments

In the last post I talked about the importance of designing a platform vs. a box and the benefits associated to that choice. Now I’d like to spend some time going into more of the architectural details that make the MDS box a platform. I’m going to try to keep the discussion as high level as possible, but at the end we are still a technology company, so I hope at least some of you will appreciate this “engineering” discussion.The architecture of an MDS Director can be simplified by listing the main three components involved in the packet switching operations: – the port module (or linecard): the element delivering the forwarding functionalities for each packet received in an ingress port and transmitted to an egress port – the crossbar (either on the supervisor or on the fabric module): the ‘N x N’ matrix providing connectivity between each port in each module – the arbiter (on the supervisor module): the scheduler deciding which port should be using the crossbar to transmit trafficBased on these building blocks, the MDS Director switches deliver a centrally arbitrated, crossbar architecture, with frame forwarding logic distributed in each port module. A packet arrives in an ingress port, a forwarding decision is made by the port module, the arbiter schedules the packet transmission based on congestion, the crossbar switches the packet to the egress port.By having switching (crossbar) and forwarding (port module) functionalities decoupled, the customer can seamlessly upgrade to the next interface speed by simply upgrading the crossbar. The investment in existing common equipment (chassis, supervisors, power supplies, etc) and port (or service) modules is preserved as new higher speed modules can now coexist in the same chassis. Capital expenses are significantly reduced by fully utilizing the open slots in existing chassis; operational costs associated to “rip-and-replace” approaches are completely eliminated. Decoupling functionalities is not the only factor required to achieve such investment protection and flexibility: the protocol and interface speed agnostic nature of the crossbar is the other key element. Such a crossbar can itself switch any packet independently of interfaces’ speed (2-, 4-, 8-, 10-Gbps) and the protocol utilized for the transmission. In fact, the exact same crossbar used in the MDS 9000 Family to carry Fibre Channel traffic is used also in other Cisco Ethernet products! The protocol agnostic aspect of the crossbar is what for example will allow existing MDS to seamlessly switch 10-Gbps Fibre Channel Over Ethernet (FCoE) packets to deliver upon the Unified Fabric vision, as previously discussed.But is this the only way to design a switch? In the storage networking industry, alternative examples inspired by the 3-stage Clos network are actually present. In this type of approach, each element is a device combining both switching and forwarding functionalities, simply meshed with the others via internal Inter Switch Links (ISLs), like it was a network of small switches. Certainly this design has advantages from a development perspective: you simply need to design that one device and you can re-use it across the 3-stages of the high-end Director, as well as in the low-end Fabric Switch version. Time to market is also reduced. But what about the customer? The customer is forced to completely rip and replace the infrastructure every time a new speed is introduced. And what about supporting a new protocol, at new higher speed, considering that that device is not protocol (Fibre Channel based forwarding) nor speed (8-Gbps) agnostic? Will yet another forklift upgrade be required or something weird will have to happen behind the scenes to use that Fibre Channel forwarding or the lower 8-Gbps speed? And in this mesh of devices, where is the central scheduler ensuring the traffic is flowing smoothly by solving congestion situations?

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.


  1. I need to contact Sugan Subramanian who wrote TN-18 at IDT and knows some stuff that I need and can't find out about from anyone else.Can you help me??

  2. Paulo,Nice summary. I would like to add one more salient feature of mds platform. By keeping the spine cards in a separate module, we can increase bandwidth required by a specific line card from spine without affecting the bandwidth requirement of other line cards. For example a spine card can support 50G bandwidth or 20G bandwidth per link. Linecard-A might require 20G bandwidth per link. Linecard-B and Linecard-C might require 50G bandwidth per link. Traffic destined to Linecard-A can only use spine-link at 20G. It would be arbitrated accordingly by arbiter. Traffic between Linecard-B and Linecard-C can use the spine-link at 50G. These two traffic running at different spine-link speed can be possible only if the spine cards are independent of linecard and supervisor. mds platform has spine in a separate slot. This feature in my opinion would be good marketing pitch in favor of mds and nexus platform.Thanks,-Sugan

  3. Thank you Paulo for mentioning that the crossbar fabric architecture is used as the foundation for Cisco's FCoE offerings. I think to many times people focus on the term Fibre Channel OVER Ethernet, and think that it is storage traffic across a normal lossy ethernet switch. When in almost the inverse is true, where a lossless low latency crossbar fabric (SAN derived) has been designed to allow Ethernet and Fibre Channel traffic to use the same links, while retaining the robustness that is required for storage transport.