Switch Architectures and Highways
Since Cisco first introduced the concept of oversubscribed Fibre Channel modules in the storage networking industry with the MDS 9000 Family, there has been quite a bit of confusion between oversubscribed and blocking architectures, often incorrectly using the two terms interchangeably. I’m going to use a simple analogy with the highway (where I spend too much time on, because of my commuting) to try to explain the not-so-obvious difference.Let’s Start with OversubscriptionOversubscription is when, during peak times, there are too many cars compared to the available highway lanes and their capacity, hence congestion happens. Nonetheless highways are built “oversubscribed” by design: outside peak times (90% of the day?), they are underutilized and traffic flows with no issues. In theory, civil engineers could build very high-capacity roads to sustain peak traffic, but that would come at a much higher cost for taxpayers and wouldn’t be justified by the actual utilization. Instead, techniques to reduce congestion during peak hours are implemented: traffic metering lights are utilized to slow down the traffic that is trying to enter the crowded highway; lanes with policies (i.e. carpool) are assigned to traffic with specific characteristics.The same happens in networking switches. “Peak times”, i.e. burst of traffic, can also occur in Storage Area Networks and the oversubscribed Fibre Channel modules act like an highway with fewer lanes than are required at peak times. Fibre Channel buffer-to-buffer credits are the traffic lights that regulate incoming packets. Director switches like in the MDS 9500 Series have advanced technologies (e.g. Port Bandwidth Reservation) that allow dedication of a specific amount of bandwidth to certain “lanes,” i.e. ports. Traffic on those ports (carpool lanes) is guaranteed to move at higher dedicated speed, independently of the traffic on other lanes. On top of that, unique Quality of Service (QoS) capabilities allow MDS 9000 customers to give higher priority to specific flows of traffic (up to 4 levels).Oversubscription is inherent to any network topology; ranges are typically 12:1 to 30:1, host-to-storage via oversubscribed InterSwitch Links (ISLs). Cisco simply decided to move that proven concept directly into the modules dedicated to the access layer, which helps consolidate server connectivity and reduce costs. And obviously, Cisco MDS 9000 Family offers full rate, non-oversubscribed module options for the highest demanding applications such as ISL, storage, and high-performance servers.The Blocking SituationLet’s go back to the highway analogy to describe the blocking situation. When I drive back to San Francisco in the evening on Highway 101, the exit I take to get into the north part of the city is typically not congested. Nonetheless, I’m stuck in traffic. Why? Highway 101 splits into my exit and into the exit to the Bay Bridge, which is always very congested (Figure 1). The traffic to the Bay Bridge is backed up even before my exit, so I get stuck in that traffic en route to my own free exit. This is an example of a blocking situation, specifically Head-Of-Line (HOL) blocking: The head of the traffic going to the Bay Bridge is blocking my free exit to San Francisco. It is clearly a bad behavior: why I wait in traffic if my exit is actually not congested?
Figure 1: Congestion on Highway 101
Figure 2: How Virtual Output Queuing solves Head Of Line blocking