Since Cisco first introduced the concept of oversubscribed Fibre Channel modules in the storage networking industry with the MDS 9000 Family, there has been quite a bit of confusion between oversubscribed and blocking architectures, often incorrectly using the two terms interchangeably. I’m going to use a simple analogy with the highway (where I spend too much time on, because of my commuting) to try to explain the not-so-obvious difference.Let’s Start with OversubscriptionOversubscription is when, during peak times, there are too many cars compared to the available highway lanes and their capacity, hence congestion happens. Nonetheless highways are built “oversubscribed” by design: outside peak times (90% of the day?), they are underutilized and traffic flows with no issues. In theory, civil engineers could build very high-capacity roads to sustain peak traffic, but that would come at a much higher cost for taxpayers and wouldn’t be justified by the actual utilization. Instead, techniques to reduce congestion during peak hours are implemented: traffic metering lights are utilized to slow down the traffic that is trying to enter the crowded highway; lanes with policies (i.e. carpool) are assigned to traffic with specific characteristics.The same happens in networking switches. “Peak times”, i.e. burst of traffic, can also occur in Storage Area Networks and the oversubscribed Fibre Channel modules act like an highway with fewer lanes than are required at peak times. Fibre Channel buffer-to-buffer credits are the traffic lights that regulate incoming packets. Director switches like in the MDS 9500 Series have advanced technologies (e.g. Port Bandwidth Reservation) that allow dedication of a specific amount of bandwidth to certain “lanes,” i.e. ports. Traffic on those ports (carpool lanes) is guaranteed to move at higher dedicated speed, independently of the traffic on other lanes. On top of that, unique Quality of Service (QoS) capabilities allow MDS 9000 customers to give higher priority to specific flows of traffic (up to 4 levels).Oversubscription is inherent to any network topology; ranges are typically 12:1 to 30:1, host-to-storage via oversubscribed InterSwitch Links (ISLs). Cisco simply decided to move that proven concept directly into the modules dedicated to the access layer, which helps consolidate server connectivity and reduce costs. And obviously, Cisco MDS 9000 Family offers full rate, non-oversubscribed module options for the highest demanding applications such as ISL, storage, and high-performance servers.The Blocking SituationLet’s go back to the highway analogy to describe the blocking situation. When I drive back to San Francisco in the evening on Highway 101, the exit I take to get into the north part of the city is typically not congested. Nonetheless, I’m stuck in traffic. Why? Highway 101 splits into my exit and into the exit to the Bay Bridge, which is always very congested (Figure 1). The traffic to the Bay Bridge is backed up even before my exit, so I get stuck in that traffic en route to my own free exit. This is an example of a blocking situation, specifically Head-Of-Line (HOL) blocking: The head of the traffic going to the Bay Bridge is blocking my free exit to San Francisco. It is clearly a bad behavior: why I wait in traffic if my exit is actually not congested?

Figure 1: Congestion on Highway 101

Exactly the same thing happens in (competitive) switches, where even a few congested ports can impact the overall performance of the switch: Exit port 4 (Bay Bridge) is congested and all the traffic to other free ports (North San Francisco) are blocked (see “Switch Without VOQ” on the left, in Figure 2). As a result:- bandwidth per slot dramatically drops, sometimes even to a quarter of the claimed marketing numbers- applications experience unpredictable and inconsistent switch performance -- latency is extremely variableSwitches like the Cisco MDS 9000 Family solve the HOL blocking situation by relying on Virtual Output Queuing (VOQ) technology: a virtual output queue for each output port in the switch is assigned to each input port. In the “VOQ Model” on the right in Figure 2, even if the port 1 wants to transmit to both port 4 (congested) and port 6 (free), the two outputs are separated and traffic can flow to the free port 6. (In the highway example it would be like having the exit to San Francisco separated from the Bay Bridge one.) Performance is therefore as-advertised and predictable, independent of traffic conditions; latency is also constant.

Figure 2: How Virtual Output Queuing solves Head Of Line blocking

To summarize, the key take-aways are:- Oversubscription is good for your network (and your budget). Oversubscription enables to optimize your costs by designing your access to the network without over-engineering it. Oversubscription along with advanced traffic-management capabilities offered by the MDS 9000 Family allows for maximum flexibility and predictable behavior even with oversubscription. In fact, oversubscription and blocking are two concepts that are different and somehow orthogonal: a switch can be blocking even if all the installed modules are not oversubscribed!- A blocking situation is the worst that could happen to your switch. A blocking architecture is not able to deliver the full amount of bandwidth even if individual switching modules are not oversubscribed or not all the ports are transmitting simultaneously, technically not using all the claimed bandwidth. The MDS 9500 Series has been rated by 3rd party test as the only Fibre Channel switch industry to deliver non-blocking 8-Gbps performance.By the way: I solve my HOL blocking situation on highway 101, by taking the previous highway 280 exit to the city. Consider it my personal implementation of VOQ.