Cisco Blogs


Cisco Blog > Data Center and Cloud

The Fallacy of Wire-Rate Switching in the Data Center

Some days I think that I have heard the question, ‘is your switch wire-rate?’ so many times that if I only had a nickel… Wire rate switching simply means that if I have a gigabit port I can put a gigabit of traffic through it to some other port without dropping any packets. Multiply this by the number of ports in the device and you have a 32-port 10Gb wire-rate switch for instance. The majority of Cisco’s switching and routing portfolio is not wire-rate. Now this is interesting given that a large portion of the market acquires these products and should be indicative of the fact that many people really do not care so much about this term. Let’s instead focus on what I like to call the ‘Fallacy of Wire Rate Switching’. First let’s look at how wire-rate is achieved and what it means. To perform at wire-rate there are two important performance metrics- one is forwarding engine performance (measured in packets per second or packet lookups per second), the second is usually referred to as backplane capacity.Forwarding Engine performance- This on a Catalyst 6500 is the performance of the PFC/DFC or what we sometimes call the EARL. EARL7 (PFC3) performs IPv4 lookups at ~48-50mpps. There are some features that when implemented on a system reduce the forwarding engine capacity because those features require a packet to either go through the forwarding engine ASICs twice or more. These are features like IPv6, some MPLS operations, GRE, etc. Backplane Capacity- This is a simplified term that specifically would mean the amount of bandwidth delivered to each slot of a modular system via a backplane or midplane. However, this term usually implies the entire forwarding path from an ingress port, through a series of Port ASICs, Fabric ASICs, Switch fabric ASICs, and then back out the egress port. It is entirely possible to have a backplane that is capable of wire-rate or even overspeed but have a blocking scenario somewhere in the packet data path between the ingress physical port and the backplane. So when someone claims,”Full Wire Rate Switching at all frame sizes’ they are implying that for each port there is a non-blocking data path through the switch port to port and that the forwarding engine has the capacity to do packet lookups, ACLs, QoS remarking, etc without slowing down the packet forwarding rates. So let’s talk about reality for a second. 1) Every vendor remotely existing in switching designs products around four metrics- Bigger, Faster, Denser, Cheaper. 2) Most vendors stretch a bit when they claim wire-rate. They tend to singularly optimize on three frame sizes to the exclusion of realistic traffic patterns. 64-byte frames are always run so they can claim a high packet per second number; 1514-byte frames are run as the non-Jumbo Ethernet Frame size max, and 9k Jumbo’s are run if the product supports Jumbo Frames (most do now). Try a 65-byte frame for fun and we have found that many products put more packets in the proverbial bit bucket than out the egress port… For a ‘realistic’ frame sizing use IMIX. It’s around a 300-byte packet size average. Or for even more fun and games intersperse 90% Jumbo Frames with 10% 64 byte frames -- on a non-arbitrated fabric this will cause massive issues almost all the time… (it is also a semi-realistic traffic pattern to see large frames and acknowledgements concurrently) 3) To build a wire-rate switch the vendor must make some sacrifices. This all comes down to physics and trade-offs. If you want to build a wire-rate switch you will make a few decisions to trim down things. Commonly what we hear is the design teams saying-”We built a wire-rate N Gigabit backplane and when you build a WIRE RATE SWITCH YOU DON’T NEED BUFFERS’. This is critical! It is arguably one of the biggest mistakes vendors make. Let’s analyze this for a second before getting off the tangent ;) VendorX makes a 200-port wire rate switch. To build this they trim the off board memory systems down a good clip and move the memory on-board to save ASIC pin count and allow ‘wider’ lookups. This improves performance and allows them to achieve wire-rate forwarding for most frame sizes. We can safely assume that a vendor can build a wire-rate switch. But have you ever seen a wire-rate network? A network with true 1:1 any to any bandwidth with no oversubscription. This means a network with no EtherChannel (which can cause hash collisions, thus link contention) no Equal Cost Multipath (same as EtherChannel), and a design that never oversubscribes the uplink. Realistically this farcical network would have to have no Spanning Tree somehow since that would block any looped paths and reduce the effective bandwidth. Even if someone can build a ‘wire rate network’ (I hope the previous paragraph dissuaded you of the notion) have you ever seen an application optimized perfectly for a wire-rate any to any network? Pretty much every app I can think of has some fan-out. Someone hits a web server which talks to a first-tier app server which fans out to 8 other app servers which poll N number of data bases, reverse the path and deliver the application/content/page to the client. Significant fan-out. Any time we look we can realistically see that SOMEWHERE in the network there is the rational possibility of oversubscription and we need buffers to handle this. When vendors build their so-called wire-rate switches they sacrifice the buffers on the altar of wire-rate! Why do buffers matter? Quite simple actually- if 2 ports are talking for the briefest instant to one port at 10Gb/s each buffers allow the packets to interleave long enough so that both ports get served without sending a packet spiraling to a lonely death. (although with small buffers that death might not be so lonely and might be more lemming-esque) If the network has a propensity to fan out, use EtherChannel, have uplinks that are smaller than the aggregate of downlinks, use Routing with Equal Cost Multipathing, or has pretty much any real world application on it you will have a fan-out and thus contention for a port from 2 or more other interfaces. These small-buffered wire-rate switches perform abysmally in these types of real-world scenarios.How should switches be designed for real-world applications and data centers? First off stop being dogmatic about wire-rate. Performance is important but based on the above separate performance from wire-rate. Performance is about effective throughput under real-world conditions. Adequate and effective buffers designed to absorb contention and congestion and effectively deliver real-world applications is of paramount importance. Intelligent dropping of the right traffic when congestion occurs is also equally important to make most effective use of the buffers that do exist. Until better L2 congestion management systems come into widespread deployment buffers are the best way to deal with this. The big four metrics of Bigger, Faster, Denser, Cheaper is a limited value proposition that is not sustainable. Let’s face it- the Biggest, Fastest, Densest, Cheapest switch on the market changes about once every 9-12 months. Data center switching systems focus more on features that could carry a taxonomy such as “More Stable, Highly Available, Faster Converging, Operationally Manageable, Feature Rich, Segmented, Secure, Integrated, Systems Level Tested, etc.” Thanks,dg

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.

3 Comments.


  1. Very nice post! It is easy to read, understandable, and answers alot of questions. I have a hard time trying to explain”” to non-technical people why Other Vendors “”get away”” with saying certain things, and why it is mis-leading.Thanks, and keep them coming!”

       0 likes

  2. nice one , where can we get more information regarding non-blocking / blocking hardware architectures on modularized switches? thanks for the good info.esteban

       0 likes

  3. Omar Sultan

    I have been building networks for more than 20 years and I have always thought of performance as a bit of a red herring both on the manufacturer side and the customer side.I think many times manufacturers turn to raw performance numbers when they have nothing better to talk about. Its akin to 0-60 times for a car. Fast times are great…if you are a NASCAR driver. For most of us, buying a car involves balancing other factors like handling, braking, and even mundane factors such as cargo capacity and if there is a place to plug in an iPod. Most people end up buying a vehicle by balancing all these factors based on their intended use (i.e. hauling lumber vs. hauling toddlers). The same goes for switch design–effective design balances features based the intended purpose of the switch.On the customer side, I have had many customers who throw bandwidth at a problem, often because they lack the time or expertise to engage in true root cause analysis. Sometimes, it is a philosophical position, usually some version of if I have enough bandwidth I don’t need QoS”” or “”the network should be big dumb pipes and I will keep my intelligence at the edge””. While there is some validity to these viewpoints, I think, in the long term, none are sustainable as a singular design philosophy. In the end, I don’t think there is any substitute for proper network design and traffic engineering.”

       1 like