Re-defining Fabric Scale: Thinking Beyond the “Box”

October 18, 2011 - 25 Comments

Today we are making a significant announcement with several new innovations across our data center and switching portfolio that showcase how our customers can build large scale-up and scale-out data center networks.  While the press release does a great job (thanks Lee!) of highlighting all the innovations across the Nexus Unified Fabric portfolio and the new ASA 1000v, two aspects of the announcement stand out quite prominently:

  1. Cisco is delivering the highest density 10GbE modular switching platform in the industry
  2. Cisco is delivering the most scalable fabric in the industry and, by extension – on the planet! (we’re told planet sounds much cooler)

No. 1 above is fairly straightforward. With our new 2nd-generation F2 line card and Fabric 2 module, at 768 ports of 10GbE line-rate switching ports running NX-OS, the flagship Nexus 7018 in a fully-loaded configuration is simply the epitome of switch scale.

No.2 is where things get interesting, because we’re no longer thinking about just the “box” but rather, how we can weave different elements across the data center into a holistic “fabric”.  This systems-based approach focuses on multi-dimensional scale transcending the box and even the data center LAN, to span between data centers, while providing feature-rich fabric capabilities.  At 12,000+ 10GbE nodes supported as part of one Fabricpath-enabled system, and with the ability to support Fabric Extender (FEX) technology (plus L2 and L3 capabilities), this approach re-defines fabric scalability at 2X the scale and half the cost point of the next best claim in the industry. More important, it achieves this in an evolutionary manner for our 19,000+ NX-OS customers, offering investment protection for brownfield deployments while raising the bar for greenfield environments!

The Nexus platforms have been around for 3+ years, and over 500 customers have deployed FabricPath on the Nexus 7000 alone since its introduction about an year ago. It is a proven technology. With Fabricpath now coming onto the Nexus 5500 platforms, the momentum is likely to spike up with a mix of both size and scale. Like I said, things get interesting.

To make it more fun, our technical experts from the product teams have taken a data-driven approach and compared Cisco’s new innovations and our box and system-scale with others in the industry.

They looked at a couple of representative examples – the first being, what it would take any other vendor to build a non-blocking 768-port 10GbE “switch”, with capabilities similar to what the Nexus 7000 could provide in a single chassis. The second example takes a look at what it takes to build a “fabric” with Cisco leveraging its Nexus portfolio and NX-OS to build that.

Take a look and let us know what you think. It is useful to note that most vendors in the industry today have no fabric capabilities to speak of, and the few that are attempting a systems approach, have really limited to no customer traction thus far. Our customers and key analysts tell us that Cisco has a multi-year innovation lead in this space, even as Cisco continues to focus on bringing the network, compute, storage and application services together with integrated management to drive productivity and efficiency across traditional IT and organizational silos.

We often get asked – “So, who needs this kind of scale”? Today, we’re seeing a variety of customers asking for L2 or L3 scale across diverse deployment scenarios. Baidu is one such example. They chose the Nexus 7000 platform with the second generation capabilities for their global search business. Check out what Rackforce has to say. Where possible, we will continue to highlight several others that have chosen the Nexus portfolio and reaping the benefits of fabric scale and fabric extensibility.

Ultimately, customers desire architectural flexibility. The underlying infrastructure they invest in has to be adaptable enough to accommodate heterogeneous workloads and diverse business requirements – whether they’re focused on traditional enterprise, Web 2.0, Big Data, Cloud/Service Provider or specialized applications like high-frequency trading or high-performance computing. Upgrading or re-purposing the infrastructure every time they encounter a new workload or a new business requirement is not an option. It is also fair to state that most customers won’t overhaul their entire infrastructure at one go, but rather selectively choose which clusters they want to optimize in an evolutionary way.

Delivering architectural flexibility has been the hallmark of Cisco’s Unified Fabric approach, and the latest innovations are yet another example of giving our customers the power to choose.  Fortunately with this, we have been able to get both mind share as well as market share, and frankly, the results speak for themselves.

So, with sincere gratitude, a big THANK YOU to our customers, channel partners and our ecosystem partners for the tremendous success they have helped drive in the 3+ years since the Nexus portfolio was introduced.

This space will continue to be both disruptive and interesting, and we promise to keep you abreast of the happenings. There will be more blogs to follow on fabric extensibility, security and services. So, stay tuned and don’t forget to mark your calendars and register for our webinar “Evolutionary Fabric, Revolutionary Scale” on October 25th, 2011. You’ll hear from customers, partners, analysts and  some of Cisco’s top technology executives on how all this comes together. Register here now

As always, we welcome your questions and comments here.

Very best,

Shashi Kiran

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.


  1. It looks like the new F2 series modules support both L2 & L3… In a greenfield environment, where we are deploying a core of 7Ks with FabricPath for L2 connectivity (to Nexus 7k-5k-2k and VPC+ to Catalyst) and using L3 routing on the 7Ks to get between networks, do we even need an M-series card anymore? Am I missing something, if we do still need the M cards for L3 then what is the L3 support on the F2 buying us.

  2. This is a great post. I looking more into this.

  3. Good to see Cisco getting past the hype and all the recent negativity. With earnings around the corner – good luck!

  4. I am a Cisco channel partner and want to congratulate you on this announcement. I am glad you are sharing competitive readout here which was missing before and welcome this direction from Cisco. However this information is not readily available and we need better training to explain this to our customers.

  5. Are you planning to discontinue to the nexus 7010 since 7009 has similar port density and is smaller sized?

    • There are no plans whatsoever to discontinue the Nexus 7010. In fact if you notice we actually also announced the 2nd generation Fabric Module for Nexus 7010 as well.

  6. You’re right Suraj. The intent was not to compare against every switching platform out there.

  7. interesting to note you haven’t compared this with Nortel’s ERS 8600/8800 or the VSP 9000
    any specific reason why this comparison wasn’t made?

  8. With FabricPath, four core 7018’s could connect up to 256 Nexus 5548 ToR switches for 9216 10GE access ports with the same 3:1 oversubscription as QFabric.

    5548’s are dirt cheap, by Cisco standards ($14k or so). The end result would definitely be under $1000/port. And that’s before you add FEX!

    Very cool stuff, Cisco!

  9. One can use the F2 modules with existing F1 or M1 modules in the same chassis. Customers who want the capacity of the F2 modules can add F2 modules in their own VDC without having to replace any of their existing modules in the system. In a similar fashion one can use features such as OTV and LISP which are available on the M-Series modules in the same Nexus 7000 system as F2 Modules.

  10. This solution is fantastic, the only drawback is the buffer size per module, it seems little?

  11. Shashi,

    Very nice slideware. Delivering a full line rate, 48 port 10 Gig Ethernet card in the Nexus 7000 is a quantum leap forward. But would be interesting to let everyone know what is required to actually USE the 48 port card. Such as ripping out all of your F1 fabric cards and replacing them with F2 fabric cards.. Also, what about the M1 and F1 line cards.. Can they use the F2 fabrics or are they needing to be replaced also? Also, since this is a Data center Core switch, why is there only 1.5 Mbytes of Buffer on each port? Seems a bit small to support large scale data center traffic patterns..

    Now if only the 7018 would shrink and provide proper cooling flow …

    (Arista Networks)

    • Dean,

      Thanks for the recognizing our hardware innovations. However, the rest of your comments are not accurate. As Victor has mentioned in his comments above Fabric 1 and Fabric 2 are interoperable with each other and all I/O Modules.

      So in the example you give one could simply add F2 modules to a system which has 1st generation Fabric-1 modules and the F2 modules will be fully functional. If one needs full performance of F2 modules then an upgrade to Fabric-2 will be needed. Similarly, we see no issues supporting M1 or F1 modules with Fabric-2 modules.

      Cisco is committed to customer investment protection and the new products continue that tradition.

      • Nikhil (and Victor), the question that has not been answered, and it seems is being avoided deliberately is can the M1 modules communicate with the F2 modules, in the switch. The support for F2 with FAB1 is not in question – of course it only runs at 2:1 now.

        QUESTION: Is it supported to run the F2 and F1 or M1 modules together?

        Does a customer that made an investment in Cisco’s next-generation modular platform in 2008 now have to remove all the I/O modules that cost significant amounts of money in order to add just a single F2 module and take advantage of the “new” performance capability?

        Does this system really offer full investment protection, or is it just for the common equipment?

        With the release of the N7K-F248XP-25 module what are the plans to support LISP and OTV, innovations announced just last year – are they possible on the F2, or do you need a separate switch with M1 modules.

        How does the calculation of total power and RU look when you go to 1536 ports (a little bit more than 768). Cisco chose 768 ports as a sweet spot purely on the basis of that being the density of a switch. This does not consider customer requirements to support network redundancy and I fancy was selected purely on the basis of marketing wanting to show a pretty picture, not solve customers issues.

        (Arista Networks)

  12. What kind of backward compatibility can I expect with the introduction of Fabric-2 and the F2 cards? I would like to continue to use the stuff I’ve already invested in while tapping into this new gear.

    • Hi Ditchdigger,

      All previously existing linecards will work with Fabric-2 so you can continue to leverage your investment.
      Fabric-2 is a hot-swappable upgrade from Fabric-1. A system can work with a combination of Fabric-1 and Fabric-2 in order to support these hot upgrades.
      New linecards (F2s) will work with Fabric-1 if desired, of course the forwarding capacity will be that of Fabric-1 (230 Gpbs per slot).
      So as you can see, you have solid investment protection and you also have the ability to tap into industry leading *non-blocking* capacity and industry leading feature sets as delivered with F2 cards.

  13. Nice post Shashi. The very-low 10W per port rating is astounding to me – someone with experience in thermal management (electronics cooling specifically). Significant reduction in power consumption and hence lower operating costs, not to mention going much much greener and not having to use additional cooling equipment…Going green from a sustainability point of view is a lot easier with Cisco’s Nexus line.

    OTV is awesome too. Workload and VM mobility between Data Centers seems like mere key stroke function – Sleep well network engineers, the server team got this one!

    • Total bummer OTV was pulled from the IETF and abandoned as a possible standard. It’s a nice technology that will now stay proprietary.

      • Des,

        OTV looks current in this IETF link to Draft 03, are we missing something?

        Network Working Group H. Grover
        Internet-Draft D. Rao
        Intended status: Informational D. Farinacci
        Expires: January 9, 2012 V. Moreno
        Cisco Systems
        July 08, 2011

        Overlay Transport Virtualization

  14. Probably you should fire your technical experts since their not comparing with the proper technology in the industry, probably they’ve been getting too much fun.

    • Thank you for your facts-based comment. I cannot say they will be fired, but can say they’re certainly fired-up! 🙂