Cisco Blogs


Cisco Blog > Data Center and Cloud

What is a FEX?

January 28, 2009
at 12:00 pm PST

So what is a Fabric Extender? In short, its going to change the way to network your server racks. The best way to understand the Nexus 2000 Fabric Extender, or FEX, is to view it as a remote linecard located at the top of the server rack. Why would you want to do this, you may ask?imageIt turns out, there are quite a few reasons. The two basic approaches to server networking are top-of-rack and end-of-row. Top-of-rack or ToR (where the switch is located, you know, at the top of the rack) benefits from a simplified, flexible cabling scheme--short intra-rack cable runs with a small number of uplinks to the agg switches. The downside of this approach is switch sprawl: 12 racks of servers usually means at least 12 switches. All those switches drive up the cost and complexity of the access layer of the network. The second approach is end-of-row switching or EoR (guess where the switch is), has every server cabled back to a single switch dedicated to a row of server racks. The upside of this approach is its a simplified management environment (one switch to manage) and efficient, but the cabling costs are significant and it is difficult and change out cabling. Its important to point out that both approaches are valid designs and the ultimate choice is driven by broader architectural requirements such as traffic patterns, L2/L3 boundary and the like.With the FEX, you get the best of both worlds. You get the low cost and cabling flexibility of a ToR design with the management simplicity of an EoR design. imageI think the management angle is where the true brilliance lies--the FEX will save you time, money and effort. As I mentioned earlier, each FEX acts like a remote line card, so you don’t have to individually manage each FEX--everything is inherited from the upstream switch. Basically you could have an entire rack of servers pre-cabled with a FEX delivered to your data center, plug it in, and that would be about the end of it. The FEX would automatically inherit the config of the upstream switch, so whatever policy and features you have configured will automatically propagate. If you need to upgrade software or deploy a new feature, simply make the change on the upstream switch, and it will automatically propagate across a dozen racks of servers.About that upstream switch--the FEX is currently supported on the Nexus 5000 with support for the Nexus 7000 and the Catalyst 6500 on the way. Currently, the Nexus 5020 can support up to 12 FEXes.For those of you who have been following Nexus developments, you have probably noticed the Nexus 5000 + FEX approach gives us the same virtual modular switch architecture as our Nexus 1000V, so if you were to Telnet into a Nexus 5000 and did a show interface command, you would see all the interfaces local to the Nexus followed by the interfaces on each connected FEX. Each FEX port is individually addressable and manageable.A couple of things to bear in mind. Because the FEX inherits the features of the upstream switch, it can support advanced features like FCoE and VN-Link, although, in the case of FCoE, the uplink is over-subscribed, so you need to pay a bit of attention to your traffic engineering. Also, while the FEX is designed to be broadly applicable, there are scenarios where it may not be the best choice--the best example of this is if your L2/L3 boundary is at the rack level.The initial iteration of the FEX is the Nexus 2148T which give you 48 copper GbE ports plus 4 SFP+ 10GbE uplinks--ideal to support your GbE attached servers while still taking advantage of Nexus and NX-OS features and easing your migration to 10GbE.For more info the the Nexus 2000 family, follow this link.

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.

14 Comments.


  1. FCOE on the FEX? I have seen the 4th queue dedicated for FCOE traffic, but haven’t heard anyone mentioning supporting FCOE out the 1 Gig ports. Does your statement mean this has changed and we can use FEX/FCOE as replacement to ISCSI over 1 Gig?

       0 likes

  2. Omar, does the 2148 support FCoE or are you saying that the Fabric Extender Architecture is capable of supporting FCoE, although presumably that would require 10Gb to the host?

       0 likes

  3. The Cisco MDS directors have had the ability to treat their smaller SAN switches as ‘remote line cards’ for a LONG TIME now.This makes it easier to manage because you don’t have to use extra domain ids etc for your edge switches, it also makes some VSAN stuff a little easier as well.But really its not a new feature.

       0 likes

  4. To further clarify, FCoE is not supported on GbE interfaces of the Nexus Fabric Extender Series first model Nexus 2148T. However, FCoE is supported on 10GbE interfaces of the parent Nexus 5000 switch which can be also be direct-attached to the servers or a future hardware version of the Nexus 2000 series that supports 10GbE interfaces for server connectivity.

       0 likes

  5. The Nexus 2000 FEX data sheet shows FEX-10Gig models. Is that on the road map? If so, what would be the benefit over just using 5020′s are the TOR?http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps10110/data_sheet_c78-507093.html

       0 likes

  6. Great Discussion, and I appreciate the clarification about the FCoE. I’d like to create a powerpoint slide demo-ing the placement(s) of the N2K in a DataCenter. Where can I get the icon?

       0 likes

  7. Seems a great product, but indeed it raises some important questions:- what about redundancy ? can you have one FEX going to two uplink switches, i guess not. So you need two FEXes in each rack.- what about oversubscription ? if you have 2x10GE uplinks, the ackplane”” of the master switch becomes 20GE per “”slot””. A C6500 today already has 40GE.- i guess a 1GE user-port FEX is not FCOE capable.- and a 10GE user-port capable FEX seems crazy. What kind of uplinks do you need to keep the oversubscription limited ? 40GE or 100GE links ? In this switch, how do you assure FCOE buffers ? you have oversubscription so how do you assure guaranteed FCOE transmission ? What happens if an uplink fails, what if all but one uplink fail ? If you connect 10x10GE servers, you will never get all the FCOE traffic across one 10GE uplink…”

       0 likes

  8. Hi,Today Fibre Channel is already available at 4G speeds today and on top of that if we are going to send traffic from a few Gigabit Ethernet interfaces then minimum speed required for running FCoE would be 10GE. Understandably line-rate speed may not be required but since servers may not drive that kind of traffic but as a technology it makes sense to run FCoE at 10GE or higher speeds. Hence, we do not plan to support FCoE on Nexus 2000 Series Fabric Extenders or Nexus 5000 Series switches at 1GE speeds.Malhar ShahProduct ManagerServer Access and Virtualization BU

       1 like

  9. Nexus 2000 Series Fabric Extender (FEX) offer 48x1GE (1000BaseT) interfaces for connectivity to the host and up to 4x 10GE uplinks to parent switch (Nexus 5000). User can connect any combination of 1 to 4 uplinks between FEX and N5K. This downlink-to-uplink ratio will determine the oversubscription level. Best case scenario – 40 hosts are connected to FEX and all 4 uplink ports are used, oversubscription is 1:1.Worse case scenario – 48 hosts are connected to FEX and only 1 uplink port is used, 0versubscription is 4.8:1.The real benefit of Fabric Extender is in the flexibility it gives by putting the network admin in charge of how much oversubscription he/she wants to achieve at the access layer. Couple that with cabling flexibility achieved by having short cable runs between server hosts and FEX or FEX and N5K, it really allows admins to decide on an optimal cabling and network design that suits their needs while providing an integrated management model where multiple FEX units are managed by a single (or pair in future) N5K.Today FEX supports management thru only one N5K switch but FEX will be managed by a pair of N5K switches in the future providing scalability and high-availability to the customers.Malhar ShahProduct ManagerServer Access & Virtualization BU

       1 like

  10. Omar Sultan

    Jeff:With the 10GbE FEX, the advantage would be to keep management simplified–you would still have only a single switch to manage. N5K/N7K + FEX gives you, essentially, a deconstructed modular switch.

       0 likes

  11. Omar Sultan

    Doug:Good point to clarify. The FCoE standard itself is only defined for 10GbE and while the FEX architecture will support FCoE, the 2148T is GbE (notice the GbE) and we don’t support it on that platform.

       0 likes

  12. Wendy, The 5020s do support the FEXes and with the 4.1.3 code base can be put into a vPC setup.Hint … start off labeling the FEXes connected to one 50xx as 101,103,105 and to the other as 102,104,106, ….

       0 likes

  13. Is there any development on increasing the number of FEXs supported on the 5020? Additionally are there any plans to increase the number of VPC host connections? I believe the current limitation is 480. Ballpark time-frames would be appreciated if possible. Thanks for the info.

       0 likes

  14. The posting says he FEX is currently supported on the Nexus 5000 with support for the Nexus 7000 and the Catalyst 6500 on the way””Do you know when it will be available on Nexus 7000 and what version and type of license will be required?Thanks in advance!”

       0 likes