What is a FEX?
So what is a Fabric Extender? In short, its going to change the way to network your server racks. The best way to understand the Nexus 2000 Fabric Extender, or FEX, is to view it as a remote linecard located at the top of the server rack.
Why would you want to do this, you may ask?
It turns out, there are quite a few reasons.
The two basic approaches to server networking are top-of-rack and end-of-row. Top-of-rack or ToR (where the switch is located, you know, at the top of the rack) benefits from a simplified, flexible cabling scheme—short intra-rack cable runs with a small number of uplinks to the agg switches. The downside of this approach is switch sprawl: 12 racks of servers usually means at least 12 switches. All those switches drive up the cost and complexity of the access layer of the network. The second approach is end-of-row switching or EoR (guess where the switch is), has every server cabled back to a single switch dedicated to a row of server racks. The upside of this approach is its a simplified management environment (one switch to manage) and efficient, but the cabling costs are significant and it is difficult and change out cabling. Its important to point out that both approaches are valid designs and the ultimate choice is driven by broader architectural requirements such as traffic patterns, L2/L3 boundary and the like.
With the FEX, you get the best of both worlds. You get the low cost and cabling flexibility of a ToR design with the management simplicity of an EoR design.
I think the management angle is where the true brilliance lies—the FEX will save you time, money and effort. As I mentioned earlier, each FEX acts like a remote line card, so you don’t have to individually manage each FEX—everything is inherited from the upstream switch. Basically you could have an entire rack of servers pre-cabled with a FEX delivered to your data center, plug it in, and that would be about the end of it. The FEX would automatically inherit the config of the upstream switch, so whatever policy and features you have configured will automatically propagate. If you need to upgrade software or deploy a new feature, simply make the change on the upstream switch, and it will automatically propagate across a dozen racks of servers.
About that upstream switch—the FEX is currently supported on the Nexus 5000 with support for the Nexus 7000 and the Catalyst 6500 on the way. Currently, the Nexus 5020 can support up to 12 FEXes.
For those of you who have been following Nexus developments, you have probably noticed the Nexus 5000 + FEX approach gives us the same virtual modular switch architecture as our Nexus 1000V, so if you were to Telnet into a Nexus 5000 and did a show interface command, you would see all the interfaces local to the Nexus followed by the interfaces on each connected FEX. Each FEX port is individually addressable and manageable.
A couple of things to bear in mind. Because the FEX inherits the features of the upstream switch, it can support advanced features like FCoE and VN-Link, although, in the case of FCoE, the uplink is over-subscribed, so you need to pay a bit of attention to your traffic engineering. Also, while the FEX is designed to be broadly applicable, there are scenarios where it may not be the best choice—the best example of this is if your L2/L3 boundary is at the rack level.
The initial iteration of the FEX is the Nexus 2148T which give you 48 copper GbE ports plus 4 SFP+ 10GbE uplinks—ideal to support your GbE attached servers while still taking advantage of Nexus and NX-OS features and easing your migration to 10GbE.
For more info the the Nexus 2000 family, follow this link.
Posted by Omar Sultan at 03:27AM PST
Colin McNamara Jan 29, 2009
FCOE on the FEX? I have seen the 4th queue dedicated for FCOE traffic, but haven’t heard anyone mentioning supporting FCOE out the 1 Gig ports. Does your statement mean this has changed and we can use FEX/FCOE as replacement to ISCSI over 1 Gig?