Nexus Category Archives

April 30, 2008

Interop 2008 Winner Best of Award for Infrastructure

Cisco Nexus 7000 Series is the recipient of Interop 2008 Product of the Year. It is the first Data Center class switch in the Nexus portfolio offering transport flexibility, operational continuity and infrastructure scalability.
Cisco Nexus 7000 Series

Posted by John Murphy at 08:32 AM Permalink | Comments (1) | TrackBacks (0)

April 14, 2008

Ummm, is your switch... Backwards????

When we introduced the Nexus Family as a family of switching products that were purpose-built for the data center we meant this at every conceivable level. Silicon, Systems, Software, Hardware, Power Architecture, even a 'Form Follows Function' physical design of our chassis.

At the Partner Summit in Honolulu last week, amidst this thing called VOG (Volcanic Fog) I had someone ask me why the Nexus 5000 looked so different than the 7000 from an aesthetics point of view. I came back to the 'form follows function' design methodology. The Nexus 5000 is designed for the data center....

What side of a server are the network ports on- THE BACK
Where does hot air need to exhaust- THE BACK

If we hold these two truths to be self evident (or at least documented somewhere by ASHRAE or a random and cursory check of most server vendors portfolios) then we can infer the following logical statement- "The exhaust and the ports need to be on the back of the LAN Switch" as this will allow shorter and less complex cable runs, simplify operations, and maintain the established airflow models.

Thus we can again infer, that just about every product from every other company, designed for a rack switching deployment is... well..... backwards!

We have a few of those as well, fine for 1 Gigabit, and enabling a transition to 10Gb. But when we purpose-built a product for 10GbE, FCoE, and designed uniquely for data center deployments - it was time to change.

Posted by Douglas Gourlay at 06:19 PM Permalink | Comments (0) | TrackBacks (0)

NX-OS 'VERY IOS-Like'

Just saw a good write up from Mark Lewis over at NEtwork World on NX-OS analysis he did off a build we sent over to him here.

'WR T' still to be added, or you can alias it ;)

Posted by Douglas Gourlay at 10:32 AM Permalink | Comments (0) | TrackBacks (0)

April 10, 2008

The Other Nexus News

By now, I would hope, you are aware of our newest addition to the Nexus family, the Nexus 5020. I’ll dig into that a bit more in a second, but the other Nexus news I wanted to note is that we have started customer shipments of the Nexus 7000 this week (yay--although I am guessing the blogger who accused us of only having cardboard models might be a bit disappointed).

So back the the Nexus 5020 launch this week. I think this announcement is going to take a while for folks to digest because there was so much to it. Not only did we add to the Nexus family, we delivered FCoE with an eco-system of partners and we rolled out data center Ethernet.

I think these two technologies are are going to be game changers once folks have a chance to get acquainted with them. FCoE now lets us take advantage of a unified data center fabric in a way that provides investment protection and is non-disruptive to the SAN and the storage team. Meanwhile, data center Ethernet ups the ante on what we can expect from our layer two transport and is an advance that will benefit not only FCoE, but also iSCSI and pretty much any other traffic you care about.

Finally, I think its important to note that those customers that will shortly be taking delivery of their Nexus 7000s (no we don’t have consensus on what the plural of Nexus is) are already good to go for both data center Ethernet support and FCoE support. Because of the planned forward investment protection in the platform, all they will need to do is add are the appropriate I/O modules--no other upgrades necessary.

Posted by Omar Sultan at 11:49 PM Permalink | Comments (0) | TrackBacks (0)

April 08, 2008

Cisco Nexus 5000 Enables Virtualization Optimization

Dante Malagrino and Ed Bugnion explain how Cisco Nexus 5000 Series switches enable virtualization optimization for data center managers.

Posted by John Murphy at 04:28 PM Permalink | Comments (3) | TrackBacks (0)

April 03, 2008

Gaming Routers and Switches

As a self-avowed bit of a gamer (yes I have my Level 70 World of Warcrack character, and can go all the way back to Ultima Online, MUDs, Zork, Bard's Tale, and Wizardry (anyone remember Werdna?) to establish a lineage...) I was reading a question today about 'What is the Best Router for Gaming'???

This is a very good question, but there are two sides to it! Sure, from home I would advocate something like a Linksys WRT54G or newer 11N radio. But what about on the hosting side of things!?!?!?!? A gaming company (of course to remain nameless) commented to me that the Cisco Nexus 7000 would be ideal for their service, being able to offer uninterrupted services to their end-users while using Zero Service Disruption upgrades to maintain security compliance and such as well. CRS is great for the peering edge routers to connect to hundreds of peers and optimize latency and ping time, especially useful for twitch games (love Call of Duty 4 btw... fabulous work by the devs there.)

Are there ways to better optimize designs of hosting farms for gaming? Differences between MMORPGs and twitch games? Can we get a decent combined MMORPG with 'twitch' combat models? What has to happen to the network to enable that...

Posted by Douglas Gourlay at 05:36 PM Permalink | Comments (3) | TrackBacks (0)

March 30, 2008

A quick word from the field...

Brad Hedlund, a Cisco Systems Engineer out of Illinois and author of a very nice blog that strives to help network engineers worldwide wrote this nice write-up on the Nexus 7000.

I like how Brad brings forward some of the key differentiators that will be sustainable for a very long time. Brad- have you had any feedback on the usability feature set on the Nexus?

Posted by Douglas Gourlay at 09:05 PM Permalink | Comments (1) | TrackBacks (0)

March 23, 2008

'Om'nidirectional Nexus

Was just reading this article on GigaOm about the Nexus 7000 as a sort of mutated blade server chassis. I wanted to reply to it that there isn't much possibility of the Nexus 7000 as a server platform itself, although it, I believe, is one of the best Server Interconnect platforms ever delivered. :)

Interesting conjecture, but doubt we'll see that as an evolutionary path unless there is a major reduction in power draw for both CPUs and DRAM.

Posted by Douglas Gourlay at 06:51 PM Permalink | Comments (0) | TrackBacks (0)

March 17, 2008

Why You'll Want This Switch, Part 3

So, as an impressionable young “yout”, as Joe Pesci would say, in the early days of networking, I learned an important lesson. I have to admit I started out as a Wellfleet bigot (http://en.wikipedia.org/wiki/Wellfleet_Communications for those of you who, sadly, have no idea what I'm talking about). At first blush, it looked to be a better box: elegant hardware architecture, checked all the boxes in terms of standards compliance, better performance (on paper at least)—hands down, the BCN won the data sheet wars.

But a curious thing happened…

...I learned that when it came to deploying features and functionality that mattered to people who’s jobs depended on the network being up and running, those people consistently chose Cisco. Case in point: I worked for a service provider that sold both Cisco and Wellfleet access routers as part of our managed router service. I learned that our field engineers would take a Cisco router out with them, even if they were installing Wellfleet routers at the customer prem. Why? Because the Cisco router did a better job of helping them troubleshoot problems bringing the network—‘nuff said. I started prepping for my CCIE shortly after.

So, fast forward a decade or so (long enough for Wellfleet to become Bay Networks and then part of Nortel), and you can see this heritage in the Nexus 7000. As we look to help our customers build more scalable systems, one of the areas we concerned ourselves with is helping the operations staff scale by giving them tools to help them be more productive. We put a lot of thought into the Nexus 7000 to make it more manageable and we think you will find we baked in some useful innovations. One of my favorites is CMP, which is an independent sub-system that gives you lights-out style management capabilities with the system. Anyone who has had to reboot a remote system and is familiar with the praying that goes on while waiting for the login prompt to come back will appreciate this feature. Folks have been doing this for years with terminal servers and modems, but we built-in the functionality and let you access the system at network speeds. Another favorite feature is a built in protocol analyzer for the control plane. We built WireShark (http://www.wireshark.org/) into the Nexus 7000 to give you a distributed sniffer network to allow you to do packet captures and decodes of control plane traffic. Off the top of my head, couple of more manageability features include Call Home and Flexible NetFlow. Finally, all the high-availability features, like the stateful restartable processes help keep the staff out of fire drill mode in the first place.

While I will always have fond memories of the venerable BCN, I think the choice then, and now, is a no-brainer. So, why do you want this switch? :)

Posted by Omar Sultan at 09:21 AM Permalink | Comments (0) | TrackBacks (0)

March 11, 2008

Don't take my Kodachrome Away...

I have to hand it to Brad Reese for being able to drive a discussion :) Ok, or maybe to fan the flames of a mild-mannered campfire into that of a forest-sweeping conflagration. But either way it is just downright funny to me the comments people are making on Network World.

I guess being called a 'Schmuck' isn't so bad, even though it means I am probably off of our northern neighbor's holiday card list for eternity.

I titled this 'Don't take my Kodachrome Away' not only because I happen to like the song, but also because we asked one of our top customers once if they thought it would be to their advantage for us to invest more in using merchant silicon, or to keep innovating in our own custom designs. This rather large web company was adamant that we keep our own silicon investments up. Why? Because if the market 'devolved' to only having two main suppliers of switch fabrics, forwarding engines, and port asics there would be a significant reduction in innovation, but also an increase in 'churn rates' of their equipment.

Not my opinion mind you, a customers.

But one worth digging into a bit... especially the last point as the one about innovation has already been supported by our competitors who praise merchant silicon, but when it comes to invention/innovation validated the custom ASIC route. (note to any disaffected silicon developers- we love you guys)

But as to the lifecycle comment- when we build high-end modular platforms we make design decisions around building platforms that last at least a decade. This requires not only foresight, participation and leadership in standards bodies, and the deliberate application of decades of lessons learned in leading the Ethernet market but more importantly it also requires an architectural decision. These architectural decisions around lifecycle preclude merchant silicon- mostly because the vendors of these chipsets have a penchant for changing their system architecture every few years. If Cisco followed this route we would be no different from the companies who change their switching platforms introducing new ones and obsoleting others at 3-4 times the rate we do.

btw- if we forklifted the Catalyst 6500 at the same rate as some of our merchant silicon based competitors we would generate a pile of Catalyst 6500s about 53 miles high... annually. Now that may be a good thing for revenue, but its not the right decision for our customers.

Happy Tuesday, and may it continue to be a colorful one...

Posted by Douglas Gourlay at 05:51 PM Permalink | Comments (0) | TrackBacks (0)

February 26, 2008

TrustSec - Secret Weapon for Flattening Networks

Ever feel like Doctor Evil? You know those scenes where the 'secret weapon' is about to be revealed? Am a personal fan of Sharks with Laser Beams on their foreheads... but notwithstanding those are hard to get into the office on Tuesday afternoons I'll settle for today's secret weapon- Cisco TrustSec.

Colin McNamara did a quick write-up on his blog here. that talks a bit about how TrustSec works to provide a layer of abstraction between the users address and the users security policy. (editorial note: you can replace user with server, application, VM, etc)

Why is this important? Well remember the good ol' days? Not back so far as to when we walked along train-tracks with BB Guns whistling 'Stand By Me' but more the days when we used the 3rd Octet of the subnet to equal the VLAN number which in turn mapped to the HSRP group number and then mapped to the subinterface number in a classic campus design? Those days were simple! We didn't have 15-500 different groups of security with segmentation rights and per-user policy and such.

TrustSec helps us get back to that simple concept of building the network you need with the addressing structure you want, then overlaying hte right policy and segmentation implementation in a scalable and manageable way.

This means you can build a flatter network. One where we could all be on the same subnet, yet have differentiated policies. This then lends itself very easily to a world of VM portability where the security policy moves with the VM.

Thus, the secret weapon :) Thoughts? I'll be interested to hear the feedback as people 'cowboy up' and try this technology out in labs and such...

Posted by Douglas Gourlay at 01:20 PM Permalink | Comments (0) | TrackBacks (0)

February 21, 2008

Why You’ll Want This Switch, Part 2

So, if bigger/faster/denser is not the reason to drop a Nexus 7000 in your network, why is a compelling reason?

In my last post, I touched on NX-OS, as the foundation for many of the cool things the Cisco Nexus can do now and in the future. One of those areas is operational continuity. We have the component redundancy that’s table stakes in a next-gen platform with redundant power, fans, supervisors and fabric modules, but we wanted to raise the bar in terms of protecting the network services this platform delivers—what we call a zero service disruption architecture. We don’t just want highly available hardware; we want highly available network services, because that’s what really counts.

One design decision we made was to loosely couple the control plane and the data plane, so we can do things like in-service software upgrade without disrupting packet forwarding—we can even do firmware upgrades to the I/O modules without disrupting packet forwarding.

One of the cooler features on the Cisco Nexus is stateful process restart. With NX-OS, we can re-start individual processes in a way that maintains state data, so, for instance, if you need to restart spanning tree, the system will maintain state data, so you can avoid the resulting chaos of having to re-learn addresses and re-establish the topology. Along those same lines, the Cisco Nexus also has checkpoint/rollback functionality, so if you push out a change that does not work as expected, instead of having to troubleshoot it in real time, you can simply rollback to the previous working config.

Finally, from the Department of Attention to Detail, we even short-pinned the fabric modules (i.e. the input pins are shorter than the output pins). Why should you care? Well, in case someone accidently yanks out the wrong fabric module, the input pins will disengage first, to stop receiving new packets, and the fabric module is fast enough to empty before the output pins have disengaged. And the system will automatically adjust to the new fabric capacity and keep humming along.

So, there are just a few of my favorite features—things I wish I had when I ran data center networks. For more info, check out this whitepaper: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/ps9512/White_Paper_Continuous_Operations_High_Availability.html

Posted by Omar Sultan at 04:26 PM Permalink | Comments (0) | TrackBacks (0)

February 20, 2008

Can we agree on how to measure performance?

Recently there was a posting from an irrationally exuberant vendor employee about the Nexus 7000. Brad Reese from Network World commented on it a bit here. While engaging in a bit of a written riposte it occurred to me that maybe we need some clearer idea, as an industry, on how to consistently measure performance.

I have heard for years of Cisco Math, Cabletron Math, etc... well, today, I would like to propose that maybe we can all find some way to agree on 'math'. After all, even though not that mathematically inclined I heard once that math is an absolute truth.

So here is my cut at 'the rules' for stating your bandwidth on a switch....

1) Do not double count per-slot bandwidth. i.e. if a slot has 40Gbs of bandwidth and can support 4x10GbE interfaces worth of traffic, then it is 40Gb of bandwidth. I agree that this is 40Gb IN and 40Gb OUT, but alas its still 40Gb.

2) Double-Count a switch fabric. i.e. a 10-slot chassis with 100Gb per slot to every slot with no blocking in the fabric would be a 2Tb switching fabric. I think that this double-counting here has meaning: it is the actual capacity of the fabric, and it does let the casual observer get an indication as to why then a 10-slot 2TB Modular switch is better than 10 separate 20-port 10GbE switches. (try building those 20-port configs into one multi-stage fabric offering 100 host facing 10GbE ports and you'll see what I am talking about....)

3) Don't separately count the local switching capacity on a line-card. Yes, most all switches nowadays that are modular have distributed forwarding in some form or fashion at the high-end. Don't go counting that too. That's like saying- "well, ya see... I have 10-slots in that there chassis. I have 10 10GbE ports on each line card and a 100Gb to the fabric. I can forward from port 1 to port 10 without crossing that fabric. So thus I have 400Gb fabric on each line card and a 2Tb main fabric. So I get 10 linecards at 400Gb each or 4Tb and a central 2Tb so I have a 6Tb switch." I don't do this, please don't also do this... it's just poor form... not to mention pretty incomprehensible to most customers...

4) Packet Per Second forwarding rates. Again, single count them. Just because the packet goes in and comes out of a forwarding engine (or the header does if its not an inline FE) doesn't mean you get to count it twice... If you can do wire-rate at 10GbE thats roughly 15Mpps @ 64byte frames. btw- a highly unrealistic model as I have never seen an all 64-byte frame data flow in any production network.

5) Aggregate PPS forwarding rates- if you have 100Mpps on each of 10 line cards then you can do up to 1000 Mpps or 1Bpps. Simple...

6) Fabric Redundancy- if your switch has all fabrics ACTIVE and all are utilized count both, or all 5 or all 8, etc. If there is a N+1 model or 1+1 model and the spare is NOT utilized unless there is an outage, don't count it. Be prepared to prove this btw as I see this one getting messed around with a lot... Also data sheets should state the performance of a single-fabric and of a fully loaded fabric.

7) Oversubscription- I like oversubscription personally. I find that it lets you increase density and balance density and performance with cost to maximize the number of devices connected to a single network node. But not withstanding the philosophy of oversubscription if you are purposefully oversubscribing something somewhere, i.e. you have a module that has 8 ports of 10GbE with a 40Gb Switch fabric connection, call that out in the data sheet. Call it: Fabric Interconnect speed or something. It's not a sin: it's a way to increase the ports and accepts the reality that not all ports are active at the same time. I would recommend vendors put the right counters in though so customers can know when oversubscription is failing them....

8) Lifecycle Capacity. Some switching platforms are designed with 'headroom'. i.e. You know you can add a higher performance switching fabric or forwarding engine in the future lifecycle of a platform. Since this is important for customers to know in order to make a longer term investment decision it should be annotated, based on the same ways of counting described above, but also flagged as something that the platform WILL deliver in its LIFETIME so it is understood as a future deliverable but one the vendor is guaranteeing their customers and potential customers that they will absolutely deliver.

Allright, I may come back and add some more thoughts to this today and tomorrow as they hit me. But does this make sense?

Posted by Douglas Gourlay at 05:01 AM Permalink | Comments (2) | TrackBacks (0)

January 31, 2008

Let's Talk Bandwidth

You know, it's time I get back to my geeky roots here some and talk about tings that I find very exciting- backplanes, fabrics, forwarding engines.... BANDWiDTH! It's one of the many reasons I love blogging, because we can talk about things that matter in IT that most don't care about: Big Pipes!

On the Nexus 7000 we use numbers like 15 Terabits, 230Gb per slot, and so one... let's take one of these and extrapolate HOW we get there. I think this may help in drawing comparisons to other architectures out there.

230Gb per slot, and how we get there...

From the I/O module to the Fabric Module there is a series of thick copper traces and high density connectors. Each pair of copper traces is today clocked at 3.125Gbps. Each fabric channel is comprised of 16 pairs of copper for 16x3.125Gbps or 50Gbps half-duplex and 25Gbps of full-duplex bandwidth per fabric channel. (each pair of Cu at 3.125Gbps is unidirectional) We have to encode on the wire and we use a 24b/26b encoding scheme that yields just north of 23Gbps of real-world bandwidth per fabric channel.

Each switch fabric chip has 26 fabric channels. 2 fabric channels connect from each switch fabric ASIC to each line card in the 10-slot chassis. 2 Fabric channels per slot @ 23Gb each is 46Gb per slot per Fabric Module.

The Nexus 7000 can sustain up to five fabric modules. So with five switch fabric modules @ 46Gbps each we have 230Gb per slot. Now I do want to clarify that this is 230Gb IN and 230Gb OUT concurrently from every slot in the system when five fabric modules are present. Bandwidth scales up and down linearly with the addition or removal of fabric modules. You need a minimum of one, and max of five.

How do we assure we use all that bandwidth effectively?

Firstly the Nexus 7000 has a virtual output queue system with fabric arbitration, or for short and Arbitrated VOQ system. VOQ is where we have a queue on each ingress I/O module for every egress port on the system. We have 1024 queues on every I/O module, one for each potential 10GbE interface.

Packets go through L2 and L3 lookup on the I/O modules forwarding engine with then replies with a Fabric Port of Exit header. This FPOE header is used by the fab ric to forward a packet to the right destination. It is also used to to identify which VOQ to put the packet into.

If we have lots of 64b frames in the VOQ we can concatenate them so we only put one FPOE header on a larger block of 64b frames. If we have jumbo frames we segment them into smaller sizes that are more optimal for the fabric and deliver more deterministic latency, especially under load.

Each VOQ can only send data across the fabric when the receiving I/O module has the capability of receiving the data and serializing it onto the egress interface. This is extremely beneficial as it pushes all congestion to the source, aligns all of the larger buffering into one place, and is one of the key technical attributed that allows us to ensure a lossless fabric architecture. The fabric arbiter also ensures that we use all fabric modules bandwidth as efficiently as possible.

15Tb? Well if you have been building a spreadsheet or following along with a calculator just know that the same system applies to the 18-slot chassis with 16 I/O module slots. And that the 3.125Gbps number we discussed per pair of Cu unidirectionally? Well we know for certain we can do over 2x that today, if not even higher...

Posted by Douglas Gourlay at 01:27 PM Permalink | Comments (1) | TrackBacks (0)