Outside of that large, black, monolithic machine in the middle of the datacenter referred to as the mainframe, there aren’t that many servers that require as many network and storage connections as the backup server. It’s not really sexy, it’s not computing Pi, generally doesn’t run a hypervisor and is bought with one goal in mind, move data. Not just some data, but a lot. These machines often move all of the data in your datacenter off of disk and onto tape, either real or virtual. In many datacenters, these backup servers are sometimes the only non-x86 platforms left due to their ability to contain high numbers of HBAs for SAN connectivity and NICs for network connectivity. They’re like the tractors of the datacenter.
I was talking to a client of mine the other day and they were talking about moving their backup servers from their existing RISC platform to x86 and Cisco UCS was on the table. The current server had a combination of 14 HBAs and NICs in it for getting to various LAN and SAN networks, to move the terabytes worth of data from various sources into their Virtual tape library infrastructure. They were using 8Gb Fibrechannel HBAs and 10GbE NICs. She was looking for a UCS server that had similar capacity in terms of slots and expansion.
I looked over at the various models of C-Series rack mount servers and didn’t see one with that number of slots, but then I realized that I can do better.
Get rid of the HBAs and NICs.
If you look at a backup server, it can be backing up from many different sources and use different protocols to move that data. For example, it could be reading data off the LAN, and then writing it to the SAN. It could read it off the SAN and write it back to the SAN. It’s even possible to read from the LAN and write to the LAN. All during the same backup window. So if you had some backup server with 6 HBAs and 8 NICs, if you have a backup job that requires reading and writing to/from the SAN, all those NICs are wasted. Likewise if you’re performing a backup job that only requires LAN access, all those 8Gb FC HBAs are wasted as well. Seems like the current model has a ton of wasted bandwidth in it.
So what if we just replaced all those HBAs and NICs with Converged Network Adapters (CNAs) that would allow the server to communicate via FCoE? Now we just need to provide enough overall bandwidth to the server and can leverage all the adapters. So let’s use the above example with 6, 8Gb HBAs and 8, 10Gb NICs.
If we look back at J Metz’s blog regarding 8Gb FC vs 10GbE FCoE, we see that an 10GbE connection can carry 50% more traffic than an 8Gb FC HBA due to a more efficient encoding method. The above server is currently spec’d out to use 4800MB/s (6 x 800MB/s). However, to get this amount of bandwith with a 10GbE CNA you’d need 4 CNAs, since each CNA can move 1200MB/s. So we’ve already cut out 2 HBAs, and down to a total of 12 adapters instead of 14.
While a 10Gb CNA doesn’t provide you with any more LAN bandwidth than the 10GbE NICs, there is one advantage. Flexibility. Any and all of the CNAs can carry both LAN and SAN traffic. Let’s say we propose using the UCS C460 which has 10 PCIe slots in it.
If the server is doing a job where it is reading from the SAN and writing to the SAN. Instead of being limited to 4800MB/s of bandwidth, the server could leverage all of the CNAs to read and write from the SAN. You could now leverage 10 x 1200MB/s or 12,000MB/s for that SAN only job. Likewise for a network only LAN to LAN only job you could leverage 10 NICs instead of 8 as before.
<added on Sept 11, 2012 to the chagrin of my editors>
I was explaining this post to a client today, and it pretty much boiled down to a simpler example with easier to understand numbers. If a 10GbE NIC can push 10Gb of data and a FC HBA can push 6.8Gb of data, then if you replaced the 8Gb FC HBA with a CNA and then rate limited the CNA to only allow 6.8Gb of FCoE traffic, you’d get the leftover 3.2Gb of LAN bandwidth for free. Who doesn’t like free? Also, leftovers aren’t that bad either.
<Editors: I’m so sorry, I’ll have the interns get you bagels>
Do you even need that much bandwidth going into and out of the server? Probably not and therefore my client could reduce the overall number of CNAs in the server from 10 down to maybe 8? How would they know how many to reduce it to? Measure the bandwidth usage on the existing server.
But what if you needed to connect to all these existing FC networks?
That my friends is where you leverage the Nexus 5000 as your access layer. Standardize the server with CNAs and unified connectivity to the access layer and then let the access layer connect to all the various LANs and SANs. With 48 or 96 onboard ports of FC/FCoE/Ethernet and not including expansion via Nexus 2000 Fabric Extenders, you’ll have all the ports you need to connect to your existing FC SANs and the ability to provide the flexibility to take advantage of the dynamic backup loads that your infrastructure requires. As an another option, for those of you that leverage the Nexus 7000 for “end of row” topologies, you could connect the server to the Nexus 7000, and then for storage connectivity, connect to your existing MDS switches via FCoE using the 8 Port, 10GbE FCoE module, installed in the MDS, providing solid multi-hop connectivity.