3-for-2: The FCoE Bandwidth Bonus

April 29, 2011 - 4 Comments

“Dude, you’re killing me!” my friend said to me.

I raised an eyebrow. “What did I do now?” Quick witted, I am.

“I know that you’re all over this FCoE stuff,” he said (actually, he didn’t use the word “stuff“, but you get the idea. “But what’s so great about a lousy 2G of bandwidth?” He sipped his beer, pausing for dramatic effect.

I was confused. “What do you mean, 2 Gig of bandwidth?”

“Look,” he said, sitting his beer down and enjoying his gotcha moment. “If I have 8Gb Fibre Channel and I move to 10Gb FCoE on my Interswitch Links, I’ve only gained 2Gb. I mean, what’s so great about a lousy 25% more?”

I shook my head. “You don’t get 25% more,” I corrected.

His smile broadened. “Aha! I knew it! There’s some overhead crap you gotta deal with, right? It’s even less than that.”

“No,” I said slowly. Now it was my turn to add dramatic effect. “You get 50% more bandwidth with FCoE.”

Every once in a while you get moments of pure schadenfreude. This was one of those moments. His moment of gotcha had spun around on him, and his look of dumbfoundedness was truly entertaining. Truly.

“What do you mean? You get 10G, and you have 8Gb for Fibre Channel. How do you work that one out?”

“Look,” I said. “When you put traffic on the wire you have to encode it, right?”

He nodded.

“And there are different ways to encode Fibre Channel than you do with 10 Gbit Ethernet. In fact, with 10Gb Ethernet you have more efficient mechanisms for encoding.”

He looked blank. It was obvious I was risking going down a technical rat-hole.

“Look at it this way,” I said, taking out a pen and starting to draw on a napkin. “Let’s talk real numbers.”

I sketched out the diagram:

“Because of the way the traffic is encoded between 8 Gb Fibre Channel and 10 Gb Ethernet, you’re talking an actual throughput of 800 Megabytes per second for 8Gb FC, and 1200 MB/s for FCoE,” I said, drawing a little star around the “50%” for effect. “The purists will give you more exact numbers but essentially these are close enough for a discussion over beer.”

“Nice,” he said, commenting on my artwork. He was pouting.

“What this means, then,” I continued, “is that if you’ve got your ISLs with FCoE traffic, like we do for the MDS line, you can have 50% more bandwidth per line. That means that you only need two links to have the same kind of throughput as three 8Gb FC ISLs.”

“Hmmm,” he said, pensive. “I guess you can have some cable reduction in the core, then.”

“I suppose,” I said. “But realistically you are getting more efficient throughput overall, which is the key thing here.”

He took another sip of his beer, this one a slightly longer gulp to give himself time to think. Putting his glass down, he asked, “What about 16Gb Fibre Channel?” he asked.

“What about it?” I responded.

“Well, where does it fit into your little chart there?”

“Good question,” I said. “16Gb FC has the same type of encoding as 10Gb Ethernet, so it goes up to about 1600 MB/sec.”

“So what you’re telling me, is that 16Gb FC isn’t 6Gbps faster than 10Gb FCoE?”

“No,” I agreed, stifling a laugh so as not to hurt his feelings. “It sits right in the middle, actually. You get a lot more bandwidth than you think with FCoE.”

“So 16Gb is only…” he started doing some calculations in his head.

“About another 33% more than 10Gb, yes,” I finished for him. I reached for the napkin and pen again. “It looks something like this.”

I wrote a third line underneath and started going crazy with my arrows and stars:

It was my turn to take a drink as he contemplated this new development. “Okay,” he said cautiously, “but that means that 16Gb FC can be good for ISLs, right?”

I nodded. “Sure,” I said. “16Gb Fibre Channel can be really good ISLs in the core, there’s no question about that. For some people this will make a lot of sense.”

He nodded. He seemed glad to be able to contribute to the conversation. “However,” I said, as his face fell, “there aren’t any 16-Gig Fibre Channel storage targets available at the moment, while there are FCoE targets available, with more coming soon.”

“Ah,” he said. “But you’ll see some come eventually, though.”

I agreed. “Very true. And as I said, this will be very useful for customers who are looking for that kind of connectivity.

“The issue comes when start looking long term,” I said. “What happens when Ethernet goes to 40Gb? Or 100? If you’re looking for pure speeds and feeds here, it gets pretty hard to ignore those kinds of numbers.”

My friend signaled the waitress for another round. Even though he had just had one, he looked like he needed a drink.

“All in all,” I said, “It depends on what you are looking to do inside of the data center. “If you’re looking to make your overall data center smarter, and more flexible, then do you want to continue to invest in separate systems?”

“Separate systems?” he repeated.

“Sure,” I said. “If you want to do 16Gb you need new equipment, right?”

“I guess,” he said.

“Generally speaking, you do,” I confirmed. “You need new equipment, new hardware, and probably new operating systems to go along with that hardware. This means that you’re going to be refreshing your equipment without a clear ‘next step’ afterwards.”

I paused for a sip. “Do you ever play pool? Eight-ball, nine-ball, something like that?”

He nodded. “Yeah, used to be pretty good, actually.”

“Well, in pool you have to line up your shots, right? You have to have an idea of what the ball you have to get after the one you’re about to hit.”

He nodded. “Well,” I said, “You have to do that in your data center as well. Once you go through an equipment refresh to accommodate this new equipment, then what? 32Gb Fibre Channel? By the time that happens you’ll be off to the races with 40 and 100Gb Ethernet. You have to line up your shots for the next refresh that’s coming.”

The next round came. Mmmm, beer!

“Okay, J,” he said. “My brain hurts. Let’s talk about something else. You gonna get the new Black Ops maps on May 3?”

I grinned. “You bet. Can’t wait for that date.”

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.


  1. J, even if you take into account the 10G Ethernet speed it’s still divided over IP, FC and one or more other protocols. Please do take that into consideration as well. So if your 10G link requires 5G of IP traffic your theory goes down the drain.

    • Thanks for reading and taking the time to reply, Erwin. As the tenor of the piece indicated, I was referring to ISLs and the bandwidth for dedicated FCoE links that would connect to MDS FC storage. Given that’s the case, there is no sharing of bandwidth (i.e., no LAN traffic) and so the situation stands as stated.

  2. nicely wrote!