FCoE versus PCI Express Multi-Root IOV, and questionable analysis
I heard some interesting noise recently comparing PCI Express Mutli-Root IO Virtualization to FibreChannel over Ethernet.Or in networking parlance PCIe MR-IOV vs FCoE(now there is a mouthful!).
I’ll get to a technical analysis in a bit but wanted to put as a disclaimer I have not read the report. Why? Because I don’t deign to pay the price of a small car for the right to read less-than-insightful reports. Additionally, there is not enough value in the seat license to warrant the expenditure, and the analysis should be independent and be analytical. While I cannot in any way prove a bias in the analysis that says ‘FCoE is a folly’ I would ask that the technical facts of the systems, the vendor support, ability to execute in the current economic climate, and the benefits today to customers be used to evaluate these technologies.
This is not the case for many of the industry analysts, just a small few. But those small few unfortunately ruin the bunch. As many analysts like Drue Reeves at Burton Group, Vernon Turner at IDC, Rob Whitely at Forrester, Andreas Antonopoulos at Nemertes, and Zeus Kerravalla at Yankee Group know we sometimes agree and sometimes disagree with each other, discuss and debate many different opinions, and in the end share a healthy respect for each other because we jointly feel that we are all trying to act in the customers best interest.
So when I read an article about the ‘Folly of FibreChannel over Ethernet’ advocating a technology designed for deployment between a North Bridge and a I/O expansion module to be extended outside of the chassis, over a wire, with a repeater bridge adapter board, to an external device that then houses the network interface modules for all of the servers that may connect into it I am a little circumspect of the analysts intention. Why you may ask?
- PCIe is a bus that is internal to the server, using physical addressing. In an increasingly virtual world using a physically addressed bus as the network is a bit odd.
- PCIe was designed to be located inside the server chassis, it was not designed nor was the data link structured to allow for longer-haul transmission.
- These implementations are called ‘Multi-Root IOV’, there is not a standard for these. It is based on the premise the PCIe can be extended outside the server box to another box at the top of the rack that houses the NICs, and other expansion boards.
- PCI is an extremely chatty protocol, very sensitive to latency, and adding any noticeable propagation delay to the transmission will noticeably degrade the performance. PCI-SIG understood this well when they standardized SR-IOV, Single Root = Single Server. But these hopeful implementations are for multiple servers.
- PCIe is not a switching standard, and therefore it does not define how two servers are going to communicate.
- The only way to create a communication path between the two servers is to translate PCIe into Ethernet or FibreChannel, then switch, then go back to PCIe, then go along the wire to the server again. Sort of purpose defeating since history has shown that stateful and CPU dependent gateways do not garner market traction.
- Not sure how a MR-IOV scenario addresses the need for Fabric-A/Fabric-B models of high availability.
- And lastly not sure at all how a MR-IOV solution would identify the Virtual Machine and signal the VM identity to the NIC to the switch for policy enforcement.
I do want to add though that if the technical, operational, protocol, and ASIC/SW implementation issues get worked out over the next several years that it usually takes to align these types of forces there may be an interesting play for the two to work together with FCoE coming out of the MR-IOV ToR device. But until that day, several years in the future at best, especially given the economic times we are facing, I think I will put my bet strongly on the FCoE implementation, especially when coupled with the VN-Link technologies introduced jointly between Cisco and VMWare. And doubly so since EMC and Network Appliance have both announced support for this architecture and product shipping.
A few proof points-
Network Appliance to Support FCoE and Nexus 5020
Chuck Hollis’s Blog from EMC discussing FCoE and why it is getting serious.
A wonderful Quote from the FCIA press release “The timely development of the FCoE standard by T11 Fibre Channel standards body allows a high degree of compatibility and interoperability for today’s FCoE products.”
FCoE = Real, shipping, in production, solid vendor ecosystem support, final stages of standards track.
dg (putting on asbestos suit right now, something tells me this is going to be a lively post/comment thread)
Posted by Douglas Gourlay at 05:47PM PST
Silvano Gai Oct 18, 2008
I think we are really comparing apples and oranges.
Ethernet is the winning technology in the Data Center.
In 2009 most of the servers will have 10GbE on the motherboard, in 2010 40GbE will start to appear on the market and 2011 will be the year of 100GbE coming to market.
FCoE is the way to carry FC over Ethernet. FCoE is a simple encapsulation of one FC frame over one Ethernet frame.
The FCoE ecosystem includes all the major network, server, virtualization and software vendors.
Just have a look at http://www.fcoe.com to have an idea of the tremendous ecosystem that is pushing FCoE.
The FCoE standard is already in letter ballot and new products are appearing on the market every day.
PCIe is a nice server bus, but it is not a network technology and therefore will never succeed competing with Ethernet.
My 0.2 cents
—Silvano