Storage Distance by Protocol – FC, FCoE, and FCIP, Part I

A long time ago I got asked to write about how to use Fibre Channel over Ethernet (FCoE) for distance. After all, we were getting the same question over and over:

What is the distance limitation for FCoE?

Now, the short answer for this can be checking out various data sheets for the Nexus 2000, Nexus 5500, Nexus 6000, Nexus 7000, or MDS 9X00 product lines. But it didn’t answer the most obvious follow-up questions: “Why?” and “How?”

Problem is, whenever you start talking about extending your storage connectivity over distance, there are many things to consider, including some things that many storage administrators (or architects) may not always remember to think about. The more I thought about this (and the longer it took to write down the answers), the more I realized that there needed to be a good explanation for how this worked.

Generally speaking, the propeller spins the ‘other way’ when it comes to storage distance.

To that end, I began writing down the things that affect the choice for selecting a distance solution, which involves more than just a storage protocol. And so the story grew. And grew. And then grew some more. And if you’ve ever read any blogs I’ve written on the Cisco site you’ll know I’m not known for my brevity to begin with! So, bookmark this article as a reference instead of general “light reading,” and with luck things will be clearer than when we started.

One of the difficulties in answer the question quickly has to do with some rather fundamental differences between “native” Fibre Channel and Fibre Channel over Eithernet (FCoE), as well as Fibre Channel over IP (FCIP). Because the teams that handle distance solutions vary from company to company, they will have different assumptions about how to solve the problem. Some of these assumptions can lead to disastrous results when someone with an Ethernet background attempts to do FC distance with FCoE without understanding that the underlying principles are so different.

Bear with me for a little while, and I can help you avoid some of those potential disasters.

Think of it this way: we have different needs inside a data center than within data centers. By and large, we usually don’t push the envelope too hard inside a data center – they are usually well within the physical limits of cable lengths, optics power, and protocol capability.

When start looking at the connectivity between data centers, the rules change a bit. Suddenly the laws of Physics play a greater role in how successful we can be.

Mixing and Matching: Curious Results

Fibre Channel is known for it’s high reliability, Ethernet is known for its global reach. So, it stands to reason to put Fibre Channel over Ethernet (FCoE) in order to get high reliability with global reach, right?

FCoE does have some great benefits in the data center. It’s designed to allow us to simultaneously share multiprotocol traffic using higher capacity, as well as keep the traditional best practices and design principles in place.

While all this is true, it seems that many people are looking to use FCoE as the “holy grail” for Data Center Interconnect (DCI) storage solutions. After all, the cost of leasing a carrier line between data centers is extremely high, and if you can share links within a data center and save money, it makes sense that you’d want to do it between data centers for the same reason.

The trouble is that storage distance architectures add in pesky rules that most intra-DC architectures don’t have to deal with: trivial things like the speed of light, for instance.

This is the first in a short series to give you an idea of what you need to know before we get to the answer to the simple question above. None of these are intended to be the end-all, be-all of FCoE over long distances, but hopefully it could give you a starting point. Afterwards, we’ll take a look at each of these important pieces of the puzzle individually, and then finally we’ll examine how Cisco has several options for extending SANs over distances.

Remember, there are many other technical resources available for you to really dive into the nuts and bolts.

Obligatory caveat: Having said, that though, this is not a textbook on how to build, plan, architect and maintain storage distance solutions. To that end I’m trying not to make assumptions about where you, gentle reader, may be in your planning stages. I’m merely trying to give a broad-brushing of the canvas to illustrate what is involved in the process. You may not, for instance, want to build the pipe yourself. You may not want to share traffic. You may not have a set idea of how much distance you need in the first place.

So, consider this to be a rudimentary primer in the components that make up a solution, not a proscription for which components you need for your project, nor a comprehensive survey of all possible technology types.

Consider yourself warned. 🙂

Pipeworks: Build It, Fill It, Use It

At a broad stroke, building and designing a SAN extension means that you have to take in to consideration three main, high-level areas:

• You must Build the Pipe
• You must Fill the Pipe, then you can…
• Use the Pipe

The physical elements involved with building the pipe help to minimize the impact of the limitations of physics that we spoke of earlier. Now, fortunately for you, I’m not a physicist. That means that I’m not going to make allusions to acronyms and mathematical formulas to make your head spin (or will I? Mwah-ha-ha-ha-ha!).

While I’ll be describing each of these in turn, we’ll be saving the gruesome details for later articles.

Build the Pipe

At the moment, it’s sufficient to know that the physical layer of SAN extension is critical to being able to create a successful SAN extension. There are a number of different elements you need to consider.

For one, depending upon the distance involved, you’ll first need to take a look at cables. Then of course you’ll also need to have the power capable to send signals across the vast expanse of inter-DC communication. This involves the powerful optic transceivers that you will need, the farther you wish to go. Typically these are -LR, -ER, and of course CWDM or DWDM (often referred to generically as xWDM) optics.

We’ll be looking at these in the next article, but for now let’s take an example.

Suppose you have an cable that has the capability of sending data 100km, but an optic that only has reliability up to 30m, then obviously you have a physical connection that will only be able to go 30m!

Think of it this way in this rough, imperfect analogy: the road can go on for 5000 miles, but your car only has enough gas for 300. That means unless you make some serious modifications to your car, you’re not going to be able to make it the whole way on one tank of gas.

Now, the differences between optics and cables and what they’re capable of involves some pretty intense math and physics and, as a result, falls outside the scope of this article. As a result, we’ll be exploring that subject a bit more in the next one.

Fill the Pipe

In reality, this is the part that gets a lot of people confused, and with good reason. There are a number of factors that contribute to making sure that the pipe is filled, and done so correctly and efficiently. This is true whether you’re talking about Fibre Channel or IP-based technologies.

For the moment, let us assume that we have the appropriate optics and cabling, so that we have the power to throw data across great distances physically. So, taking it from a very rudimentary perspective, you have information here, and you want to get it there:

Fibre Channel

On a switch (or line card on a switch) on either end, there needs to be enough memory on either side of the link to be able to handle the transmission and receipt of the data. Fibre Channel requires that the frames in an exchange arrive in the same order as they are transmitted, but there are several ways to accomplish this.

When using Fibre Channel, we use a source-based mechanism for filling the pipe, called Buffer Credits (also known as Buffer-to-Buffer Credits). Each Fibre Channel frame takes up a credit to be sent. We’ll get into this in a bit more detail in a subsequent article, but for now what typically happens is that on a switch where multiple ports share an ASIC (where the memory buffers are located), we’ll shut down all but one port and donate more buffer credits to that port.

Here’s why:

Having a pipe like the one shown in the figure above, you’ll see that the longer the link, the more empty it becomes. These links are expensive, however, and we want to fill them up as much as possible.

Because each port has credits which determine how many frames we can send in one go, we can donate some of the buffer credits to a single port when we want to look at longer distance. When we donate buffers to an extension port we can make better use of the link:

FCoE

FCoE, on the other hand, has a receiver-based mechanism for ensuring lossless, called Priority Flow Control. Again, we’ll be looking at this in greater detail later, but for now the important bit to know is that for FCoE it’s the receiving switch that controls the flow of data.

When the memory coffers fill up on the receiving switch, a PAUSE is sent back to the sending switch. This means that the receiving ASIC needs to have enough memory to be able to handle all the frames inside the pipe and all the frames that will go into the pipe during the time it takes to send a PAUSE back.

For this reason it’s important to note that while the Fibre Channel protocol is maintained during the process, the Ethernet Link Layer is different, so it’s not as simple as “just” using FC techniques for FCoE distance.

FCIP

A third method of filling the pipe is the Fibre Channel over IP (FCIP) protocol that is commonly used. When we get into FCIP in detail, we’ll have to look at how TCP windowing and transmission sizes can affect and help facilitate long-distance Fibre Channel communication.

If this seems to be a little light on the technical details, don’t panic. The remaining articles are going to be a bit more involved and give you some of the nitties and gritties about what you need to know about extending your SAN across distances. This is really just the starting point for understanding the basic requirements that we have to keep in mind when exploring our distance options.

The next article will discuss something that is often overlooked when thinking about distance: The Physical Layer (Optics and Cabling).