Digging Deeper in VXLAN, Pt 3, More FAQs

October 3, 2011 - 4 Comments

So, here is our final installment–we are wrapping up with some of the more common questions were are seeing.  In you missed the earlier posts, be sure to check out Part 1 and Part 2. I also have a couple of earlier posts introducing VXLAN and answering some of the initial questions.

So, onto the FAQs…

How do VXLANs relate to VN-tags ?

Some people have asked what is the relationship between VN-tags (aka 802.1Qbh) and VXLANs. Does one rule out the other ? The answer is a definite ‘no’. The VN-tag exists on the link between a VM and the virtual switch (or ‘VM-switch’). If such a switch has support for VXLANs then the VMs in question can get connectivity through VXLANs. A packet will never need to have both at the same time. In the context of Cisco products, VM-FEX technology remains complementary to VXLANs.

So now we can migrate VMs across subnets ?

There is also some confusion over what implications VXLANs have for VM mobility. Claims such as “VXLANs permit mobility across L3 boundaries” have been taken to mean different things.

We want to make clear that VMs connected to VXLANs do not need to change their own IP addresses as a consequence of this technology. Essentially, VMs connected to a VXLAN remain unaware that they are on a VXLAN – just as they are usually unaware of running on VLANs. It is up to the underlying network to ensure connectivity between such VMs and provide layer-2 semantics such as mac-layer broadcast and unicast traffic. As a consequence, any mobility event – live or otherwise – has no effect on the internals of the VM.

At the same time, since the native Ethernet frames are carried over an IP encapsulation, the tunnel endpoints themselves do not need to be on the same VLAN or IP subnet in order to ensure connectivity of the VXLAN. This creates the potential for a VM on a certain VXLAN to move between hosts which are themselves on different subnets. It is important however not to interpret this to mean live VM migration is now immediately possible across subnets, since other factors can get in the way. As an example, live VM migration itself requires transfer of VM data between two hosts. This transfer may not be possible or officially supported across subnets. All that VXLANs ensure is connectivity to the same perceived layer 2 broadcast network regardless of which host it is on (assuming of course that the network supports VXLANs) and regardless of which subnet the host connects to. However, VXLANs do not, by themselves, circumvent other impediments to live VM migration – such as the transfer issue mentioned above.

What about routing across VXLANs ?

So, you are thinking “This is all well and good to interconnect VMs at layer-2 in a scalable way, but how do they talk to the real world of corporate networks and the internet” ? In other words, who routes between VXLANs ? Or between VXLANs and VLANs or VXLANs and the global internet ?

The answer to this will evolve over time just as it did with VLAN technology. If a router is ignorant of 802.1Q tagging it cannot route across VLANs unless someone else terminates VLAN tagging on its behalf. For instance an 802.1Q-capable L2 switch can strip the tag and forward native Ethernet frames to/from the router. The router would then only support one “VLAN interface” on each physical interface.

With VXLANs, too, VXLAN capable devices could take the responsibility for stripping the encapsulation before forwarding the packet to the router for routing. If VXLAN functionality remains confined to virtual switches, the router, too, will need to be a virtual router i.e. routing software running inside a VM. As and when non-virtual physical switches support the VXLAN format, real physical router boxes can connect to them. Of course, as in the VLAN case, this will limit the number of routable VXLAN interfaces on the router. A better solution would be for the router itself to encap/decap the VXLAN tunneled packets so it can support a large number of VXLAN interfaces on the same physical interface.

One intermediate step between using purely virtual routers and having physical routers support the VXLAN encapsulation would be for the L2 network to permit bridging between VXLANs and VLANs. This will allow both physical and virtual devices – whether routers or other nodes – to connect into VXLANs without requiring an upgrade to their software. Such a bridging functionality is defined in the proposed draft standard for just such purposes.

In many cloud provider environments, tenants may be able to directly create VXLANs within their portion of the public data center (sometimes called an Organization’s Virtual Data Center). The IP addressing of the tenant-administered VMs on these VXLANs will in general not be coordinated across different tenants. This makes NAT a very desirable feature on the router or routers directly attached to such client administered VXLANs. Ditto for VPN connectivity to the tenant’s own network.

Are VXLANs secure ?

With proper precautions they can be just as secure as regular networks. It is true however, that there is a greater risk of attacks and users must understand this and take measures to guard against it. In the worst case, an attacker can inject himself into a VXLAN by sending IP-encapsulated packets from anywhere. Of course this requires access to the IP network. A first line of defense is to have a perimeter firewall that denies IP traffic with the VXLAN encapsulation from the outside.

This does not prevent attacks from the inside. For that users would need to control access at internal routers to ensure that only authorized tunnel endpoints can inject packets into VXLAN tunnels. This can be done statically (knowing the physical topology of the network) or by employing additional IP security mechanisms that guarantee encryption and/or authentication.

Rather than re-invent this particular wheel, the VXLAN draft lets users make use of existing methods to secure VXLAN tunneled traffic, while pointing out where the risks lie.

What about network services ?

Since VXLANs are expected to be deployed in hosted environments people naturally want to know how to enable network services (firewalls, IPS, load balancing, WAN optimization) for VXLANs. The answer to this is pretty much the same as for routers. Either the services need to be enabled in endpoints that can be attached to VXLANs (i.e. virtual in the immediate future), or these services need to become VXLAN aware or someone needs to perform a bridging function between VXLANs and whatever it is that the services understand (physical interfaces, VLANs etc.)

So, if you are intrigued by VXLANs, you may want to ping you Cisco account team–we will be kicking off a closed beta soon.

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.


  1. Hi,

    so can the NX1000V currently do the bridging function from 802.1q VLAN and a VXLAN?

    …meaning can the NX1000V have local virtual ports (VM ports), uplink ports (inside an 802.1q VLAN) and remote VXLAN (remote) ports on the same bridge domain?

    I ask this because as the upstream NX5K and 7K cannot support VXLAN, one would need to do the L3 on these with traffic delivered through normal VLAN tagging interfaces.



  2. Denton, this is a good point…and actually what our Cisco implementation does!

  3. A suggestion:

    The source port of the Outer UDP header is computed from a hash of the Inner headers. I’d suggest always setting the most significant bit, so the port will be >32768. People are going to try running VXLAN through firewalls across a VPN. If the Inner headers hash to a well-known port like 53, it will cause them some grief in having to turn off inspection of packets from VTEP IPs.
    15 bits is still a reasonable amount of variance for LACP/ECMP distribution.

    More VXLAN thoughts: