With the adoption of overlay networks as the standard deployment for multi-tenant network, Layer2 over Layer3 protocols have been the favorite among network engineers. One of the Layer2 over Layer3 (or Layer2 over UDP) protocols adopted by the industry is VXLAN. Now, as with any other overlay network protocol, its scalability is tied into how well it can handle the Broadcast, Unknown unicast and Multicast (BUM). That is where the evolution of VXLAN control plane comes into play.
The standard does not define a “standard” control plane for VXLAN. There are several drafts describing the use of different control planes. The most commonly use VXLAN control plane is multicast. It is implemented and supported by multiple vendors and it is even natively supported in server OS like the Linux Kernel.
This post tries to summarize the three (3) control planes currently supported by some of the Cisco NX-OS/IOS-XR. My focus is more towards the Nexus 7k, Nexus 9k, Nexus 1k and CSR1000v.
Each control plane may have a series of caveats in their own, but those are not covered by this blog entry. Let’s start with some VXLAN definitions:
(1) VXLAN Tunnel Endpoint (VTEP): Map tenants’ end devices to VXLAN segments. Used to perform VXLAN encapsulation/de-encapsulation.
(2) Virtual Network Identifier (VNI): identify a VXLAN segment. It hast up to 224 IDs theoretically giving us 16,777,216 segments. (Valid VNI values are from 4096 to 16777215). Each segment can transport 802.1q-encapsulated packets, theoretically giving us 212 or 4096 VLANs over a single VNI.
(3) Network Virtualization Endpoint or Network Virtualization Edge (NVE): overlay interface configured in Cisco devices to define a VTEP
VXLAN with Multicast Control Plane
Read More »
Tags: #ciscochampion, Cisco Nexus 9000, CSR1000v, Nexus 1000, Nexus 7000, Nexus 9000, VXLAN
Given the tremendous interest in VXLAN with MP-BGP based EVPN Control-Plane (short EVPN) at Cisco Live in Milan, I decided to write a “short” technology brief blog post on this topic.
VXLAN (IETF RFC7348) has been designed to solve specific problems faced with Classical Ethernet for a few decades now. By introducing an abstraction through encapsulation, VXLAN has become the de-facto standard overlay of choice in the industry. Chief among the advantages provided by VXLAN; extension of the todays limited VLAN space and the increase in the scalability provided for Layer-2 Domains.
Extended Namespace – The available VLAN space from the IEEE 802.1Q encapsulation perspective is limited to a 12-bit field, which provides 4096 VLANs or segments. By encapsulating the original Ethernet frame with a VXLAN header, the newly introduced addressing field offers 24-bits, thereby providing a much larger namespace with up to 16 Million Virtual Network Identifiers (VNIs) or segments.
While the VXLAN VNI allows unique identification of a large number of tenant segments which is especially useful in high-scale multi-tenant deployments, the problems and requirements of large Layer-2 Domains are not sufficiently addressed. However, significant improvements in the following areas have been achieved:
- No dependency on Spanning-Tree protocol by leveraging Layer-3 routing protocols
- Layer-3 routing with Equal Cost Multi-Path (ECMP) allows all available links to be used
- Scalability, convergence, and resiliency of a Layer-3 network
- Isolation of Broadcast and Failure Domains
IETF RFC7348 – VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks
Scalable Layer-2 Domains
The abstraction by using a VXLAN-like overlay does not inherently change the Flood & Learn behavior introduced by Ethernet. In typical deployments of VXLAN, BUM (Broadcast, Unicast, Multicast) traffic is forwarded via layer-3 multicast in the underlay that in turn aids in the learning process so that subsequent traffic need not be subjected to this “flood” semantic. A control-plane is required to minimize the flood behavior and proactively distribute End-Host information to participating entities (typically called Virtual Tunnel End Points aka VTEPs) in the same segment – learning.
Control-plane protocols are mostly employed in the layer-3 routing space where predominantly IP prefix information is exchanged. Over the past years, some of the well-known routing protocols have been extended to also learn and exchange Layer-2 MAC addresses. An early technology adoption with MAC addresses in a routing-protocol was Cisco’s OTV (Overlay Transport Virtualization), which employed IS-IS to significantly reduce flooding across Data Center Interconnects (DCI).
Multi-Protocol BGP (MP-BGP) introduced a new Network Layer Reachability Information (NLRI) to carry both, Layer-2 MAC and Layer-3 IP information at the same time. By having the combined set of MAC and IP information available for forwarding decisions, optimized routing and switching within a network becomes feasible and the need for flood to do learning get minimized or even eliminated. This extension that allows BGP to transport Layer-2 MAC and Layer-3 IP information is called EVPN – Ethernet Virtual Private Network.
EVPN is documented in the following IETF drafts
Integrated Route and Bridge (IRB) – VXLAN-EVPN offers significant advantages in Overlay networking by optimizing forwarding decision within the network based on Layer-2 MAC as well as Layer-3 IP information. The decision on forwarding via routing or switching can be done as close as possible to the End-Host, on any given Leaf/ToR (Top-of-Rack) Switch. The Leaf Switch provides the Distributed Anycast Gateway for routing, which acts completely stateless and does not require the exchange of protocol signalization for election or failover decision. All the reachability information available within the BGP control-plane is sufficient to provide the gateway service. The Distributed Anycast Gateway also provides integrated routing and bridging (IRB) decision at the Leaf Switch, which can be extended across a significant number of nodes. All the Leaf Switches host active default gateways for their respective configured subnets; the well known semantic of First Hop Routing Protocols (FHRP) with active/standby does not apply anymore.
Summary – The advantages provided by a VXLAN-EVPN solution are briefly summarized as follows:
- Standards based Overlay (VXLAN) with Standards based Control-Plane (BGP)
- Layer-2 MAC and Layer-3 IP information distribution by Control-Plane (BGP)
- Forwarding decision based on Control-Plane (minimizes flooding)
- Integrated Routing/Bridging (IRB) for Optimized Forwarding in the Overlay
- Leverages Layer-3 ECMP – all links forwarding – in the Underlay
- Significantly larger Name-Space in the Overlay (16M segments)
- Integration of Physical and Virtual Networks with Hybrid Overlays
- It facilitates Software-Defined-Networking (SDN)
Simply formulated, VXLAN-EVPN provides a standards-based Overlay that supports Segmentation, Host Mobility, and High Scale.
VXLAN-EVPN is available on Nexus 9300 (NX-OS 7.0) with Nexus 7000/7700 (F3 linecards) to follow in the upcoming major release. Additional Data Center Switching platforms, like the Nexus 5600, will follow shortly after.
A detailed whitepaper on this topic is available on Cisco.com. In addition, VXLAN-EVPN was featured during the following Cisco Live! Sessions.
Do you have appetite for more? Post a comment, tweet about it and have the conversation going … Thanks for reading and Happy Networking!
Tags: #CLEUR, Cisco, cisco live, Cisco Nexus, Cisco Nexus 9000, data center, EVPN, ietf, network, nexus, rfc7348, SDN, VXLAN
The Cisco Nexus 1000V has been supported in VMware vSphere hypervisor since 4.0 release (August 2009) up to the current vSphere release 5.5 update 2. We are happy to announce that the Nexus 1000V will continue to be supported in the latest vSphere 6 release which VMware recently announced. Customers who are currently running Nexus 1000V will be able to upgrade to the vSphere 6 release and the new vSphere 6 customers will have the Nexus 1000V as part of their choices for virtual networking.
Cisco is fully committed to support the Nexus 1000V product for our 10,000+ Advanced Edition customers and the thousands more using the Essential Edition software in all future releases of VMware vSphere. Cisco has a significant virtual switching R&D investment with hundreds of engineers dedicated to the Nexus 1000V platform. The Nexus 1000V has been the industry’s leading virtual switching platform with innovations on VXLAN (industry’s first shipping VXLAN platform), and distributed zone firewall (via Virtual Security Gateway released in Jan 2011).
The Nexus 1000V also continues to be the industry’s only multi-hypervisor virtual switching solution that delivers enterprise class functionality and features across vSphere, Hyper-V and KVM.
In the last major release of the Nexus 1000V for vSphere, version 3.1 (August 2014) we added significant scaling and security features and we continue to provide subsequent updates (December 2014) with the next release planned for March 2015. The recently released capabilities include:
- Increased scale per Nexus 1000V:
- 250 hosts
- 10,000 virtual ports
- 1,000 virtual ports per host
- 6,000 VXLAN segments with ability to scale out via BGP
- Increased security and visibility
- Seamless security policy from campus and WAN to datacenter with Cisco TrustSec tagging/enforcement capabilities
- Distributed port-security for scalable anti-spoofing deployment
- Enhanced L2 security and loop prevention with BPDU Guard
- Protection against broadcast storms and or attacks with Storm control
- Scalable flow accounting and statistics with Distributed Netflow
- Ease of management via Virtual Switch Update Manager (VSUM) – a vSphere web-client plug-in
One of the common questions coming from our customers is whether VMware is still re-selling and supporting the Nexus 1000V via VMware support?
VMware has decided to no longer offer Nexus 1000V through VMware sales or sell support for the Nexus 1000V through the VMware support organization as of Feb 2nd 2015. We want to reiterate that this has NO IMPACT on the availability and associated support from Cisco for the Nexus 1000V running in a vSphere environment. Cisco will continue to sell Nexus 1000V and offer support contracts. Cisco encourages customers who are currently using VMware support for the Nexus 1000V to migrate their support contracts to Cisco by contacting their local Cisco Sales team to aide in this transition.
For questions or help, please reach out firstname.lastname@example.org
Tags: ACI, Cisco Nexus, Cisco UCS, Nexus1000V, VMware, VMware vSphere, vsg, vsphere 6, VXLAN
[Note: Register today for our upcoming live ACI webcast: “Is Your Data Center Ready for the Application Economy”, January 13, 2015, 9 AM PT, Noon ET, featuring ACI customers and several key ACI technology partners.]
At the most recent Gartner Data Center Conference in Las Vegas, after some insightful discussions with customers and analysts, we came up with a great demo idea and proof point that highlights a key feature in our Application Centric Infrastructure (ACI) platform. This particular demo centers on the unique visibility of the ACI Fabric to faults in the underlying physical network.
Joe Onisick, Principal Engineer in the ACI team at Cisco, compares this ability in ACI to SDN technologies that employ only virtual overlay networks in the following video. With overlay networks, such as a VXLAN tunnel, the resulting virtual network (and all the management and analytics tools) has a much harder time isolating faults within the physical infrastructure. The overlay is designed to “tunnel” through the physical network, simplifying and obscuring the physical topology and issues with any specific network node. Before going much further, I’ll let Joe provide the details in this quick, 3 minute video:
Read More »
Tags: ACI, APIC, application centric infrastructure, SDN, VXLAN
There has been some seismic activity happening in Bay Area and the epicenter for all Virtual Networking shifts is right here at Cisco HQ in San Jose. (Our sympathies go to all those affected by the real earthquake further to the north.) At Cisco, it’s all about the applications and the shift to dynamic network virtualization. Cisco pioneered virtual networking with Nexus 1000V virtual switch and recently incorporated it in the application aware Application Virtual Switch (AVS), for Cisco ACI-enabled networks. Cisco is excited to announce the availability of Nexus 1000 Release 3.1 of Nexus1000V for vSphere (available for download here). We are showing the upcoming generation of the virtual switch at VMworld in San Francisco this week.
Nexus1000V is the edge switch for virtual environments, bringing the network edge right up to the virtual machine, and connecting virtual ports to the physical network and beyond. The Nexus 1000V is the foundation for our virtual network overlay portfolio, including all of our virtual L4-7 application and security services, our cloud orchestration software, VXLANs and more. It is also at the heart of AVS, a purpose-built, hypervisor-resident virtual network edge switch designed for the Application Centric Infrastructure.
Release 3.1 is a new major release enabling enterprise and cloud provider customers running the vSphere hypervisor to leverage the distributed virtual firewall VSG, expand VXLAN footprint in the datacenter, improve secure isolation thru Cisco TrustSec and dramatically simplify updates through Cisco VSUM (Virtual Switch Update Manager). Most of the new features are value add to the Advanced Edition. New customers will need a Ver 3 specific license to use the full functionality of Ver 3. Existing customers with support contract are automatically entitled to free upgrade to Ver 3. AVS incorporates Nexus 1000V capabilities with consistent application policy enforcement for virtual workloads and unprecedented end-to-end visibility for applications in your data center.
Features of the new Nexus 1000V Release 3.1:
- Increased Scalability (Advanced Edition) – More than doubles the scale from the previous release. The virtual switch now supports 250 hosts/servers per switch with 10,000 ports per switch. In addition it supports 4094 active VLANs and 16 million VXLAN (6144 active VXLANs) per switch across 6144 port profiles.
- VXLAN control plane: BGP based control plane across multiple virtual switches provide expanded Layer 2 domain footprint that can potentially support nearly 40,000 VMs in a single domain
- Increased Resiliency – Supports headless Port bring up where Virtual Machines can be bought up on the host even if VEM is offline i.e. the VSM is not reachable by VEM. Both VSM headful and headless VM vMotion is supported.
- Cisco TrustSec 2.0 (Advanced Edition) – Continues to extended Cisco TrustSec solutions for network based segmentation of users and physical workloads, leveraging Security Group Tags (SGT) for defining security segments and SGACL support (Enforcement) and Native(in-line) SGT tagging.
- BPDU Guard – Keeps virtual network safe from misconfigured VLANs and strictly enforces VLAN boundries. It prevents Misconfigured VLAN Rogue devices from flooding the network
- Storm Control – Prevent network disruptions from a broadcast, multicast, or unknown-unicast traffic storm.
- Simplified Deployment, upgrade and visibility with Cisco VSUM – Cisco VSUM is a FREE virtual appliance that enables Server and Network administrators to Deploy, Upgrade and Monitor Nexus1000V and to Deploy and Upgrade Cisco AVS from within their vCenter web interface.
- Customer Experience - Here’s what one of our Beta customers, Josh Coen says about Cisco VSUM. Josh is a Principal Cloud Architect with Varrow and has been working in the IT industry since 1999, with a heavy focus on virtualization and storage since 2008.
Nexus 1000V has already reached the 10,000 customer milestone with some customers purchasing 1000+ CPU licenses. Nexus 1000V continues to provide the foundation for the most advanced virtual networks by supporting, 1) multiple hypervisor environments, such as VMware vSphere, Microsoft Hyper-V and Openstack KVM 2) the most extensive set of virtual network services, including ASA 1000V Cloud Firewall, distributed zone-based virtual firewall, vWAAS WAN optimization, the Cloud Services Router (CSR) 1000V, Cisco Prime Network Analysis Module (NAM) and advanced service insertion and chaining technology, vPath and 3) a true management control plane that provides greater policy and control features for richer networking functionality.
We’ll be showing a lot of these features this week. Come by our booth and check it out. If you are around #VMworld this week, give us a shout out on twitter using Cisco hash tag #ciscovmw. For those of you that can’t make it out to VMworld, listen to the review of these new features in Ver 3.1 in this webcast.
Tags: ASA 1000V, Cisco ONE, CSR 1000V, Hyper-V, NAM, Nexus 1000v, OpenStack, TrustSec, Virtual Network Management Center, Virtual Security Gateway, virtual switch, VMware, vmworld, VNMC, vPath, vsg, VSUM, VXLAN