Given the tremendous interest in VXLAN with MP-BGP based EVPN Control-Plane (short EVPN) at Cisco Live in Milan, I decided to write a “short” technology brief blog post on this topic.
VXLAN (IETF RFC7348) has been designed to solve specific problems faced with Classical Ethernet for a few decades now. By introducing an abstraction through encapsulation, VXLAN has become the de-facto standard overlay of choice in the industry. Chief among the advantages provided by VXLAN; extension of the todays limited VLAN space and the increase in the scalability provided for Layer-2 Domains.
Extended Namespace – The available VLAN space from the IEEE 802.1Q encapsulation perspective is limited to a 12-bit field, which provides 4096 VLANs or segments. By encapsulating the original Ethernet frame with a VXLAN header, the newly introduced addressing field offers 24-bits, thereby providing a much larger namespace with up to 16 Million Virtual Network Identifiers (VNIs) or segments.
While the VXLAN VNI allows unique identification of a large number of tenant segments which is especially useful in high-scale multi-tenant deployments, the problems and requirements of large Layer-2 Domains are not sufficiently addressed. However, significant improvements in the following areas have been achieved:
No dependency on Spanning-Tree protocol by leveraging Layer-3 routing protocols
Layer-3 routing with Equal Cost Multi-Path (ECMP) allows all available links to be used
Scalability, convergence, and resiliency of a Layer-3 network
Isolation of Broadcast and Failure Domains
IETF RFC7348 – VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks
Scalable Layer-2 Domains
The abstraction by using a VXLAN-like overlay does not inherently change the Flood & Learn behavior introduced by Ethernet. In typical deployments of VXLAN, BUM (Broadcast, Unicast, Multicast) traffic is forwarded via layer-3 multicast in the underlay that in turn aids in the learning process so that subsequent traffic need not be subjected to this “flood” semantic. A control-plane is required to minimize the flood behavior and proactively distribute End-Host information to participating entities (typically called Virtual Tunnel End Points aka VTEPs) in the same segment – learning.
Control-plane protocols are mostly employed in the layer-3 routing space where predominantly IP prefix information is exchanged. Over the past years, some of the well-known routing protocols have been extended to also learn and exchange Layer-2 MAC addresses. An early technology adoption with MAC addresses in a routing-protocol was Cisco’s OTV (Overlay Transport Virtualization), which employed IS-IS to significantly reduce flooding across Data Center Interconnects (DCI).
Multi-Protocol BGP (MP-BGP) introduced a new Network Layer Reachability Information (NLRI) to carry both, Layer-2 MAC and Layer-3 IP information at the same time. By having the combined set of MAC and IP information available for forwarding decisions, optimized routing and switching within a network becomes feasible and the need for flood to do learning get minimized or even eliminated. This extension that allows BGP to transport Layer-2 MAC and Layer-3 IP information is called EVPN – Ethernet Virtual Private Network.
Integrated Route and Bridge (IRB) – VXLAN-EVPN offers significant advantages in Overlay networking by optimizing forwarding decision within the network based on Layer-2 MAC as well as Layer-3 IP information. The decision on forwarding via routing or switching can be done as close as possible to the End-Host, on any given Leaf/ToR (Top-of-Rack) Switch. The Leaf Switch provides the Distributed Anycast Gateway for routing, which acts completely stateless and does not require the exchange of protocol signalization for election or failover decision. All the reachability information available within the BGP control-plane is sufficient to provide the gateway service. The Distributed Anycast Gateway also provides integrated routing and bridging (IRB) decision at the Leaf Switch, which can be extended across a significant number of nodes. All the Leaf Switches host active default gateways for their respective configured subnets; the well known semantic of First Hop Routing Protocols (FHRP) with active/standby does not apply anymore.
Summary – The advantages provided by a VXLAN-EVPN solution are briefly summarized as follows:
Standards based Overlay (VXLAN) with Standards based Control-Plane (BGP)
Layer-2 MAC and Layer-3 IP information distribution by Control-Plane (BGP)
Forwarding decision based on Control-Plane (minimizes flooding)
Integrated Routing/Bridging (IRB) for Optimized Forwarding in the Overlay
Leverages Layer-3 ECMP – all links forwarding – in the Underlay
Significantly larger Name-Space in the Overlay (16M segments)
Integration of Physical and Virtual Networks with Hybrid Overlays
It facilitates Software-Defined-Networking (SDN)
Simply formulated, VXLAN-EVPN provides a standards-based Overlay that supports Segmentation, Host Mobility, and High Scale.
VXLAN-EVPN is available on Nexus 9300 (NX-OS 7.0) with Nexus 7000/7700 (F3 linecards) to follow in the upcoming major release. Additional Data Center Switching platforms, like the Nexus 5600, will follow shortly after.
A detailed whitepaper on this topic is available on Cisco.com. In addition, VXLAN-EVPN was featured during the following Cisco Live! Sessions.
I’d like to give you an inside look at our Allen Data Center and go over how Cisco IT is adopting new technologies and capabilities while at the same time running the business. I’ll answer your top of mind questions and cover topics such as: Read More »
It’s an exciting time in to be in our industry, especially as we witness how technology continues to reshape how we connect and communicate through a myriad of applications and devices not only within our own companies, but also with our customers and partners.
At the epicenter of this technological transformation, we continue to find that the network is what ultimately enables these applications and their users to connect. We also quickly find that if this same network is not ready to deal with the ever increasing influx of devices, new applications with varying traffic patterns, and 24 x 7 access from pretty much anywhere, it can quickly turn into an IT departments nightmare.
It is exactly to deal with these new types of requirements that the award-winning Nexus 9000 Series (made up of both the Nexus 9500 and Nexus 9300 portfolios) was introduced into the market almost 11 months ago. Now, over 600 customers have purchased this new switching family and are experiencing the positive impact that having a high performing, scalable, programmable, and resilient data center network has on application performance and overall user quality of experience in both traditional and Application Centric Infrastructure (ACI) architectures.
Today we are happy to announce the addition of three new switches into the Nexus 9300 Series as well as a 6-port 40Gbps module to deliver more flexibility and form factor options to meet different architectural needs. The new products are:
Cisco Nexus 9372TX: 1-rack-unit switch supporting 1.44 Tbps of bandwidth across 48 fixed 1/10-Gbps BASE-T ports and 6 fixed 40-Gbps QSFP+ ports
Cisco Nexus 9372PX: 1-rack-unit switch supporting 1.44 Tbps of bandwidth across 48 fixed 1/10-Gbps SFP+ ports and 6 fixed 40-Gbps QSFP+ ports
Cisco Nexus 9332PQ: 1-rack-unit switch supporting 2.56 Tbps of bandwidth across 32 x 40Gbps QSFP+ ports
6-port 40 Gigabit Ethernet Module for the Nexus 93128TX, 9396TX , and 9396PX for connectivity options to meet your needs
These new switches deliver high performance, additional buffers, as well as support for VXLAN routing in a compact form factor. In addition to this, support for the Cisco Nexus 2000 Fabric Extenders has also been added to the Nexus 9300 portfolio. So if you already had Fabric Extenders in your data center or are looking for a scalable and operationally simplified architecture – you can now have the best of both worlds.
But it doesn’t end there – in case you missed it, Cisco recently announced the availability of the Application Policy Infrastructure Controller (APIC) making the creation of a more simplified, robust, application-centric infrastructure a reality with the Nexus 9000 Series as the network foundation. You can read more about it here – in Craig Huitema’s blog, which outlines not only new products on the nexus 9000 series including 100Gbps on the Nexus 9500, but also how we have simplified the introduction of the Nexus 9000 and ACI into data centers through different ACI starter kits and bundles. In addition, for those of you that want to deploy the Nexus 7000 in combination with the Nexus 9300s, new bundles that bring together the Nexus 7000 and Nexus 9300 are also available.
As you can see, we continue to deliver the products and architectural options that will allow data centers of all sizes to address increasing and changing application requirements. Between the Nexus 9300 and Nexus 9500 portfolios and their ability to be deployed into 3-tier, spine/leaf, or ACI architectures, customers can benefit from more connectivity options and a diverse set of form factors to meet varying data center needs. I invite you to learn more about the Nexus 9000 Series at www.cisco.com/go/nexus9000.
Welcome to Episode 2 of #CiscoChampion Radio, a podcast series by and for Cisco Champions as technologists, hosted by Cisco’s Amy Lewis (@CommsNinja). This week we’re talking about Application Centric Infrastructure (ACI).
Cisco Champion: Colin Lynch (@UCSguru)
Cisco Subject Matter Expert: Joe Onisick (@jonisick)
How ACI lets you manage a network cohesively instead of box-by-box
What a network looks like in ACI mode vs. stand-alone mode
How ACI works with network protocols like spanning-tree and TRILL
Upgrading the Nexus 9000 Series to ACI
When ACI makes sense for your business
As IT departments move to private cloud offerings, DevOps methodologies, and continuous integration capabilities, many segments of the data center market have a strong need for more open, programmable, and application-led networks. In these fully automated environments, network automation for infrastructure as a service (IaaS) or applications on demand is becoming essential. As discussed in a recent blog postby Ravi Balakrishnan, the Cisco Nexus 9000 offers the industry’s 1st open and extensible application policy model helping businesses increase agility, flexibility, and scalability and automate repetitive manual tasks, reducing the time to deployment and easing maintenance tasks.
A recently-issued Lippis Report provides validation that the Cisco Nexus 9000 product line offers the most comprehensive open programming tools and functions available that can either be leveraged independently, or put to work in unison with other platform capabilities. The report found that the benefits of Cisco Nexus 9000 programming environment include investment protection and improved business agility through support of open protocols, APIs and standards that leverage customers’ existing networking, services including security, physical and virtual compute, and storage assets and accelerate network application deployment times to minutes improving business agility through centralized management.
Cisco 9000 programmability enables use cases across the whole IT delivery chain in terms of being able to orchestrate and automate provisioning of network infrastructure. Applications now have special, real-time access to network buffers, congestion and state information, so that they can actually make better choices and decisions on how they’re delivering services to end-users. In addition, troubleshooting can be automated through applications having much deeper visibility into the network.
The specific use cases for Cisco NX-OS API enhancements span data center network engineers and experienced DevOps personnel in cloud and large enterprise IT organization. For network engineers, NX-OS APIs can simplify and automate common network infrastructure provisioning challenges as well as offer automated troubleshooting through enhanced network visibility.
DevOps personnel may leverage NX-OS APIs and automation tools to create their own custom scripts and leverage the NX-API into other tools with which they are already familiar to customize network device data and use it in the way that’s important for them to either deliver competitive business value or to reduce OpEx through automation.
Cisco 9000 Programmability Highlights
The Cisco NX-OS enhancements for the Cisco Nexus 9000 Series supports numerous capabilities that aid automation and orchestration including providing investment protection through the support of new automation capabilities in the future. Centralized, fine-grained access to Cisco 9000 networking resources is enabled through support for XML, JSON, representational state transfer (REST), remote procedure call (RPC), NetConf, Python scripting, Bash and Broadcom chip-level shell access, and Linux containers for development of custom applications. These APIs have full read and write access to the Cisco 9000 platform, providing programmability, automation, and system access. Cisco-NX-OS also supports APIs enabling rapid integration with existing management and orchestration frameworks. These include OpenStack interfaces to provide Cisco policy consistency across physical, virtual, and cloud environments.