For the third year in a row, Cisco participated in the VXLAN BGP EVPN interoperability testing at the European Advanced Networking Test Center (EANTC).
The interoperability showcase and test results are featured in the EANTC white paper as a part of the MPLS + SDN + NFV World Congress. This year, a total of seven vendors participated in the VXLAN BGP EVPN interoperability testing, showing significant industry adoption year over year.
VXLAN BGP EVPN – 2 RFCs, 3 Drafts, 7 “Options”
As with many early-stage technologies, VXLAN BGP EVPN has seen many proposed implementation options and variations. The latest solution is comprised of many pieces of the standard bodies. For example, VXLAN data-plane itself is covered in RFC 7348 while the overarching definition of the BGP EVPN control-plane is covered in RFC 7432.
In order to define the operational models for VXLAN BGP EVPN in more detail, additional drafts have been issued. While the EVPN-overlay draft (draft-ietf-bess-evpn-overlay) specifies the control-plane for Layer-2 operation, the routing functions are separated into the first-hop routing (draft-ietf-bess-evpn-inter-subnet-forwarding) and the IP subnet routing (draft-ietf-bess-evpn-prefix-advertisement) drafts. This amounts to 2 RFCs and 3 IETF drafts.
The various RFCs and drafts provide different implementation options. There are two to three options per draft, but paired with the amount of guiding documents, the permutation becomes quite large. In order to provide some additional context to the testing results, below explains the different implementation options and what they entail.
Layer-2 Service Interface
In “draft-ietf-bess-evpn-overlay”, there are two modes of operation describing how the EVPN Instances or EVIs are configured and how the associated information is carried over the BGP control-plane. EVPN Instance or EVI are Virtual Private Networks (VPNs) in the terminology of EVPN.
The first option, called “VLAN-based”, is described as “single broadcast domain per EVI”. The EVI is the equivalent of a VPN in EVPN terminology. In this option, the tenant VLAN is mapped to a single EVI where the entire routing-policy is applied (Route-Target in BGP terminology). In this 1:1 mapping approach, the single broadcast domain is represent with a VLAN or a VNI respectively. The VLAN/VNI is associated with an EVI which provides the most granular control for importing routes, specifically the MAC addresses.
The second option, termed “VLAN-aware,” is represented in the section “multiple broadcast domain per EVI”. For this option, multiple VLAN/VNI combinations are bundled into a single EVI. This bundling requires slightly less configuration, as multiple VLANs/VNI only require a single route-target. However, when automated configuration options are considered, this is no longer an advantage. In fact, the loss of granular control of importing and injecting routes on a per VNI basis now becomes a disadvantage. In both VLAN-based and VLAN-aware options, the data-plane is similarly populated where a single VNI identifies the local MAC-VRF. Similar as an IP-VRF for Layer-3 domains, the MAC-VRF defines the logical boundary but this time for Layer-2 domains.
Cisco has followed the VLAN-based approach. This approach, coupled with the auto-derivation of BGP EVPN Route-Targets and Route-Distinguishers, provides the most granular option to populate hardware tables appropriately.
Today, given the obvious advantages, all vendors participating in the EANTC interoperability testing have converged to the VLAN-based approach, allowing broad interoperability testing between vendors.
First-Hop Gateway (Integrated Route and Bridge or IRB)
With the goal of driving benefits from Layer-2 forwarding and inter-subnet forwarding with EVPN, a first-hop gateway option had to be defined based on the approach of Integrated Routing and Bridging (IRB). Just like Layer-2, there are different use-cases with Layer-3, and as a result two different modes of operation were defined in “draft-ietf-bess-evpn-inter-subnet-forwarding”, namely, Symmetric and Asymmetric IRB.
Asymmetric IRB follows a more traditional approach, where the first-hop gateway for the End-Points performs the routing operation only at the ingress. This results in a bridge-route-bridge operation or a similar approach as employed for Inter-VLAN routing. With Inter-VLAN routing followed by bridging to the respective destination through the Layer-2 VNI (L2VNI), the device hosting the first-hop gateway function is required to have all possible destination MAC/IP binding information. This implies that the MAC-IP binding information of all local as well as remote End-points needs to be known at the first-hop gateway device which is inherently a scaling limitation.
Symmetric IRB uses a bridge-route-route-bridge approach. Whenever there is a routing operation done at the ingress, there is a symmetric routing operation performed at the egress. Routed traffic from ingress to egress is forwarded via a transit segment, defined on a per-VRF basis and termed the Layer-3 VNI or L3VNI. For all routed traffic that goes in any direction, the L3VNI is stamped in the VXLAN header. This is different from the Asymmetric IRB scenario where, depending on the destination, the routed traffic will carry the L2VNI associated with the destination subnet. The symmetry provided by Symmetric IRB ensures that only MAC/IP bindings associated with locally attached End-Points are required at the gateway, reducing both the required software and hardware state.
With the aim for a distributed first-hop gateway approach paired with optimal scale and no hair-pinning, Cisco implemented the Symmetric IRB approach. Symmetric IRB provides the most scalable approach by not creating a pollution of MAC/IP adjacency information across all the devices performing first-hop gateway function.
While there was initially a wide adoption of Asymmetric IRB among the original authoring vendors other than Cisco, most of the newer entrants have implemented the more scalable Symmetric IRB approach.
IP Subnet Routing (IP-VRF)
Most vendors agreed on how IP Subnet routing can be done using two primary options, along with a third combination of the two. In “draft-ietf-bess-evpn-prefix-advertisement”, all three options are defined with very creative names. The “interface-less” approach embeds the next-hop’s MAC address (RMAC) as part of the same BGP NLRI where the IP Subnet prefix is sent. With this approach, all information necessary for routing is embedded in a single BGP update.
This is different from the other two “interface-full” approaches, where for every next-hop, an additional BGP advertisement is created to provide the next-hop’s MAC address. The reason for this is in the primary implementation of the “interface-full” approach, the next-hop is numbered, and for each of these next-hop IP addresses, the respective IP-to-MAC mapping is required. In one flavor of the ‘interface-full’ approach, the next-hop is numbered and it is necessary to send IP-to-MAC mapping for each next-hop IP address. Even with the unnumbered option for “interface-full,” the additional BGP prefix is still required for serving the recursive look-up on the next-hop.
Cisco implemented the more natural way “interface-less” operates, with no additional MAC route advertisement required on top of the IP subnet route. Cisco also adopted the variation of the “interface-full” option based on the unnumbered making. Cisco is the only vendor with both unnumbered based approaches for IP subnet routing in BGP EVPN implemented and with the ability to host both options at the same time. Cisco can also route between “interface-less” and unnumbered “interface-full” based implementations.
Convergence to a Common Implementation
As you can see, the various VXLAN BGP EVPN options presented in the IETF drafts can result in quite a few permutations. This, together with the respective vendor decision regarding choice of implementation, made interoperability quite difficult in the early days.
Over the last year or so, there has been a convergence to a specific set of options. This convergence has reduced the complexity for users. The original co-authoring vendors along with the late-comers are fairly aligned with the following common set of options for EVPN:
- For Layer-2 Service Interface: VLAN-based
- For First-Hop Routing: Symmetric-IRB
- For IP Subnet Routing: Interface-Less
The test results from the EANTC Interoperability Showcase of 2017 highlight the convergence to a common implementation set, mirroring the functionality Cisco currently supports and has been shipping for the last 2 years for VXLAN BGP EVPN.
Cisco continues to drive enhancements on top of its VXLAN BGP EVPN solution, always with industry standards and openness in mind. Together, the BGP EVPN control-plane and data-plane agnosticism has helped to drive new approaches and data-planes like Segment Routing. The single control-plane approach across multiple domains makes intra Data Center use-cases very attractive, while also enabling seamless elasticity to other domains.
MPLS + SDN + NFV World Congress Public Multi-Vendor Interoperability Test 2017
Interoperability Showcase 2017 – Whitepaper
Very neat write-up!
a very complex scheme, good to see Cisco leading from the front.
cool blog! we’ve also being working with the BGP EVPN control plane outlined in RFC 7432 to bring SR to the host.