Routed Optical Networking – It’s About the Architecture

Traditionally, the networking industry has deployed architectures that include two, or sometimes three, layers.  These layers are:

• An IP layer carrying IP/Ethernet/L3/L2 services
• A DWDM layer for efficiently transporting massive traffic loads
• In some cases, a discrete OTN layer for private line services

As technology advances, it’s important to step back and ask, “why are we doing this?” and to re-assess whether this architecture makes sense for the next 20 years.  It’s a fact of life that technology has a lifecycle that organizations form around. Over time, those organizations assume a life of their own, and we find ourselves architecting around the organizations that are in place.  Failing to recognize this, we constrain ourselves to innovations within the architecture as we know it, resulting in only incremental improvements in performance, capacity, and cost.  We know from many discussions with our customers that those incremental improvements will not resolve the stress they are feeling as capacity demands grow much faster than their revenues.   It’s time to step back, consider the technology innovations that are now available, and determine if a new architecture can yield significantly better outcomes for our customers, as opposed to continuing to incrementally innovate on the architectures of the past.

Why 3 Layers?

The IP layer continually evolves as IP services are the dominant traffic type in the network, having long ago surpassed TDM traffic that was the dominant traffic type of the past.  Some customers also deploy a discrete OTN layer for aggregating lower rate ethernet services or to deliver private line services.  As traffic continues to grow, DWDM is the preferred solution for efficiently aggregating traffic onto the existing fiber plant.  Each of these layers has its own planning, deployment processes, maintenance practices, and lifecycles.

Routers that were deployed in the 1990s-2000s were the most expensive resource in the network, so there was significant incentive to “bypass” router ports wherever possible.  This led to the need for a separate switching infrastructure at the DWDM layer – and ultimately ROADMs were developed to support flexible optical add/drop – all in support of bypassing “expensive” routers.  If a customer had a traffic need from A to Z, they would seek to provision an optical path from A to Z and avoid passing the signal through intermediate routers.  This made perfect sense, but what often happens is that the A to Z path is provisioned at the highest capacity the optical layer can support (e.g. 400G), while the true traffic demand is much less.  The result is that many express wavelengths in customer networks are lightly loaded – in fact, underutilized.

This was an important insight we discovered by modeling over 50 customer networks and looking at the actual IP traffic demands separate from the optical layer capacity that was in place.

Why Now?

Multiple technologies have pivoted, creating an opportunity to examine the network in a new way.  That, combined with several technological, industry, and commercial innovations, led us to conclude that the time is now to consider a new architecture for the next 20 years.

• Massively Scalable Silicon:  For years, router silicon was the limiting factor in how much capacity a single router could offer, and we built complex multi-bay configurations to increase the overall capacity at a single router node.  With recent advances in silicon, we are now seeing 1RU devices that support 12Tb/s, growing to 25Tb/s and beyond.  In a chassis-based configuration – Cisco offers 260Tb/s of capacity in less than one bay today – more capacity than most customers need at a single point in the network for the foreseeable future, but certainly scalable for those ultra high capacity needs.  With these advances in silicon, the cost per bit of a router port at 100G has been reduced significantly – to the point where the router port is no longer the most expensive resource in the network.  This raises the question of why we should still consider bypassing router ports.Our network modeling of actual customer networks suggests that, for most networks, it will be more cost effective to transport IP services “through” routers rather than “around” routers.  Why is that?  Most bypass wavelengths in the network today are lightly loaded, and even when we consider anticipated five-year demands, those wavelengths still remain lightly loaded.  In many cases, the utilization is well below 20%.  As the underlying traffic demand is far less than the deployed wavelength capacity, it makes sense to take advantage of IP aggregation to efficiently pack 400G wavelengths that are being carried between routers, as opposed to deploying lightly filled wavelengths expressing “around” routers.This is the true source of economic gain in a Routed Optical Network – leveraging the IP layer to efficiently aggregate services and drive higher utilization on the wavelengths traveling between routers.  In many cases, hop-to-hop will require more interfaces than an express path.  However, the services are being efficiently aggregated at the IP layer onto one path, rather than having many (lightly loaded) express wavelengths.  The result is far fewer wavelengths required in the network, and this is the primary source of CapEx savings.

• Pluggable Coherent Optics:  With advances in silicon and silicon photonics, we are now able to deliver 400G coherent wavelengths in a pluggable that may be plugged directly into the router.  Importantly, this pluggable has the same form factor as short reach (e.g. 2km) pluggables – so no custom line card is required in the router for coherent optics.  This was a significant barrier with previous attempts at integrating DWDM into the router – custom line cards were required for DWDM.  This complicated planning, deployment, sparing, and operations.With a common form factor, the customer has one line card used for all applications (short reach or coherent).  Further, these coherent pluggables are open and standards-based, meaning that for the first time ever interoperability is possible across vendors.  This provides customers with assurance that they are not locked into a given vendor, allowing them to take advantage of supply chain efficiencies that arise when standards exist.  As coherent pluggables replace chassis-based transponders, customers realize a significant savings in both space and power.  It is our conviction around the value of pluggable coherent optics to our customers that led Cisco to acquire Acacia in March 2021.

• Open Standards:  DWDM remains one of the few areas in networking where standardization has not taken place.  Arguments around pace-of-innovation and the need for differentiation have long clouded the arguments in favor of standardization, but we are now at a point where there is general agreement among vendors, customers, and suppliers that the incremental improvements offered by proprietary solutions do not outweigh the benefits of standardization.  Open Line Systems allow third-party DWDM wavelengths to be carried over line systems, which may include muxes/demuxes, ROADMs, and amplifiers.  Open Line Systems allow customers to avoid vendor lock-in (technically or commercially), while industry efforts like OpenConfig, Open ROADM, OIF, TIP, and other forums provide a path to a truly open DWDM layer.  Open standards have proven to be good for customers and to help drive supply chain efficiencies.The time has come for DWDM to be standardized.  The Routed Optical Networking architecture leverages open coherent pluggables in routers (and in traditional transponder-based solutions), and open line systems so customers can realize the benefits of “open.”  Open wins.

• Common Automation and Management Approach:  As operators embrace NetOps and deploy more automation, orchestration, and virtualization to respond faster to their customers and reduce OpEx costs, the traditional network management stack does not work. There’s a need for end-to-end visibility and cross-domain provisioning across IP and optical network layers as operators look to drive the network based on intent.  Vendor-neutral programmatic configuration across domains is essential.  Real-time and granular visibility along with AI/ML analysis is key to faster troubleshooting, better planning, automated optimization, and network self-healing.  Open networking models and software interfaces such as OpenConfig, Open ROADM, ONF, TIP, and IETF have helped in this evolution.  The industry efforts also tackle the actual architecture to enable third-party interoperability by defining the hierarchical model.  This provides that domain-specific network elements will communicate northbound via an open interface and open device models to a domain controller that in turn speaks northbound to a hierarchical controller via open interfaces and open service /topology models.  Cisco’s recent acquisition of Sedona Systems was motivated by our belief that hierarchical control will be a necessary ingredient in converging network layers.

• Circuit Emulation and Private Line Emulation: Converging network layers is one of the key objectives of the Routed Optical Network – it delivers on network simplification, as well as both significant CapEx and OpEx cost reduction.  Traditionally, networks were layered due to where we were in the technology lifecycle.  As an example, the delivery of transparent, dedicated bandwidth services required an OTN switching platform or a dedicated transponder.  With the introduction of Circuit Emulation (CEM) and now Private Line Emulation (PLE), we no longer need dedicated layer-specific hardware to address services.  With the efforts of the industry and IETF, we’ve defined CEM and are in the process of defining PLE, which will enable network providers to not only carry legacy TDM services, but also high-capacity wavelength services over a converged packet network using modern routing protocols – all while matching or even exceeding existing service level agreements (SLAs).  Our customers can now define their own Routed Optical Networking journey and the pace at which they converge their networks.

• Sustainability:  Beyond our focus on the challenges of networking, we have a responsibility to seek ways to reduce power consumption and CO₂ emissions. With the Routed Optical Networking architecture, network providers can increase their capacity and scalability while reducing footprint and power for a lower carbon impact.  Network modeling of the solution shows up to 40-45% power reduction and real estate savings.  With IOS XR7 modularity, programmability minimizes human intervention during set-up and operation, reducing onsite operations.  Fewer truck rolls, fewer maintenance windows, less packaging, and more recycling will help you achieve your CO₂ emission goals with our leading technological innovations.

How Do We Get from Here to There?

Obviously, many customers have existing networks that have years of remaining useful life.  Can these customers take advantage of Routed Optical Networking?  Absolutely.  Routed Optical Networking is an architecture, and architectural shifts take time.  Most customers can begin the Routed Optical Networking journey by leveraging pluggable coherent optics – either directly in routers for those who are seeking to gain power/space efficiency, or in transponders for those who want to preserve existing operations models or transition these over time.  ROADMs are part of existing networks and can certainly remain in place.  As routing infrastructure is upgraded to support massively scalable capacity, more services can take advantage of the full benefits of the Routed Optical Networking (e.g. leveraging the IP layer to efficiently aggregate services onto 400G coherent wavelengths delivered with pluggable technology), while existing services remain on the existing infrastructure.  As new networks are planned, a more significant shift to a Routed Optical Networking architecture employing increased hop-to-hop transport with a simplified DWDM layer can be considered.

As with the introduction of any new architecture, skeptics abound.  Those vendors who perceive Routed Optical Networking to be a threat to their business-as-usual will argue that the architecture of the past 20 years is the right architecture going forward or may argue that a proprietary solution is warranted.

Call your Cisco Account Manager and let’s have a discussion.  We’ll be happy to model your network and use your data to help you decide what architecture makes sense for your specific needs.