Insights About the Global Internet Routing Table Reaching the 768k Milestone


April 22, 2019 - 2 Comments

Back in 2014, I wrote an article that highlighted that global Internet routing table passed the 512,000 or 512k route mark. Today we know that another significant milestone has been reached, as we passed the 768k route mark!  Many have predicted Internet outages may be expected. In short, the “sky is not falling”. The possibility of TCAM resource exhaustion at 768k routes is a known issue that we all know has been coming for some time. There is no related security vulnerability, and it cannot be easily triggered by a remote, untrusted user.

As I mentioned in my previous article, we’ve known for some time that the Internet routing table growth could cause Ternary Content Addressable Memory (TCAM) resource exhaustion for some networking products. TCAM is an important component of certain network switches and routers can store routing tables. It is provides deterministic and fast table lookups. TCAM is “reverse memory” that excels in match lookups.

However, not all platforms running Cisco IOS XR and Cisco IOS XE Software use TCAM for forwarding decisions. For instance, the Cisco NCS 5500 Series uses TCAM for their FIB lookup. On the other hand, the Cisco ASR 9000 uses search memory instead. While TCAM is a fast resource, it is expensive in cost and power consumption so it is to be used with care in any hardware design.

No matter who provides your networking equipment, it needs to be able to manage the ongoing growth of the Internet routing table. We recommend confirming and addressing any possible impacts for all devices in your network, not just those provided by Cisco. The products that could be affected include those with a default configuration supporting 768k routes.

Modern routers such as the Cisco ASR Routers support millions of routes. For instance, the Cisco ASR 1000 Series Aggregation Services Routers support up to 4 million routes.

The best practices and workarounds in my previous article are still valid. If you are running legacy devices, consider action if any of those devices were previously hard set to 768K routes.

For example, this document describes how to identify and resolve a common problem caused by growth of the Internet routing table. The document describes how you can detect when a Cisco ASR 9000 Series Aggregation Services Routers reaches its prefix limit. The router logs messages such as the following when the limit is reached:

LC/0/3/CPU0:Apr  22 01:15:18.103 : fib_mgr[169]: %ROUTING-FIB-4-RSRC_LOW :
CEF running low on DATA_TYPE_TABLE_SET resource memory. CEF will now begin
resource constrained forwarding. Only route deletes will be handled in this state, which may result in mismatch between RIB/CEF. Traffic loss on certain prefixes can be expected. CEF will automatically resume normal operation, once the resource utilization returns to normal level.

Once the line cards begin to display the %ROUTING-FIB-4-RSRC_LOW message, an outage for some prefixes occurs. This simply means that new prefixes can’t be programmed in hardware until a resource is relinquished (e.g., a prefix is withdrawn).

Route filtering and the use of a default route can also be used to decrease the number of routes in an affected device. Prefix lists can be used as an alternative to access lists in many BGP route-filtering commands. The use of prefix lists provides significant performance improvements when loading and performing route lookup of large routing tables. Additional information about BGP best practices and configuring prefix lists is available at: click here.

Implementing the recommended workarounds ahead of time will help your network avoid any performance degradation, routing instability, or impact to availability.  Cisco designs all networking devices with scale in mind.



In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.

2 Comments

  1. This is not so much a BGP problem specifically. The growth of the table became exponentially worse since the big IPv4 address exhaustion scare. BGP does fairly well with the increased table size, but that also heavily depends on the platform it runs on and how the resources of that platform are able to be used. In the short term, there are also some sane ways to reconsider prefix length acceptance and announcement where /24s are the minimum accepted today.

  2. It is time to redesign routing BGP protocol. IPv6 address space would pose even greater challenge. Current inter domain routing strategy has reached its limits and cannot cops with the future..