Written by Omar Santos, Incident Manager, Cisco Product Security Incident Response Team (PSIRT) Security Research and Operations
Since the early 1990s, we’ve watched as the number of entries on the Internet routing table has steadily grown. In 2008 the table reached 256,000 routes, triggering action by network administrators to ensure the continued growth of the Internet. Today we know that another significant milestone has been reached, as we officially passed the 512,000 or 512k route mark!
Our industry has known this milestone was approaching for some time. In fact it was as recently as May 2014 that we provided our customers with a reminder of the milestone, the implications for some Cisco products, and advice on appropriate workarounds.
If you would like to revisit that information, you can find the customer support article here: The Size of the Internet Global Routing Table and Its Potential Side Effects (12 May 2014)
Full text of the customer support article below:
Since the early 1990s, we’ve watched as the number of entries on the Internet routing table has steadily grown. It wasn’t that long ago (2008) that the table reached 256k routes, triggering action by network administrators to ensure the continued growth of the Internet. Now we have passed that another significant milestone – the global routing table has passed 512,000 routes.
As an industry, we’ve known for some time that the Internet routing table growth could cause Ternary Content Addressable Memory (TCAM) resource exhaustion for some networking products. TCAM is a very important component of certain network switches and routers that stores routing tables. It is much faster than ordinary RAM (random access memory) and allows for rapid table lookups.
Networking Product Implications
No matter who provides your networking equipment, it needs to be able to manage the ongoing growth of the Internet routing table. We recommend confirming and addressing any possible impacts for all devices in your network, not just those provided by Cisco. The products that could be affected include those with a default configuration supporting 512k routes. From Cisco’s perspective, this includes:
- Cisco Catalyst 6500 Switches
- Cisco 7600 Series Routers
- Cisco ASR 9000 Series Aggregation Services Routers configured with Trident-based line cards (typhoon-based line cards are not affected)
- Cisco ASR 1000 Series Aggregation Services Routers with 4GB (devices with 8GB or RAM or higher can scale to up to 1,000,000 routes)
The Good News – Workarounds Are Available!
Cisco has published information on several workarounds that can be applied by our customers, including changing the default configuration for affected devices. In some cases this may require a reload of the device or line card. See below for the links to this customer information.
Cisco Catalyst 6500/Cisco 7600 Series Supervisor Engine 720
The following document describes how to customize the forwarding information base (FIB) ternary content addressable memory (TCAM) on Catalyst 6500 switches that run the Supervisor Engine 720: click here.
This guidance is specific to the Supervisor Engine models SUP720-3BXL and SUP720-3CXL. The “non-XL” versions do not support more than 256,000 IPv4 routes.
Click on the embedded links for additional information about the Cisco Catalyst 6500 and the Cisco 7600 Series Supervisor Engine 720 capabilities.
Cisco ASR 9000 Series Aggregation Services Routers
The following document describes workarounds available for the Cisco ASR 9000 Series Aggregation Services Routers. When a Trident-based line card reaches its prefix limit, the message %ROUTING-FIB-4-RSRC_LOW occurs, causing potential traffic loss on the line cards: click here.
Cisco ASR 1000 Series Aggregation Services Routers
Cisco ASR 1000 Series Aggregation Services Routers with 4GB can scale to up to 500,000 IPv4 or IPv6 routes. Cisco ASR 1000 Series Aggregation Services Routers with 8GB of RAM or higher can scale to up to 1,000,000 routes. The following document provides an overview of the number of supported routes: click here.
Additional Workarounds
Route filtering and the use of a default route can also be used to decrease the number of routes in an affected device. Prefix lists can be used as an alternative to access lists in many BGP route-filtering commands. The use of prefix lists provides significant performance improvements when loading and performing route lookup of large routing tables. Additional information about BGP best practices and configuring prefix lists is available at: click here.
Security Considerations
The possibility of TCAM resource exhaustion at 512k routes is a known issue that we all know has been coming for some time. There is no related security vulnerability, and it cannot be easily triggered by a remote, untrusted user.
This website is a great resource that provides the current state of the Internet routing table. This could help Cisco customers when configuring route filtering.
Implementing the recommended workarounds ahead of time will help your network avoid any performance degradation, routing instability, or impact to availability. Having just passed the 512,000 route milestone, now is the right time to verify and ensure your network is prepared to manage a 512k entry Internet routing table.
Hi I have one question for you.
In our Cisco 6504, I saw a weird log as following
“Aug 13 02:35:21.442: %MCAST-SP-6-ETRACK_STATS_LIMIT_EXCEEDED: Number of entries in IGMP snooping explicit-tracking statistics has exceeded the maximum limit (32000)”
I want to know is this log related to the 512K limit.
And this command “sho platform hardware capacity pfc” show me this
72 bits (IPv4, MPLS, EoM) 524288 432158 82%
I am confused about it because I do not know about the IGMP or multicast very well. Thank you very much.
Hi Harry,
Thank you for your comment and for reading the article. These messages are not related to the 512K routing table limit. These are messages related to the Internet Group Management Protocol (IGMP) snooping feature.
The IGMP snooping functionality examines IGMP protocol messages within a VLAN to discover which interfaces are connected to hosts or other devices interested in receiving this traffic. Using the interface information, IGMP snooping can reduce bandwidth consumption in a multi-access LAN environment to avoid flooding the entire VLAN. The IGMP snooping feature tracks which ports are attached to multicast-capable routers to help it manage the forwarding of IGMP membership reports. The IGMP snooping software responds to topology change notifications.
The number of explicit-tracking statistics entries are bounded to avoid monopolizing of system resources by IGMP/MLD snooping. The explicit-tracking statistics database maximum size is set to the same as that of explicit-tracking limit. The statistics are split into two banks: permanent and volatile. The statistics entries will be stored permanently (volatile) until the permanent entries threshold is reached, after which the statistics will be stored temporarily up to the database maximum size. When the size of the database exceeds the permanent threshold, a group will be removed on receiving an IGMP/MLD leave. Groups can be removed whether they were installed above or below the permanent threshold. If the number of statistics entries grows beyond the maximum size of the database, then no
statistics will be stored for any newly formed groups.
If the number of groups being used in the network exceeds the recommended value of the explicit-tracking database, you can increase the size of the database using the following command:
ip {igmp | mld} snooping limit track {0-128000}
If the number of groups being used by hosts exceeds the recommended value of the explicit-tracking database, but the number of groups currently active is less than the database size and you require complete statistics, you can set the explicit-tracking limit to the maximum using the ip {igmp | mld} snooping limit track {0-128000} command. If you are interested only in statistics of the groups currently active in the system, you can clear the IGMP/MLD snooping statistics to free up space in the database. The following command can be used to clear snooping statistics:
clear ip {igmp | mld} snooping statistics [interface [type slot/port | vlan x]]
You can use the following command to list the amount of the entries in your device:
switch# sh ip igmp snooping stat
Current number of Statistics entries : 40602
Configured Statistics database limit : 48000
Configured Statistics database threshold: 38400
Configured Statistics database limit : Not exceeded
Configured Statistics database threshold: Exceeded
So to resolve the problem, please try increase the size of the database.
If you are not using the IGMP snooping feature you can disable it by using this command:
ip igmp snooping limit track 0
If you have any additional questions or need technical assistance on this feature and messages, please contact the Cisco TAC at:
http://www.cisco.com/c/en/us/support/web/tsd-cisco-worldwide-contacts.html
Thanks again!
Omar Santos
How looks about Cisco 7206 VXR with NPE-G1 and NPE-G2 and 2G DRAM.
Hi Tony!
Thanks for reading the article. Cisco 7200 routers (both with NPE-G1 or NPE-G2) are enterprise-based routers. Both the route processor and the chassis are in end of sales and end of support for a few years. Some small service providers used them, but nowadays they are not expected to carry the full global Internet routing table.
Depending on the RP and the models, 7200’s could not even the full routing tables even a few years ago. The following link provides a comparison of the models.
http://www.cisco.com/c/en/us/products/routers/7200-series-routers/models-comparison.html
2GB of DRAM will be limited in current situations and with the expected growth of the Internet.
Route filtering and the use of a default route can be used to decrease the number of routes in an affected device. Prefix lists can be used as an alternative to access lists in many BGP route-filtering commands. The use of prefix lists provides significant performance improvements when loading and performing route lookup of large routing tables.
http://www.cisco.com/web/about/security/intelligence/protecting_bgp.html#8
Even after route filtering/summarization, I recommend to always monitor the memory usage in your device you can use the “show ip bgp sum” command and look for the output “BGP using XXX total bytes of memory”, as shown below:
route>show ip bgp sum
BGP router identifier 1.1.1.1, local AS number 65000
BGP table version is 17394190, main routing table version 17394190
BGP using 376930499 total bytes of memory <<<<
Hope this helps.
Regards,
Omar
Hi Omar
Thank you for your explanation
My questions is about this old router if they have also a Ternary Content Addressable Memory (TCAM) that limited the number of prefix ?
If i look the following printouts then i can see that the BGP use only 193MB for BGP
On this router is a 2GB DRAM installed and 1345218252 Free.
Isn’t it ?
sh ip bgp sum
BGP router identifier xxx.xxx.xxx.xxx, local AS number 65000
BGP table version is 191357010, main routing table version 191357010
502359 network entries using 64301952 bytes of memory
1346747 path entries using 70030844 bytes of memory
413476/87387 BGP path/bestpath attribute entries using 51271024 bytes of memory
209736 BGP AS-PATH entries using 7562090 bytes of memory
646 BGP community entries using 27854 bytes of memory
2 BGP extended community entries using 48 bytes of memory
510 BGP route-map cache entries using 18360 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 193212172 total bytes of memory
BGP activity 7445408/6924193 prefixes, 77660045/76272510 paths, scan interval 60 secs
show memory
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 423E660 1927023796 581805544 1345218252 1331981412 1323044020
I/O 78000000 67108864 6099412 61009452 60954752 56516412
Transient 77000000 16777216 44264 16732952 9190196 16732676
What do you think
Will crash the router over der 512000 Prefix ?
Thank you in advance
Toni
The 7200 NPE-G2 support up to 2GB of memory (which is what you have). Technically speaking, that should be sufficient to hold more than a million routes. On the other hand, once the device is booted and the operating system loads the processes use the remaining memory. How much memory depends on each configuration. For BGP memory usage it depends on things like how many paths per prefix are being imported, type of route (IPV4, VPNV4) and attributes associated with the routes. The best recommendation is to monitor the memory utilization using the aforementioned commands and/or SNMP.
Hi Tony!
Thanks for reading the article. Cisco 7200 routers (both with NPE-G1 or NPE-G2) are enterprise-based routers. Both the route processor and the chassis are in end of sales and end of support for a few years. Some small service providers used them, but nowadays they are not expected to carry the full global Internet routing table.
Depending on the RP and the models, 7200’s could not even the full routing tables even a few years ago. The following link provides a comparison of the models.
http://www.cisco.com/c/en/us/products/routers/7200-series-routers/models-comparison.html
2GB of DRAM will be limited in current situations and with the expected growth of the Internet.
Route filtering and the use of a default route can be used to decrease the number of routes in an affected device. Prefix lists can be used as an alternative to access lists in many BGP route-filtering commands. The use of prefix lists provides significant performance improvements when loading and performing route lookup of large routing tables.
http://www.cisco.com/web/about/security/intelligence/protecting_bgp.html#8
Even after route filtering/summarization, I recommend to always monitor the memory usage in your device you can use the “show ip bgp sum” command and look for the output “BGP using XXX total bytes of memory”, as shown below:
route>show ip bgp sum
BGP router identifier 1.1.1.1, local AS number 65000
BGP table version is 17394190, main routing table version 17394190
BGP using 376930499 total bytes of memory <<<<
Hope this helps.
Regards,
Omar
My question is: so far which is the available amount of Internet Routing if we have a peak?
Hello James,
In terms of predicting the table size in the IPv4 and IPv6 networks, many folks have said “that the sky is the limit”. Routing scalability will continue to be one of the most
important challenges facing the Internet.
The Internet continues along a path of seemingly unavoidable growth. The number of registration will continue to grow at a very fast pace. The IPv4 routing information base (RIB) growth has been controlled by the limited IPv4 address space. IPv6 address space is substantially larger. IPv6 uses a 128-bit address, allowing 2^128, or approximately 3.4×10^38 addresses, or more than 7.9×10^28 times as many as IPv4.
Keep in mind that as IPv6 continues to be widely deployed, you can expect that the routing table growth for IPv6 will only intensify the situation.
There is a lot of effort in the industry to come up with solutions for this paradigm. A good example is the Locator/ID Separation Protocol (LISP). LISP optimizes routing. LISP is a Cisco innovation that is being promoted as an open standard. Cisco participates in standards bodies such as the IETF LISP Working Group to develop the LISP architecture. The following link provides more information about LISP:
http://www.cisco.com/go/lisp
Regards,
Omar
Hello, this was a very informative article but I have one question though. On the ASR 1000 Series routers you stated that with the RP1 with 4GB of RAM it will only scale to 500,000 routes yet the article you link to states that it will scale to 1,000,000 (table 2, 5th and 7th lines). Which is it?
Is the limitation on the ASR1002 on the RP1 or the ESP card or both? We have asked TAC and were told we are OK becuase we have an ESP10 card that can hold up to 1 million entries in the forwarding pane, even though we only have the embedded RP1 cards that are fixed at 4GB and not upgradeable.
Hello Karl and James,
Thank you for the great comments. The aforementioned limitation in the ASR1000 was in the RP. The limit of the routes you can have in an ESP is also dependent on the model.
Starting with the RP limitations:
The following is the official Cisco ASR 1000 Series Route Processor Data Sheet:
http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/data_sheet_c78-441072.html?cachemode=refresh
With 8-GB memory:
● Up to 1,000,000 IPv4 routes or 1,000,000 IPv6 routes
● BGP RR Scalability up to 8,000,000 IPv4 routes or 6,000,000 IPv6 routes
With 16-GB memory:
● Up to 4,000,000 IPv4 routes or 4,000,000 IPv6 routes
● BGP RR Scalability up to 24,000,000 IPv4 routes or 17,000,000 IPv6 routes
The following is the Cisco ASR 1000 Series Embedded Services Processors (ESP) Data Sheet, which lists the limits of each ESP model:
http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/datasheet-c78-731640.html?cachemode=refresh
The ESP 10 supports:
1,000,000 IPv4 or 500,000 IPv6 routes
Multicast: 64,000 routes and 1,000 groups
All other models/limits are listed in the aforementioned link. Hope this helps.
So if you have an ASR1002 it has the built in RP1 (non-upgradeable) the max IPv4 routes the platform will hold is 500K. Even if the ASR1002 has an ESP10 the RP1 TCAM will start dropping routes?
Hi Karl,
If you have an ESP10 you will be able to hold 1,000,000 IPv4 routes. As mentioned earlier, the ESP 10 supports:
1,000,000 IPv4 or 500,000 IPv6 routes
Multicast: 64,000 routes and 1,000 groups
The following is the Cisco ASR 1000 Series Embedded Services Processors (ESP) Data Sheet, which lists the limits of each ESP model:
http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/datasheet-c78-731640.html?cachemode=refresh
Hope this helps!
Omar
The ASR1002-RP1 with the ESP-5 and 4GB of memory does have the 500k route limit. But since it does not use TCAM, what is the impact to the router if it does hit that limit?
I also want to know
According to the data sheet
RP1(4GB) IPv4 1,000K
ESP5 IPv4 500K
ESP10 IPv4 1,000K
The maximum routes determined by the limit of ESP?
RP1(4GB) + ESP5 IPv4 500K
RP1(4GB) + ESP10 IPv4 1,000K
Yes this is still the crux of the issue that no one has answered!
They keep saying RP1 with 4GB handles 500K routes then go on to talk about the capacity of the ESP card. So far as I can tell from other sources the TCAM on ESP card on the ESP card does not store routes only ACL,QoS mappings etc
Omar – if a customer has the original ASR1002 with the built in RP1 (fixed at 4GB) can they increase the routes it can hold by replacing an ESP5 with an ESP10 card? or does the whole ASR1002 have to be replaced because regardless of what the ESP card can do if it has an RP1 it can never hold more than 512K routes.
The number of supported routes in the ASR 1000 seriesis determined by the combination of both the RP and the ESP. When the system has an RP1 + ESP5, the limit is 500k (because of the ESP5).
When the system runs an RP1 + ESP10, the limit is 1 million routes (both RP1 and ESP10 reach the limit).
Subsequently, you can upgrade ESP5 to ESP10 on ASR1002 to increase the route scale from 500k to 1M.
What defines the “global table”?
I have the “full table” from Tier 1 providers like Level3 (AS3356), Telia (AS1299), NTT, TATA and XO … they all vary slightly, but are around 500k each now, 499,x (though peaked about 503,650 10-14 days ago)
I have smaller private peers/customers that add another 1k or so routes… and my “show bgp summ” right now is “500795”
It was as high as 504,000 14 days ago.
I do note that “route-views” is at “532470” at the time of this post (2014-09-03 0320 UTC).
… but other than large networks with many customers/peers with non-transit routes … few networks should have passed the 512k IPv4 TCAM (at 72 bytes, 524,288 route) limit, correct?
I’m not sure how to see Level3’s internal BGP table, HE.net is at 508k. (Which providers are past 524k routes?)
The context for my question is we apparently were hit by 512k in that we had 499,760 routes and were at “95% TCAM utilization” per the 7600 router I was working with (2 weeks ago). We had really bad packet loss issues that we could not resolve, and after reboot (we had pre-staged the TCAM memory to 768k) the routes went to 503,650. The increase of 4800 new routes simply on the reboot (confirmed on multiple routers) — is something I don’t understand.
My hypothesis, our 7600/RSP720 in 3CXL mode was lying to us… even though we were at 95% — the 499k routes was about the max it could support, not the 524,288 is thinks it could take….?
What defines the “global table”?
I have the “full table” from Tier 1 providers like Level3 (AS3356), Telia (AS1299), NTT, TATA and XO … they all vary slightly, but are around 500k each now, 499,x (though peaked about 503,650 10-14 days ago)
I have smaller private peers/customers that add another 1k or so routes… and my “show bgp summ” right now is “500795″
It was as high as 504,000 14 days ago.
I do note that “route-views” is at “532470″ at the time of this post (2014-09-03 0320 UTC).
… so, “global table” … other than large networks with many customers/peers with non-transit routes … few networks should have passed the 512k IPv4 TCAM (at 72 bytes, 524,288 route) limit, correct?
What providers have more than 512k? How is this seen publicly?
I’m not sure how to see Level3′s internal BGP table, HE.net is at 508k. (Which providers are past 524k routes?)
The context for my question is we apparently were hit by 512k in that we had 499,760 routes and were at “95% TCAM utilization” per the 7600 router I was working with (2 weeks ago). We had really bad packet loss issues that we could not resolve, and after reboot (we had pre-staged the TCAM memory to 768k) the routes instantly went to 503,650. The increase of 4800 new routes simply on the reboot (confirmed on multiple routers – over 24 hours) — is something I don’t understand.
My hypothesis, our 7600/RSP720 in 3CXL mode was ‘lying’ to us… even though we were at 95% — the 499k routes was about the max it could support, not the 524,288 is thinks it could take…? Hence after reboot it went to 503k (wish I was had rebooted on the 512k memory to see what would have happened … our 7600’s are giving us a lot of issues lately – so could have been something else the reboot alone cleared)
# sorry for the double-post, I didn’t realize I “replied” to a different post prior, could not delete that reply.
PS, our router said “Current IPv4 FIB exception state = FALSE” prior to reboot, was at 95% TCAM used, and I’ve gotten HORRIBLE support in my Cisco case (SR 631474023) …
## router captures
### PRE REBOOT ###
16:11:46.763 UTC Mon Aug 18 2014
Module FIB TCAM usage: Total Used %Used
9 72 bits (IPv4, MPLS, EoM) 524288 498844 95%
144 bits (IP mcast, IPv6) 262144 9 1%
#sh mls cef su
Total routes: 498793
IPv4 unicast routes: 496738
IPv4 Multicast routes: 4
#show mls cef exception status
Current IPv4 FIB exception state = FALSE
#### POST reboot ####
L3 Forwarding Resources
Module FIB TCAM usage: Total Used %Used
9 72 bits (IPv4, MPLS, EoM) 802816 503722 63%
144 bits (IP mcast, IPv6) 122880 9 1%
18:22:23.343 UTC Mon Aug 18 2014
#sh mls cef su
Total routes: 503648
IPv4 unicast routes: 501593
IPv4 Multicast routes: 4
MPLS routes: 2049
IPv6 unicast routes: 2
IPv6 multicast routes: 3
EoM routes: 0
#sh mls cef maxi
FIB TCAM maximum routes :
=======================
Current :-
——-
IPv4 – 768k
MPLS – 16k (default)
IPv6 + IP Multicast – 120k (default)
You’re killing me Omar Santos. Why didn’t you mention that the integrated 4GB RP1 found in an ASR 1002-F actually supports 1000,000 IPv4 routes?