Cisco Blogs

Lights Out On Game Day

January 9, 2012 - 2 Comments

You don’t have to be a football fanatic to feel for the 49ers at Candlestick Park on December 19th, as their nationally-televised game against the Steelers was delayed not only once, but twice, amounting to a 35-minute total delay in the game. You can expect delays due to weather in blistering cold regions, but delays in San Francisco? (And we’re long past the days of the California brown-outs…) It’s no wonder this week the PG&E CEO called it the ‘most embarrassing moment of his career.’ (See interview.)

It turns out the cause was two-fold –  first due to a split wire, and second due to a faulty switch. As a Bay Area native, a San Francisco tax payer, and a wife to an athletic director who lives and breathes sports, I’m no critic, just a sympathist to a tough situation. I think we all feel for this event in some regard.

This recent mishap got me thinking about the network, and how many of us would be on the line if the lights went out…especially when our CEO was present. For most network administrators, if our WAN or border routers went down with no back-up plan, it could easily be the most humbling moment of our lives. Think of what a mere 35-minute outage could cause in your network…Lost customers? Failed communications? VDI sessions broken in client meetings?

And that’s where the Cisco ASR 1000 Series’ set of hardware and software redundancy features really comes in handy. Let’s look at the Cisco ASR 1006 or 1013, which include built-in hardware redundancy. They allow two Route Processors (RPs) and two Enhanced Services Processors (ESPs) in the same router. If an active RP experiences an event that makes it unable to forward traffic (such as a hardware failure, a software failure, an OIR, or a manual switch), the standby RP immediately becomes the active RP. And if an active ESP experiences a hardware or software event that makes it unable to forward traffic, the standby ESP becomes the active ESP with less than 50 ms of interruption.

This helps for failover – if a processor fails, the standby processor immediately becomes the active processor with little or no delay and zero impact on forwarding. The failover happens completely within the same router, so a second standby router is not needed. And it also helps during upgrades – a software upgrade can be handled on the standby processor while the active processor continues normal operation.

Now let’s look at software redundancy. In the ASR 1001, 1002 and 1004, Cisco IOS Software runs as one of many processes within the operating system. (This is different than on traditional Cisco IOS, where all processes are run within IOS, allowing for unique software redundancy opportunities.) Specifically, a standby IOS process can be available on the same RP as the active IOS process. This standby IOS process can take over in the event of an IOS failure or parity errors, and can also be used to upgrade subpackage software, such as applying critical bug or vulnerability fixes, without impacting the non-hardware redundant systems.

There are many more failover and redundancy features on the Cisco ASR, and even better when coupled with the Cisco ISR G2s, to keep your network up and running. But for now, let’s just think about our failover plans…and wish the 49ers luck this postseason with the lights on at Candlestick Park.

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.


  1. I myself was in a similar situation at work today, the WAN went down and with no backup plan, we essentially had to stop work for 2 hours. In our business that is a lot of lost revenue. This just highlights for me the importance of good redundancy devices.