Cisco Blogs

From Telco Tragedy to Recovery in 28 Days

March 1, 2011 - 0 Comments

By Steven Shepard, Contributing Columnist

As companies grow large, we have a natural human tendency to declare them incapable of getting out of their own way. We see them as slow, inefficient and plodding dinosaurs surrounded by fleet-of-foot small companies that are so much more capable of getting things done — because of their apparent smallness.

And while there are certainly large companies that suffer from bigness, there are also quite a few that demonstrate amazingness, if you’ll allow me to make up a word (for agility, beyond compare). One of them? The mighty, much criticized Bell System, created in the transition zone between the 19th and 20th centuries.

Yes, they created the Public Switched Telephone Network (PSTN) in the U.S., an amazing feat by anyone’s estimation (modern historians maintain that it would be technically difficult, if not downright impossible, to build it today).

But long after the system was built — precisely 100 years later, in fact — the collective Bell System had the opportunity to demonstrate its amazingness by converting a colossal organizational tragedy into a public success.

Firestorm in the Heart of New York City

The event began innocently enough. Early on the morning of February 26th, 1975, a piece of electrical equipment in the basement of the New York Telephone Company 2nd Avenue switching center in New York City spewed a cloud of sparks into the air.

The sparks started a small fire which ignited the plastic insulation on nearby cables. The cables, which ran from the cable vault in the basement to the upper floors via holes cored in each floor, provided a conduit for the fire to spread — and spread it did.

Within a few hours it had moved up from the basement to the upper floors of the building, engulfing multiple levels of carrier, switching, multiplexing, power, and service provider transport equipment in smoke, flames, and soon thereafter, thousands of gallons of water.

The high-profile central office, which served lower Manhattan, was essentially cut-off from the rest of the Bell System network – no inbound communications, nothing outbound. The event ushered in the worst communications disaster in American history.

Needless to say, local public safety vehicles transporting police officers and numerous firefighters descended on the building. But more immediately — and perhaps more importantly — hundreds of Bell System personnel and resources began to arrive on the scene within the first hour of the conflagration.

While firefighters took care of the immediate danger, the Bell Companies  (22 of them at the time), Western Electric (later to become Lucent Technologies), and Bell Laboratories began to funnel recovery resources into what many believe to be the greatest telecom infrastructure recovery story of all time — although most people outside our industry have never heard about it.

Mobilization of a Dramatic Recovery Effort

It was the ability to quickly marshal the resources required to restore service that is the most remarkable part of this story. Five thousand employees worked around the clock, doing whatever needed to be done. Bell Companies diverted resources from other projects to New York to aid in the recovery.

A Main Distribution Frame (MDF), the massive iron structure that supports the hundreds of thousands of miles of jumper wire in the office, was shipped from the western U.S. for installation in the 2nd Avenue office. Normally it takes six months to install and turn up an MDF; this one was shipped, assembled, installed, wired, tested, and turned up in four days.

Twenty eight days after fire and water destroyed the equipment in the building and knocked the whole wire center offline, service was restored to Manhattan — 100% of it. Just 28 days: It can take longer than that to get a mere dental appointment. Thus was the power and the amazingness of the Bell System.

Big? Sure. Slow-moving? Sometimes; but when it came to taking care of customers and preserving service, whatever you do, don’t step in front of a purposeful juggernaut.

>>More… Connected Life Exchange

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.