I do enjoy the (sometimes irreverent!) perspectives of some articles on data center in the UK publication “The Register“, and the story of how a data center change went wrong made me laugh, cringe and cry at the same time – the change being when an electrician cut the wrong wire and brought down a 25,000 square feet data center!!!
Let’s have a look at what went wrong here, and then I’ll relate this to one of our more fundamental services, that of “change support”. Some may call this a “boring” service option, however “fundamental” is much more appropriate, as the following story will show. Finally, I’ll point you to a free white paper to illustrate the cost benefits.
If you’ve worked in IT for any length of time, you’ll know that configuration changes and (incorrect) cable cuts are some of the biggest sources of network and data center unplanned downtime – that is, outages. However, even in 2016, it’s amazing how the lack of stringent change control processes is all too common a source of outages and service downtime. Let’s look at a real life example.
Going back to The Register story, you have to feel for the poor data center engineer, that they name as “JF”. He estimated the chance of failure at 1%. He didn’t quite realise this to start with, but his management thought that this was “acceptable”. Except that the 1% chance happened – and an electrician cut the wrong cable, resulting in a complete outage. 25,000 square feet of DC equipment suddenly became very quiet. Ouch! It took 36 hours to recover the data center – due in no small part of complete lack of change planning – for example, there was no-one available or on standby from the application, sysadmin, network admin, and DBA teams. As the author says, “because with just a one percent chance of failure why did it need to bother?”. Ouch! And poor JF got a real hard time, shall we say (he quit a month later).
Within our Cisco Network Optimization Service (NOS) and Data Center Optimization Service (DCOS), one of the service options is “Change Support”. You could argue it’s a pretty boring service: we help customers plan and execute changes in networks and data center. Or rather, we help them avoid major outages abd business impacts via comprehensive planning and risk mitigation. That isn’t so boring, in fact it’s business critical!
Change Support provides you with a Cisco expert experienced in managing network and data center change. They help you plan for failure scenarios, for rollback in the event of issues, and they lend weight to you when helping senior management understand the implications of the change. I bet you poor JF wished he had the backing of Cisco NOS Change Support for his change.
This and more are available as options in Cisco Optimization Services. As you’d expect, Cisco Optimization Services have been shown to generate substantial return on investment. Read the Forrester Total Economic Impact white paper to learn more, or ping me via Twitter or the comments section below to discuss.
If companies don’t have a structured change support policy then just about anything can go wrong and recovery can be long and painful. Our company has a great change control process but for those who don’t Cisco Support sounds like a great solution.
thanks Janice, really appreciate your comment – and hope you enjoyed the tsory above 🙂 – good to hear you have a strong process! Even for customers with strong processes, our consultants can give you a view on “state of the art” in order to drive further improvements. Food for thought I hope.