In collaboration with Christie Pinschmidt
“If it ain’t broken, don’t fix it.”
This time-worn expression has been applied to countless situations, by many organizations, over the years – often with unfortunate results.
The downside of this approach is especially apparent when it comes to managing network software. Too often, IT teams delay network software upgrades because they view the process as painful, tedious, and time-consuming. As a result, they hold off until a critical issue arises – typically a security vulnerability that requires an all-hands-on-deck “fire drill” to remedy.
Cisco IT has taken steps to establish a more consistent process for managing network software upgrades prescribed by the Cisco Product Security Incident Response Team (PSIRT). PSIRT is a dedicated, global organization that receives, investigates, and publicly reports security vulnerability information related to Cisco products and networks. Often, resolving a reported incident requires upgrades to products under active support from Cisco.
Cisco IT recently realized that it needed to address two scenarios associated with “chasing” PSIRT upgrades:
- Having to implement these upgrades by working weekends – and sometimes by scrambling to address zero-day vulnerabilities.
- Struggling to handle an ever-increasing number of lower- and medium-priority PSIRT upgrades.
The objective was clear: enable faster, easier, and more frequent upgrades of network elements, while maintaining a secure environment. As part of its mission, Cisco IT sought to reduce the number of noncompliant/undefined network devices to zero, while also making the process as painless as upgrading a mobile phone.
Harnessing a controller and automation to deliver faster, easier SWIM upgrades – at scale
To achieve these goals, Cisco IT is harnessing the power of Cisco DNA Center to perform operating system software image management (SWIM) upgrades faster and more consistently than ever before.
Cisco DNA Center is a powerful network controller that, among other things, enables zero-touch device provisioning and SWIM features that reduce device installation or upgrade time from hours to minutes. Best of all, it allows Cisco IT to deliver SWIM upgrades at scale.
The ability to conduct SWIM upgrades at scale is critical for Cisco IT, which has a goal of upgrading every device managed by the Cisco Network Service (NWS) organization – about 35,000 elements – at least twice per year. These networks span Cisco’s campus LAN, WAN, data centers, and branch offices (about 400), along with partners and Cisco’s remote workers who have managed connections (CVO/MVO). The network devices comprise access points (about 14,000), work-at-home devices such as CVOs and MVOs (about 10,000-11,000), and “big boxes” such as switches, routers, and firewalls (about 9,000-10,000).
Cisco IT’s twice-yearly upgrade objective is designed to align with the network software upgrade schedule set by Cisco’s Enterprise Networking and Meraki business unit (BU), which releases PSIRT bundles (critical releases, major patches, etc.) every two quarters for each platform. In addition, the BU sprinkles smaller updates throughout the year.
Cisco IT quickly realized it could reach and sustain twice-yearly upgrades of 35,000+ devices only by leveraging network controllers like Cisco DNA Center to implement SWIM. Using Cisco DNA Center, Cisco IT’s engineers can perform SWIM upgrades simply by selecting an image, clicking a few buttons, and leveraging automation capabilities to upgrade devices automatically.
The solution currently utilized by Cisco IT is, admittedly, relatively basic – it performs SWIM tasks on a list of devices via simple automation, then updates and pushes pre- and post-checks to the change record and closes the change. In the future, however, Cisco IT sees the potential to fully automate the upgrade process, so that engineers don’t even need to touch the system. Each device type would have its own upgrade window, and the system would perform the check-in and check-out steps entirely on its own.
Driving significant early-stage benefits
Although Cisco IT is still in the initial stages of implementing its twice-yearly SWIM upgrades across the company’s 35,000+ NWS-managed devices, early returns are promising:
- By achieving consistent, twice-yearly upgrades at scale, Cisco IT is establishing a standard for customers to follow. Few, if any, Cisco customers are currently upgrading network elements twice per year. In fact, some are currently not performing any upgrades over the lifetimes of their products.
- Cisco IT’s solution incorporates simple automation to accelerate and smooth network upgrades – with the potential for faster and even more frequent upgrades in the future.
- The use of Cisco DNA Center improves Cisco IT employees’ productivity and experience by eliminating tedious manual patching and upgrading.
- Most important, the solution allows Cisco IT to address the BU’s critical upgrade bundles in a timely fashion. This improves the company’s security posture by identifying vulnerabilities caused by obsolete versions of the O/S software, or by having too many software versions on the network.
Based on Cisco IT’s early SWIM upgrade success, perhaps it’s time for a new catchphrase: “Fix it before it’s broken.”