$2.8 trillion. That was the estimated cost impact of poor-quality software to organizations in the U.S. alone in 2018.[1] With 100 times more code being managed in 2020 than 10 years ago, the chances for software error and resulting costs are increasing exponentially and globally. The importance of software quality has never been higher.[2]
Software quality is one of the most important ingredients that has led to the incredible success of the Cisco Catalyst 9000 switching family, the fastest ramping product in Cisco history. With four million+ units deployed in mission-critical networks at more than 70,000 organizations globally, the Cisco IOS XE software stack and specific feature sets for the Catalyst 9000 have proven to be incredibly reliable.
It didn’t happen by chance. Starting with software design and architecture, Cisco developers adhere to a quality mindset culture that spans from product architecture to customer experience.
In the Cisco Enterprise Switching team, we follow four fundamental tenets to deliver quality in products such as the Catalyst 9000 and related solutions: Quality@Source, Automation, Solution Focus, and Customer Engagement (Figure 1).
Quality@Source
Quality@Source starts with product design and architecture. Catalyst 9000 consists of five different product families of fixed, stackable, and modular switches such as the Cat9200, Cat9300, Cat9400, Cat9500 and Cat9600 with over 100 SKUs and 10+ different NPUs. Despite thousands of combinations, the Cisco development team built common Cisco IOS XE code with a single binary. It has significantly cut down the time it takes our customers to certify and deploy the Catalyst 9000 in their networks.
The development phase includes comprehensive code reviews, intersection analysis, pre-commit sanity checks, and unit testing to achieve baseline integrity. The code is built with serviceability in mind and is evolving with predictive analytics. It enables Cisco engineers to proactively analyze and fix issues before failure occurs. They can then take corrective action on the network or provide patches proactively.
Automation
In every release of Cisco IOS XE running on the Catalyst 9000, tens of new capabilities are added. To augment the quality maintained during the development phase, automated tests are run continuously to make sure that the quality of the baseline features is protected while new features are added. As new automated scripts are continuously added to the software repository, engineers measure the efficacy of the scripts to optimize runtime and effectiveness
Solution Focus
During the Solution Focus phase, the team tests and validates code maturity by testing customer use cases and with stress, scale, and longevity tests that simulate the way features are deployed in real-world networks.
Due to a rich set of features and capabilities, Catalyst 9000 is deployed in thousands of networks with very different deployment architectures. For example: The network architecture of a very large university is very different than a network deployed in a hospital environment, and they are both very different than a large financial institution with thousands of branches.
The Catalyst 9000 team has built multiple customer vertical profiles based on our knowledge of these networks. These customer profiles contain use cases simulating large customer configurations that are closer to real-world deployment scenarios. Customer Validated Design profile configurations and use cases are made available to customers on the Cisco website for planning their lab certifications and deployments.
Customer Focus
After solution focus testing, the code is tested in the field. The code is deployed in Alpha network which is used by developers and testers every day to do their work. It provides them with a real network simulation and provides feedback on how the product is functioning. After it has been tested on the Alpha network, the product is made available for Early Field Trial (EFT) to customers. Once feedback received from EFT is addressed, the product is ready for First Customer Shipment (FCS).
As World War I hero William A. Foster famously said: “Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution.”
The Cisco Enterprise Switching Software Team truly believes in this philosophy and models it every day.
[1] “The Cost of Poor Quality Software in the U.S.: A 2018 Report,” Consortium for IT Software Quality, 2018.
[2] “Devs are managing 100x more code, in more languages, for more platforms than ever,” Sourcegraph, 2020.
Check out our Intent-Based Networking video channel.
Subscribe to the Networking blog
Awesome!! Way to go Cat9K team!! Proud to be part of such an high performing team.
have not received one good solid code version for the 2K+ Cat 9300Ls in my space.
If you can contact Cisco TAC and provide more details, we can have follow-up on your specific case
We’ve got hundreds, and yeah, they intermittently just sorta “give up” on certain tasks until rebooted.
TAC’s solution: Go to 17.x.
The implication: All of 16.x is incapable of basic functionality and won’t be fixed.
Quality.
If only other major products was like this. I’m looking at you DNA-C ( if you replace all the products mentioned in the above article with DNA-C, you would think its April 1st already ).
And yet using cat 9300L’s, two switches in the same stack manage to go split brain and send packets in a loop around the stack ring and then out into the wider network bringing the site down.
Engineers claim its faulty stack cables but it’s not – and even if it was, the link should be detected and held down not causing looping of traffic(!)
Cisco’s proposal: have a routed LAN/Fabric
My proposal: fix basic functionality first or I’m going elsewhere.
Ditto. And if stacking is to blame I wonder why it continues to get leveraged in Campus deployments.
Please reach out to Cisco TAC for your specific issue so that they can look at switch as well as configuration as next step.
This is so heinous – ‘reach out to Cisco (outsourced) TAC’ – do you not get that this response is a validation of the absolute parody of this post?
What I don’t understand is that if you are doing all of these things why do I have simple defects reappearing, again and again, that should have been caught in even a marginally comprehensive regression test? Shouldn’t every feature and feature combination be tested completely in an automated fashion for every single release – my experience with IOS, IOS-XE, and NX-OS has not even come close to this – we get a defect in a version, it gets fixed in a new version, and then it reappears in the next version.
Why does this quality story end when it is shipped? Who makes sure this works when the second customer gets their hands on it?
Also, I hope that you can respond after Blog Post Shipment, but the quote about Quality came from William Trufant Foster, an economist active in the 1920’s.
Thanks for your comment. Quality story does not end after shipment. Customer engagement is big part of quality and experience. Engineering works very closely with Cisco CX and escalation organizations to make sure that quality experience continues and if there are any issues, they are addressed. Regarding quote, l will check again. Thanks for pointing
I fact-checked quotes.net and quote about quality is attributed to Willian A Foster who was US Marine
Cisco personnel should probably not post as Anonymous.
Fact check the quote – unlikely the quote was delivered by a 21yo. ‘ER’ seems to have the right of it as it’s far more likely to have come from a learned economist
I’ve been running Cisco gear for 20+ years. Cisco, wake up, you have a serious quality problem. Code for the Cat 36/38 (from 3.7+) and all of then Cat 9ks has been a disaster. You can’t keep leaning on your market strength, we have options and are getting fed up. I won’t even start in on the dozens of never realized, half completed, barely adopted (mostly out of customer fear that Cisco will abandon, rebrand, or reimagine the vision) ‘solutions’ the last 8 years or so.