DevSecOps: Lessons Learned

This is Part 4 in our Four Part DevSecOps Blog Series

In Part 3 of our blog series DevSecOps: Automation for Assurance, we covered our automation journey to help us scale security across hundreds of development teams via our Continuous Security Buddy (CSB) capabilities. In the final blog of this series, we will share our success factors, lessons learned, and what we are thinking next as we continue on our DevSecOps journey.

Our Success Story

Adoption of DevSecOps and the security improvements thereof has been quite impressive and has exceeded our own initial expectations. For example, since rolling out CSB minimal viable tool 9 months ago, it is now running in 72% of accounts hosting Cisco’s Cloud offers. Additionally, 97% of these accounts, on average, received a health score of A or B in their daily report indicating a healthy security posture relative to the guardrails we published in part 2.

This has been a fantastic journey at Cloud Speed and along the way we have learned some valuable lessons that are important to share:

Cloud is about “Doing” more than “Telling”

Hackathons provided the opportunity to collaborate with all cross-functional disciplines and deliver on critical security areas mentioned in the guardrails. It also provided a platform for team members to learn from each other, develop empathy and create solutions together rather than being ‘told by InfoSec’. Many of the InfoSec participants had little to no experience in developing Cloud solutions and their security expertise in combination with hands-on doing helped them leap-frog very quickly.

Timing Matters

Timing the initial launch of CSB with the roll-out of an Enterprise Agreement with AWS allowed us to provide a carrot (volume discounts) along with the stick (access for security and cost management visibility). Teams were eager to benefit from the discounts and took the 15 minutes they needed to install CSB in order to join the master account for Cisco that helped with consolidated billing and discounts.

Start Small and Grow

Taking an MVP mind-set of releasing minimal capabilities initially and iterating based on learnings and user feedback was key. Though the teams went with the MVP initially, the CSB team needed to quickly deliver capabilities that created value for the DevOps teams. This continuous visibility via the daily security health reports enabled the teams to self-remediate issues and gain confidence in the security posture of their offer. Iterative development helped the CSB team design the solution in a way that scaled with usage.

Guard-rails vs. Pass/Fail

Security practitioners often dread the word “approval” as it is often situational and comes down to risk management. The Guard-rails approach provides the range of compliance needed based on the situation at hand and allows the teams to manage their risks, A CIS (Center for Internet Security) benchmark score of 80% is within the acceptable risk for an internal host than a hard pass/fail rating.

Cultivating Partnerships

Establishing key partnerships across Operations between IT, InfoSec, Procurement and Product Operations in supply-chain gave us the multiplying effect where the aligned efforts helped us move faster in the same direction ultimately helping the Business Units in their mission to deliver Cloud offers with agility.

Credibility built upon Trust

For the DevOps teams to allow the InfoSec team to have access into their accounts requires building credibility and trust. This means being open and transparent in what we do with the access provided and being available for support if there are any questions or issues. The CSB team along with the partnering organizations created a Spark room (Cisco’s product for team collaboration) and auto-enrolled anyone installing CSB to this chat room via a Bot. No more opening tickets and waiting for a help desk agent to answer questions or address issues. The CSB DevSecOps team are always on the Chat room answering queries and it’s amazing to see the community members support each other. Some went as far as offering code fixes when a feature failure happened.

Taking Risks

Members of the InfoSec team took this journey on boldly. An InfoSec Architect played the role of a Product Owner and trusted his peer as the Product Technical Lead rather than giving into the temptation of making technical decisions. The Product Owner, Product Technical Lead and Scrum Master worked in unison building out the vision and with a small scrum team delivered CSB as one unified team guided by a common goal – always a recipe for success.

Skill-sets Matter

Let’s face it, the InfoSec practitioners didn’t code. Complementing the team with resources from AWS professional services and Developers was important to deliver on the DevSecOps principles in CSB. Now we have developers learning more about Security and the InfoSec teams learning to code and sharpening their development skills.

Drinking our Champagne

Any security solutions we build need to be secure and can’t become the weakest link for our users. We practiced separation of duties to bring a peer group to review and approve the security practices in building and operating our CSB. Fortunately, there were only a couple of gaps that the team addressed quickly.

We hope this blog series helps many of you take on your own DevSecOps journey and that you find the practices we shared valuable.

To Recap,

Part 1 of our series shares the what and why on DevSecOps including our Manifesto,

Part 2 shares the Hack-a-thon approach to building the Security Guard-rails,

Part 3 shares the automation and tooling, and

Part 4 is about the valuable lessons learned and success factors

Looking ahead, we will be expanding on the DevSecOps practices to drive more automation across a Multicloud environment and also embedding more automation in the Continuous Integration/Continuous Delivery pipelines. I firmly believe that Security Automation integrated into DevOps workflows to address security more pervasively, consistently and quickly is the holy-grail to business agility, security talent shortage and dealing with increasing security threats.

We are looking forward to hearing your feedback and if there are practices you have put in place that can make us all more effective. It certainly takes a village and as our CISO often says – “getting everybody on the boat,” is key to our success!