Product security covers quite a broad spectrum of knowledge areas within the realm of technologies applied to enable communications in this highly connected world. However, there is a natural tendency to first focus on the basic capabilities of the product itself. But later, questions arise such as “Is the product in operation vulnerable and if yes, what are the next steps to protecting against the vulnerability?” or “What can I do if I suspect a security issue with a product?” As much as one would like to sustain 100% immunity against any vulnerability or issue, events happen, inherent product weaknesses are discovered or new attack vectors and methods arise to expose ways to compromise a product’s operation or behavior. At Cisco, the people that rapidly converge on such occurrences or the potential for such occurrences are the Incident Managers (IM) who reside at the core of the Product Security Incident Response Team (PSIRT) within Security Intelligence Operations (SIO). I think it is fascinating how well this team seamlessly executes with the precision, efficacy, and timeliness on a day-in-day-out basis covering a large array of complex hardware, software, and technologies. The IM focuses on driving the underlying processes around the discovery of security disclosures and issues related to Cisco products and networks. I hope you will find that this article provides you with an informative and personal perspective on the IM role that is integral to the ongoing efforts essential to protecting the Cisco customer.
The IM must diligently investigate and report on security vulnerability information and issues related to Cisco products and networks. An IM can be called into action via multiple communications channels including, but not limited to these methods:
- Externally via E-mail or Hotline Call by a Cisco customer, independent customer, etc.
- Internally within Cisco by any group, such as a product business unit.
The steps that an IM performs on a security disclosure or issue can culminate in the publication of one of three PSIRT documents:
The process flow diagram that follows is also available on the Cisco Vulnerability Policy webpage and shows the numerous steps that an IM is engaged in. The IM’s efforts are highly collaborative among several Cisco groups ranging from Security Intelligence Operations (i.e. Applied Intelligence for AMBs) to Development, to Release Operations, and of course the customers.
The IM uses the Common Vulnerability Scoring System (CVSS) as part of the standard process of evaluating reported potential vulnerabilities in Cisco products and determining which vulnerabilities warrant a Cisco Security Advisory or other type of publication. CVSS is used to convey vulnerability severity. More information about CVSS is available at the FIRST.org web site
Similar to my previous article in this series on individual contributing security roles at Cisco, the following section is taken from an interview that I conducted with the goal of providing a closer look into a day in the life of a Cisco PSIRT Incident Manager and also helping those of you considering this type of career in the information security field.
What range of background does an Incident Manager (IM) role have?
A lot of the individuals come from diverse backgrounds. We focus on a range of products, i.e. everything that Cisco does. So someone needs strong investigative skills that can be applied to any product in question. You need to be personable, since you must be able to communicate with anyone at any time. You also need to be able to adapt to a variety of communications, whether it be voice, email, video, in-person, etc. Adapting to the individual’s temperament (i.e. stressful situation) while being able to communicate urgency and importance is an essential soft skill. You also need to be able to converse both technically and non-technically in order to properly translate issues into a form that can be readily understood and applied.
What does an IM do to come up to speed?
Please keep in mind that this not a junior position. An IM is considered a seasoned professional with previous experience within the security arena. That said, we have a lot of documentation for this position covering particular responsibilities from a procedural perspective. In this case, the IM more often than not hits the ground running. We have mentors who are senior members of the team. For example, a mentor will help you write the first few security advisories. We “learn by doing” is pretty much the theme.
What do you like most about your role? (i.e. Why would you want to do this)?
It’s exciting to see the new vulnerabilities as they come along and know that you are protecting customers from the bad guys that want to do harm to their network. I have the opportunity to interact with people from all over the world. You never know who (i.e. customer, developer, product manager, etc.) you are going to work with from anywhere and at anytime. Therefore, this role is both dynamic and challenging at the same time.
Can you share some insights on what your day-to-day core activities involve?
The predominant focus is on managing our caseloads, which involves an average of 20 cases happening at the same time. From a process perspective, we will pick up a customer case, make an initial contact with the reporter and jump into an investigative mode. We also focus on evaluation and assessment of the risk that the issue may pose. I should point out that this happens even when we are participating in major events like Cisco Live or security events.
How does your day usually begin?
We check e-mails to make sure nothing critical happened over night. Then we pull up the case queue/backlog and then prioritize what needs to be worked on for the day. We may also start some advance preparation for meetings that might happen during the day.
What is a key aspect (i.e. qualitative or quantitative) of being successful?
It is imperative to have good organization skills. This includes the ability to plow through email as we typically receive 200-300 per day. Therefore, being able quickly filter, distill, and absorb large volumes of information is critical in order to keep up with the flood of information. Along those lines, you need to also sift through a fair of amount of spam, which can be in various languages. It is also key to filter out what you do not need to give your attention to.
Are there any other aspects of your job that are especially essential as you work with your team members or others that depend on your work?
Your availability is essential since not only do others need to communicate with you, you also need to communicate with them. At times, this involves e-mail communications that underscore the necessity to have excellent written skills. This is highly evident in our most visible communications which is through the PSIRT Security Advisories.
What else can you recommend to others regarding optimizing their practices based on your experience?
We believe in responsible disclosure and that results in providing the information that our customers and partners need. We also want to make sure that we publish information in as timely of a manner as possible to make sure that our customers are protected as much as possible.
What do you like to do for fun when you are not a practicing IM?
I enjoy photography and astronomy. For example, I showed several co-workers the Venus transit (Venus past in front of the Sun for 8 hours) event that won't occur again for more than 100 years. I also enjoy spending time with family at the beach. I also practice Yoga, and was a fixed-wing pilot.
In case you have not already noticed, many of the PSIRT IM’s have been regular contributors to the Cisco Security Blog over the years where they have shared valuable knowledge on a variety of security topics that you might find interesting. Meanwhile, I encourage you to check out the wealth of information and resources available on the SIO portal as well as related security blog articles covering several other roles behind the scenes that are dedicated to delivering early-warning intelligence, threat and vulnerability analysis, and proven Cisco mitigation solutions to help protect networks. I welcome your questions or comments on this or any other product security role at Cisco.