And why 100% detection is grossly misleading

It is with great pride that we received the latest Breach Detection Report from NSS Labs, in which Cisco achieved a 100% detection rate – we simply couldn’t be more pleased to have our products so well-represented and validated in the market, and we truly believe we have the best, most effective security products available today. You can get your complementary copy of the NSS Labs report here.

The Results

“The Cisco FirePOWER 8120 with NGIPS and Advanced Malware Protection received a breach detection rating of 100.0%. The FirePOWER 8120 proved effective against all evasion techniques tested. The solution also passed all stability and reliability tests.”

If you are not familiar with the NSS Labs Breach Detection Report, its simple premise is based on detecting a breach, especially those that bypass traditional detection and protection methods like antivirus and firewalls, and that this detection happens through any means possible. A full description from the 2016 BDS Comparative report for Security from NSS Labs:

“The ability of the product to detect and report successful infections in a timely manner is critical to maintaining the security and functionality of the monitored network. Infection and transmission of malware should be reported quickly and accurately, giving administrators the opportunity to contain the infection and minimize impact on the network. As response time is critical in halting the damage caused by malware infections, the SUT should be able to detect known samples, or analyze unknown samples, and report on them within 24 hours of initial infection and command and control (C&C) callback. Any SUT that does not alert on an attack, infection, or C&C callback within the detection window will not receive credit for the detection.”

This means that the Cisco products detected 100% of the tested breaches within 24 hours, an impressive testament to our commitment to delivering truly effective security to our customers.

The Challenge – Reducing Operational Space

While a tremendous accolade to our engineers, the result is indeed bittersweet. Cisco prides itself in having products that can perform so effectively, but we also work diligently to guide our customers, and a “100% detection” claim without context would confuse security practitioners who look at long periods of arbitrary time, versus reducing the operational space of the adversary.

Therefore, it is questionable whether Cisco or any other vendor should even claim 100% detection as a proof point. Is this a useful measure to push vendors to build better products and provide improved value for our customers? Of course. But in the end, 100% detection of a breach within 24 hours is not what we should be striving for. Asking a simple question illustrates the point well:

If two products scored 100% with product A detecting 100% of breaches within 5 minutes and product B detecting 100% of breaches within 1380 minutes which would you prefer?

Which product do you think the attacker would like to face if given the choice?

Which product would you think the defender would like to have given the choice?

We believe that the time it takes to detect the breach is the better measure and the goal for this measure should be zero minutes. That’s because it would reduce the operational space of the adversary, which is the space and time an adversary has in which to operate after breaching a system. This is far more representative of the effectiveness than the ultimate detection of a breach at some arbitrary future time. Reducing the operational space available to the adversary is what limits the amount of damage done once a system has been breached and it is this time that is the key factor to successfully identifying and mitigating the breach.

Proper Measurement: A ‘For Instance’

How you should measure the effectiveness of a system is to measure the total amount of time it takes the product to detect the totality of tested breaches. For example – If we assign a greater value to faster detection than we do to slower detection then we can assess overall product effectiveness that is weighted by time. In doing this we do not need to impose arbitrary limits on the length a solution can take to detect a breach and we can better represent the value of that detection. We have referred to and reported on our insights to this metric as our Time to Detection (TTD) since December 2014 and have consistently reduced it from a median of 50.2 hours to a median of 13 hours for this reporting period. Data is available in our 2016 Midyear Cybersecurity Report.

I’ve taken the liberty to model how one such assessment for this breach test might work. While there are many models we could apply, I’ve just inverted the time values and used that as a scoring metric to keep it simple. Simply put, if you have 1440 total minutes to detect a breach and you detect it within one minute you will be given 1439 points for the detection.

EG:

Note that both products detected 100% of the breaches within 24 hours though one product performed significantly better in the TTD and thus reduced the unconstrained operational space of the adversary – and the resulting risk and exposure to your business.

While full data is necessary to perform an exact assessment, we can apply a similar approach with the data that is available from this test. I’ve one such summary below and have eliminated product names as to not mislead defenders or violate usage terms.

What do you think?

There are, of course, other considerations that should be taken into account, such as the actual operational cost of the technology, the real impact of false positives, the value of blocking, and the operational burden any given technology will place on your people and processes. Right now, I would like your feedback on this proposed approach and how it relates to your experiences. Do you think there is a better way to score the effectiveness of security products in detecting a breach? What else do you think is needed or can be improved? Please help us understand what key performance indicators you would like to see, and how you currently measure them or think they can be measured so we can help you deliver more accurate and timely representations in our products.

In the meantime, we do maintain our pride over our current test results, because as stated before, it’s currently our industry’s only measurement. Take a look and consider if we had these measures, as well as the ones I mention above, how that could ultimately help us all better advance our defenses against the adversary.