Cisco Blogs

Cisco CSIRT on Advanced Persistent Threat

March 23, 2011 - 1 Comment

For corporations, Advanced Persistent Threat (APT) is a widely publicized yet little understood topic.  Does it exist?  Is it a real threat?  How can an organization tell if it is impacted?

The Cisco Computer Security Incident Response Team (CSIRT) is a global team of information security professionals responsible for the 24/7 monitoring, investigation and response to cyber security incidents for Cisco-owned businesses. CSIRT engages in proactive threat assessment, mitigation planning, incident detection and response, incident trending with analysis, and the development of security architecture. This article will provide the Cisco CSIRT team’s perspective on APT, and is the fifth in a series of blog posts on related issues from CSIRT’s point of view.  As with the other posts, provided here are some real-world examples and techniques that will hopefully help organizations utilize existing tools and processes, or even understand gaps in security infrastructure.  Read on to find out more.

In the past, people wanting to steal corporate secrets needed physical access to smuggle data.  Think of the scenario played out in movies of an agent taking pictures of pages of documents using a micro camera , or physically breaking into a facility to steal data. The interconnectivity of the Internet with corporate networks has possibly forever changed the need for that scenario. Exposure of organizations’ sensitive secrets is no longer limited to physical facilities, it is now extended to all computer device endpoints, desktops, mobile phones, web servers and network devices – the entire connected network. With the new world of interconnectivity, we are bringing together bad guys with their targets in new and unforeseen ways. To make matters worse, often parts of this network ecosystem are not nearly as well protected as the physical. As these changes/exposures have happened at Internet speed, companies’ funding and focus on the protection of the cyber area is still dwarfed by the attention to the physical. This inequity of a large footprint of exposure along with inconsistent controls and protection has provided cyber criminals of all levels with a wild west bonanza of criminal opportunity.

To begin, I want to clear up a myth on APT. If anyone attempts to sell your organization on a hardware or software solution for APT, they either don’t understand APT, don’t really understand how computers work, or are lying – or possibly all three.  If there were a way to identify/detect APT that could be written on an ASIC or software signature that you deploy, it wouldn’t be Advanced Persistent Threat. It would be EDPTD (Easy to Detect Persistent Threat for Dummies) or perhaps Annoyingly Persistent Threat. Unfortunately, taking advantage of the extreme and complex nature of such a threats, parts of the security community attempt to sell ineffective APT defense with promises of alchemy.

Possibly the best documented recent APT case was the “Google Hack” or “Aurora.”  It used multiple unknown, undetected zero-day attacks, and hid in the noise of typical malware present in all organizations.  During the aftermath of the public release of information about this intrusion, many CSIRT team members discussed the impacts to their constituents and the appropriate actions for moving forward.  In one discussion, one of the attendees offered, “we have always sent infections to be re-imaged to mitigate risk/impact.  Now I am concerned that we may have been destroying the only link or evidence of a much deeper systemic problem that, until fixed, will be reused over and over again.  How can I tell the difference between an APT infection that I should monitor and investigate forensically versus the normal malware I send to have reloaded?” Like many good and important questions, there are no easy answers for this.  As previously stated, there is no magic software to fix this issue and it is often extremely difficult to tell the difference between the normal malware noise and more advanced attacks.  With APT, like any other hard problem, the solutions may be complex, but the methodology to solve is simple: identify what your available options are and then execute.

To illustrate that methodology, I will go through a couple of examples. At Cisco we have on average 120,000 Windows machines connected to our desktop networks. A certain percentage of these machines suffer from malware. We can’t do deep forensics and network logging/analysis on all of the infected machines. However, that doesn’t mean we can’t do it on some of them. Three years ago we instituted a program to provide these deeper analytic network and system forensics to a select group of employees that are likely targets for APT.  We looked at a subset of people that have access to more interesting data. For this group we absolutely follow up and do this more advanced watching and investigation. This has both detected and prevented some APT type activities.  Did it detect and stop all of them? The answer to this question is unfortunately somewhat easier: no, it did not. We have very valuable data, and remembering the advanced and persistent part of this threat, us interrupting some of the attempts does not stop the attack.  Another example: as of today, many APTs use PDFs to spread. This methodology has a high success rate.  If you have the ability, capture and store all PDFs that come into your company over email, along with the associated email headers.  On a regular basis, do some automated, additional checking beyond your company’s anti-virus solution to help detect PDFs that contain more than the content. Even though normal anti-virus will not detect or stop these threats, often is it relatively easy to see they are modified with simple string searches or running multiple anti-virus scanners. These are just two quick examples, but if you think through your defensive strategies you can apply common sense thinking to come up with many others. You can also view my recent netflow blog entry for other examples.

Most of a company’s ability to detect and respond to these and other sophisticated attacks rely on the effective deployment of a set of well-understood computer security incident response pillars:

  • First, you have to have the ability to produce,  collect and query logs; the more the better, but at least the important ones from a security perspective  (host logs, proxy, authentication and attribution logs).
  • Secondly, you need some form of deep packet inspection that covers all the important choke points on your network.
  • Thirdly, you need the ability to quickly query your network connections or flows through netflow or a netflow-like service across all network choke points. For these three important defensive tools you will have to have the ability to query past, real-time, and set up alerting for future events. Tying this all together you will need analysts and investigators to do the bulk monitoring and deeper investigations. This team and their detection infrastructure has to have the ability to quickly change and adapt to today’s threat.
  • Fourth, and possibly the catalyst to making all of the above work, you will need to develop trust-based relationships with other organizations to share intelligence on events. This is a long process that you will have to purposely resource and tend. A great start to that would be joining an organization like FIRST’s charter is to help facilitate this type of information sharing. When all of these pillars come together in some meaningful way – once you have intelligence on potential issues – I can’t stress enough that you need to be able to quickly, nimbly operationalize this intelligence, and to ensure you have COMPLETE, 100% coverage for the above tools. This is the most important thing you can do for your organization for APT.
  • Lastly, you will need to either have in-house or hire some degree of malware analysis.  Referencing the previous PDF example, as you find one of these and pull apart what it what doing, who it targeted, and where it was connecting, this will provide the most valuable intelligence of how your company is being attacked today. You will take the relevant intelligence and feed in immediately to the above detection infra, which, if consistently deployed, will lead to a complete understanding of the impact of that attack historically, and right now for others that may be similarly affected, with alerting set up for any future occurrence.  By the very nature of APT it will be as unique as possible, and any intelligence you have has a limited lifespan.

One of the other questions I often hear on APT is, “how can I tell if my organization is impacted?”   Evidence for proof of advanced levels of well-funded intrusions are often hard to find, and even harder to attribute. Is that connection to an unknown website  – that we see initiated by malware – evidence of the more  common spam and other criminal bot enterprises, or is it evidence of a more serious company targeting for data? The answer to this question is an unfortunate one. Due to the ubiquitous nature of world network connectivity combined with the prevalence of malware and software/hardware vulnerabilities, if your organization has something in the digital form that can be resold, or is of any real economic interest, you are being targeted. The sophistication and persistence of these attacks are directly related to the value of that data. To give one example, oil and gas data that Exxon has created over years of extremely expensive exploration is publicly discussed as an example of advanced attack with data exfiltration giving a priceless boon in R&D for whoever has that data.

Another common question I hear is, “who are responsible for these APT type attacks?” The answer to this, unfortunately for corp CSIRTs, is of less consequence than it could be if there was more real recourse. Hopefully this is a fluid situation where these types of attacks will eventually come with penalty to the attackers.  Exact attribution is often hard and the list of aggressors in the space is growing – the commonly known and publicized ones are well documented, but there are also many that would be a surprise, using APT as a ways to end. The leaked HBGary emails, for example, revealed that APT services for sale is commonplace.

In conclusion, APT attacks easily go undetected, and are more common and severe than many think.  Organizations are at extremely high risk and need advanced detection capabilities. Takeaways from this article are that if you have something of interest and you’re not seeing APT attacks in your organization, it is probably not that they are not occurring or that you’re safe. It’s more likely that you may need to rethink your detection capabilities. The state of the art in response to APT does not involve new magic software hardware solution divining for APT, but relies more on asking the right questions and being able to effectively use the existing detection tools (logging, netflow, IDS and DPI). Incident response, while important and needed, will never be a fix this issue. The better you architect controls around your data the less your exposure is and corresponding risk. The front line for this problem is being fought at  company level but will only be solved with public private partnership leading to nation level consequence.

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.


  1. Mr. Reid:
    I thoroughly enjoyed your series on netflow. This article also sheds valuable insights on the subject at hand. I like the fact that you bring real-life, actual-use perspective in your articles. Thanks and I look forward to future articles from you.