This post is co-authored with Levi Gundert and Andrew Tsonchev.
Update 2014-03-21: For clarity, the old kernel is a common indicator on the compromised hosts. We are still investigating the vulnerability, and do not yet know what the initial vector is, only that the compromised hosts are similarly ‘old’.
Update 2014-03-22: This post’s focus relates to a malicious redirection campaign driven by unauthorized access to thousands of websites. The observation of affected hosts running Linux kernel 2.6 is anecdotal and in no way reflects a universal condition among all of the compromised websites. Accordingly, we have adjusted the title for clarity. We have not identified the initial exploit vector for the stage zero URIs. It was not our intention to conflate our anecdotal observations with the technical facts provided in the listed URIs or other demonstrable data, and the below strike through annotations reflect that. We also want to thank the community for the timely feedback.
TRAC has recently observed a large malicious web redirect campaign affecting hundreds of websites. Attackers compromised legitimate websites, inserting JavaScript that redirects visitors to other compromised websites. All of the affected web servers that we have examined use the Linux 2.6 kernel. Many of the affected servers are using Linux kernel versions first released in 2007 or earlier. It is possible that attackers have identified a vulnerability on the platform and have been able to take advantage of the fact that these are older systems that may not be continuously patched by administrators.
The attack itself happens in multiple stages. Attackers compromise an existing website, append a line of JavaScript to multiple .js files hosted on the site. This causes visitors to load and run a new JavaScript file served from a second compromised host. Utilising a two stage process allows attackers to serve up a variety of malicious content to the visitor. We observed the second stage sites serving what appears to be pay per view fraud pages, where the visitor’s browser loads multiple advertisements to generate revenue for the attacker. However, there is anecdotal evidence that visitors have been infected with Trojan malware as part of this final step.
The line appended to the .js files takes the form of:
Two comments containing a hexadecimal colour reference flank an instruction to the browser to download JavaScript from a PHP file on a second compromised website. The name of this PHP file always follows the same pattern: 8 mixed case alpha numeric characters with a parameter accepting an eight digit id value. The PHP file only responds once per ID value. If accessed more than once, the page returns a 403 forbidden error, otherwise malicious JavaScript is returned to be executed in the visitor’s browser.
Many of the affected hosts have been identified as compromised and cleaned.
An example response message from a cleaned tier one host.
AV products may detect the JavaScript redirect as being similar to that previously used in the Blackhole exploit kit. However, we have no evidence to suggest that this campaign is related to Blackhole rather than an example of code reuse. Indeed, this may be related to the Mesh Network identified by Sucuri.
The speed of spread of this attack has been dramatic, with almost 400 distinct hosts being affected each day on March 17 & 18.
At the time of writing, we have identified in excess of 2700 URLs that have been utilised in this attack. The attackers have subverted existing, legitimate websites to affect unsuspecting users. Security awareness campaigns that train users to be wary of unknown websites may not be effective against trusted websites that become compromised to serve malware. Although users of Cisco’s Cloud Web Security solution are protected from this attack, we observe that approximately 1 in 15 of our clients have had at least one user who has been intercepted attempting to request an affected URL.
The servers affected by the attack are distributed throughout the world, with a particularly high incidence in Germany and USA.
This large scale compromise of an aging operating system highlights the risks posed by leaving such systems in operation. Systems that are unmaintained or unsupported are no longer patched with security updates. When attackers discover a vulnerability in the system, they can exploit it at their whim without fear that it will be remedied. In April 2014, Windows XP will become unsupported. Organisations urgently need to review their use of unsupported systems in operation. Such systems need to be upgraded where possible, or regularly monitored to detect compromise. Organisations should consider their exposure to risks from the use of unsupported systems by partners and suppliers, in addition to the dangers of user interaction with such systems over the internet.
Large numbers of vulnerable unpatched systems on the internet are tempting targets for attackers. Such systems can be used as disposable one-shot platforms for launching attacks. This makes it all the more important that aging systems are properly maintained and protected.
Identified tier 0 and tier 1 affected web sites.
Erm… Out of the 40 subversions of 2.6, which version is vulnerable? Does the attack still work if *only* the kernel is updated to 3.0?
Thanks for the comment. We saw affected machines with a whole range of kernel 2.6 subversions. Version 2.6.18 appeared to be particularly prevalent.
I suspect that we’re seeing this as a marker for older machines rather than as an indication of a kernel vulnerability. It is likely that the attackers have identified a vulnerability in some software that is common to the affected servers that we haven’t found or can’t see.
I doubt that upgrading the kernel will magically resolve the problem, but I’m sure that diligently keeping all the installed software including the kernel up to date and fully patched will minimise your exposure to such breaches.
2.6.18….as in RHEL 5’s currently supported kernel version?
Even if the vulnerability is in the stock 2.6.18, Redhat has 2.6.18+BunchaPatches and is unlikely to have the same holes.
If you think about a webserver, it’s much more likely to be a webserver hole than a kernel hole. The kernel just doesn’t do all that much processing of a request for there to have a hole there. Most of the parsing happens in userland, most likely httpd.
Saying “kernel 2.6” is just a good way of dating the installs until you figure out where the real issue is. Is it http? Is it glibc? is it some apache module?
Can you provide information on the initial attack vector that allows compromise of the server? The mention of the kernel gives me the impression that the servers are not being compromised via the HTTPd, but wording later in the article calls that assumption into question. Is this an article about compromised web servers hosting client side attacks, or about server attacks that then have the secondary function of hosting client side attacks.
Thanks.
We haven’t identified the initial attack vector. We have no reason to suspect that the attack isn’t via http. I’d be very interested to hear from any affected sys admins if they identify how the attackers gain access.
To my eyes the attackers motivation appears to be to redirect as many web visitors as possible to their second stage compromised hosts. Possibly this two step process is to introduce more flexibility into the types of malicious payloads that can be delivered, or it may indicate that there are two separate malicious actors with two separate revenue models.
So this article is just click bait with no facts, just “anecdotes”. Has the gutter level now been reached where you post scare stories with no facts?
Isn’t it irresponsible to make an accusation against a particular kernel, rather than, say, a version of Apache? You clearly have zero information about the initial attack vector yet you leap straight in and claim that it is somehow a kernel defect. This is irresponsible, and damages Cisco’s reputation.
Thanks for taking the time to comment. We don’t know what particular software is being exploited in this attack. My best guess is that the kernel version is a marker for some other installed software that is unpatched or for which a vulnerability has been discovered.
I’m certainly not suggesting that the kernel is being exloited.
So since you clearly have very little concrete information (except some fancy charts) why don’t you either eliminate your conjecture (ie guesswork) or reduce it to the lowest common denominator such as “… some computers have a vulnerability”. Delete the Linux kernel reference promptly and your clumsy mistake might not tarnish your professional reputations so much. Or maybe you have already shot yourselves in the foot at very close range.
Are you providing a list of the affected URLs?
There are the links to two text files containing lists of the affected sites at the bottom on the page.
http://blogs.cisco.com/wp-content/uploads/compromise_1.txt
http://blogs.cisco.com/wp-content/uploads/compromise_2.txt
The first site on compromise_1.txt seems to be running “Apache/2.2.26 (FreeBSD) DAV/2 mod_ssl/2.2.26 OpenSSL/0.9.8y”, which does not quite sound like it’d be running Linux at all. As others have already pointed out, I would not blame this on a Linux kernel bug yet.
We took a look at your list of affected websites and they are not even all running Linux. http://www.whitefirdesign.com/blog/2014/03/21/ars-technica-and-cisco-provide-another-example-of-bad-security-reporting/
The 2.6 kernel series ran from 2003 – 2011 with plenty of patches. For a production system, anything from 2006 on isn’t really old. The way this article leads off is very misleading. You make it sound like it is the kernel itself that is the problem, with no direct proof. To quote you:
“I suspect that we’re seeing this as a marker for older machines rather than as an indication of a kernel vulnerability. It is likely that the attackers have identified a vulnerability in some software that is common to the affected servers that we haven’t found or can’t see.”
That is a very confusing statement to make in face of the article. You also keep referring to web servers, what web server are we dealing with? People are assuming Apache, but there are many more web servers for Linux available.
This blog post slowly starting to trend in tech circles and a lot of bloggers are confused. Maybe re-write the whole thing?
Although it’s good to know that users of Cisco’s Cloud Web Security solution are protected from this attack.
Provide the CVE or it doesn’t exist.
A vulnerability of this magnitude would have a CVE. Responsible Disclosure guidelines *strongly* prefer that you disclose vulnerabilities to the vendor before notifying the public. This gives the vendor a chance to fix the problem before it becomes more widely known to the hordes of script kiddies. The timeline of this fix depends on the severity of the problem and the number of attacks out there in the wild.
In addition, this procedure reduces the change of a security researcher making a mistake. Cisco can issue CVEs, so why didn’t it do so in this case? If normal Disclosure guidelines are not followed, then it feels like the author is trying to gain their 15 minutes of fame by making big claims.
Please don’t go spreading rumors with vague information on a Friday afternoon.
As critical as I was of the original article, the post by ‘CVE’ is just ill informed.
“Provide the CVE or it doesn’t exist.”
Really? Without access to the compromised sites and related logs Cisco would not be able to identify the access vector even if the host was compromised via a well known vulnerability. Each of the sites could have multiple well known vulnerabilities and the unauthorized access could have occurred in slightly different ways for each. There is also the limited possibility of the use of an attack vector that is not publicly known.
I don’t know any of the people behind the article, but I imagine that they had two reasons for writing the article:
1. Raise public awareness of a threat with interesting characteristics.
2. Raise awareness of their products and services.
I would hazard to guess that if they knew of the attack vector that publishing it here would support both of those goals. Even without the vulnerability information this article still has value for incident response in that it provides context, scope, and a potential timeline.
An article on a vulnerability should contain solid information on an attack vector. Lacking this information leads me to wonder if the stated compromised sites have all been uniformly compromised or if this situation is the result of multiple vulnerabilities. As it stands this is mostly FUD.