Malware authors use a variety of obfuscation techniques to foil researchers and operate as covertly as possible on a user’s system. To that end, some of the techniques, like frequent changes of the executable (possibly daily) are designed to obstruct basic detection techniques. Often times, given a specific piece of executable code, it is not trivial to determine if the code is a piece of malware or just a random piece of software. Fortunately, there are variety of techniques to help someone determine if a piece of code is malicious or not.
Many of these techniques partially come from forensics or malware reverse engineering disciplines. Most of these techniques will work on all types of malicious files, although packer detection and entropy will work best on executable files. A previous blog post titled “A Brief History of Malware Obfuscation” by Mike Schiffman provides background information on malware obfuscation. Below, I’ll highlight several of the techniques and give a brief discussion of the good and bad of each technique.
File Checksum Checking Services: -- This is the simplest, perhaps most efficient means to determine if a piece of software is malware or not. This is normally accomplished using an MD5 checksum. Several of these services exist, with the most popular being Virus Total and The Malware Hash Registry. Both Virus Total and Malware Hash Registry run each sample against multiple antivirus scanners and store the results. Therefore, all you need to do is calculate an MD5 checksum for your file and query either of these services to see if the file has previously been identified as malware. There are some subtle differences between the services though. For instance, with Malware Hash Registry it’s simple to submit a query with several MD5s, allowing for some batch or bulk processing. With Virus Total, you can submit the file and they will do the back-end logic of calculating a checksum and returning the results. The benefits of these services is that it is quick and easy; however, with MD5 checksum comparison, it will only tell you if the MD5 matches a previously submitted file that was determined to be malware. This is important. If the sample is relatively new, the likelihood of detection is much lower since it may not have been submitted or detected by the antivirus engines when submitted.
In certain cases, it may be possible that the MD5 check returns a false positive (meaning the sample is not really malware but some engines detect the sample as malware). The likelihood of having a false positive is relatively small, and is usually limited to just a few engines detecting the file, and therefore if there are more than a few engines that have identified the sample as malware it is likely to be malware. Likewise, lack of detection should not mean the sample is not malware; instead it just means that it was not detected as malware. As stated in the introduction, often malware authors will make minor changes or repackage their malware every few days to thwart detection via MD5 checksum or antivirus, and in some cases the malware is repackaged daily. Therefore, often times new samples may not be detected.
Cumulative Anti Virus Testing: Virus Total , NoVirusThanks, Threat Expert, and Jotti provide interfaces to allow the sample to be submitted and actually run against several of the different antivirus engines. If the MD5 of the sample is not found to be malware, this is the best next step. This is still prone to the same false positives and false negatives as the MD5 tests against cumulative antivirus services. This is a big benefit when the md5 returns no results. Once again, newer samples may not be detected, or may have a detection rate that is low enough not to warrant confirmation of malware. A previous blog post details the effectiveness of Antivirus on new malware samples.
Context Triggered Piecewise Hash (CTPH): Context triggered piecewise hashes are more often referred to as “fuzzy hashes.” A fuzzy hash is similar to an MD5 checksum, but it differs in that it compares parts of the file. The most commonly used tool for performing fuzzy hash comparisons is ssdeep. Since fuzzy hashing is good at finding files where most of the file is the same and a few sections are different, fuzzy hashing is good at finding files that are similar to other known files. Most often when using ssdeep, the hash itself is not important, just the percentage similar to other files. For fuzzy hashing to be effective it must be compared to something, and therefore having some sort of malware library greatly increases the effectiveness of using fuzzy hashes to determine if a suspect file is malicious or not. It should be noted that packing renders fuzzy hashing mostly useless since it obfuscates the file; however, if the file is not packed and there is some sort of malware library to use for comparison, fuzzy hashing is a very simple and effective technique.
Packer Detection: Packers condense and obfuscate the original file. To the malware author, packing a file allows the author to disguise the original code. In some cases, packers can be polymorphic and provide different binaries every time the file is packed. The obfuscation provided by packers makes antivirus detection more difficult. In addition, packing often complicates reverse engineering efforts by security researchers, and so packing provides the malware author several benefits. A malware author can pack the same malware with different packers and end up with files that look nothing alike.
The percentage of malware that is packed is significant, with some estimates reflecting that 70-80 % of malware is packed. Meanwhile, some non-malware is also packed, but the percentage is much smaller, perhaps less than 5%. Therefore, if a suspicious file is packed, it is highly likely to be malware. Some other file attributes, such as imports and sections can be used to increase the accuracy, but that will not be covered here.
Several tools can be used to determine if a file is packed. PEiD is a common tool that detects a large number of packers. Since many forms of packer detection rely on string matching certain characters within the file, it can be prone to false positives. If a malware author develops a custom packer, it is likely to create a false negative condition. So if a file does not appear to be packed, it doesn’t necessarily mean the file is not malware. It is possible that it is packed with a custom packer, or just not packed.
Entropy: Encryption and compression increase the randomness of characters in a file. Entropy measures the statistical variation of bytes in the stream of data. Packed files display high entropy scores, and therefore measuring entropy can be used somewhat like packer detection, and in particular is useful for identifying a packed file when no signature exists for the file. Since it only measures entropy, it is possible that legitimately compressed and encrypted files could trigger false positives.
Run Time Heuristic Analysis and Sandboxing: This method can be very effective, although it is not practical to someone without the capabilities to run the specimen in a controlled environment and compare the sample’s actions to that of other samples. This method requires a system monitoring component that will report the changes performed by the specimen on the system, as well as what network activity happened while the specimen was running in the controlled environment.
For example, a particular specimen may install itself in certain locations, download specific files and connect to certain hosts. The Malware author may use a custom packer and change how the file is packed, thereby creating an entirely new file that does not resemble any other malware. When the sample is run in the controlled environment, however, it will perform the same actions, and is therefore is the same piece of malware.
Some malware samples contain checks so they will not run in many of these environments. As stated earlier, this is not really practical unless you have a controlled environment in place. With the proper setup this technique can work very well. An alternative that is often easier without the infrastructure in place is to just run the malware on a host and observe its actions. This method will require the host to be re-imaged.
Summary: Unfortunately, there is still no simple way to reliably determine if a suspect file is malware or not. To validate our suspicion that a file is malware, we must use several techniques. Some of the techniques, such as checksums and packer detection, are pretty simple to use, while other techniques, such as runtime analysis, require specific infrastructure or more information (such as entropy). Most of the time, by using these techniques and through a little bit of human interpretation, we can make a pretty good determination of whether a sample is malware and what it may be intended to do.