Print3M
's Hub
HomeBlogNotesSecuriTree
All notes

Malware Detection Techniques

1. Static (signature) detection

AV compares the scanned file with a database of known malicious signatures and patterns. The database must be up to date. It tries to find unique strings, checksums, sequences of bytes, file format manipulations or other specific conditions. It looks on the static file not on the running process. E.g. a hardcoded raw shellcode is a very specific sequence of bytes which can be easily detected. In general, static detection is easily circumvented.

YARA is one of the most common tools that are used to build detection rules.

1.1. Hash detection

This is the simplest subset of static detection. It simply compares a hash from known malware database with the suspicious file hash. It's very easy to be evaded: by changing at least 1 byte in the file, the file hash will be completely different.

1.2. IAT checking

Import Addres Table contains function names and DLLs that are used by the process. By checking IAT, a security solution can look for suspicious combination of functions. E.g. a ransomware very often uses a set of file manipulation and cryptografic functions. This is a signature that can be detected.

2. Heuristic detection

Heuristic detection was introduced to spot suspicious characteristics that can be found in new and modified versions of existing malware. The detection process might consist of:

  • Static Heuristic Analysis - decompiling and comparing code snippets to known malware. This is based on the percentage of accuracy between the known malware (stored in the database) and the newly scanned one.
  • Dynamic Heuristic Analysis - the suspicious program is placed inside a controlled VM or a sandbox which is then analyzed for suspicious behaviors. This is why malware developers embed anti-sandbox techniques to detect presence of the sandbox environment.

3. Behavior-based detection

Security solution continues to look for suspicious behavior while the malware is running. Once the suspicious behavior is detected the security solution will conduct an in-memory scan of the running process. If the process is identified as a malicious, it is terminated.

Additionally, some process activities are so suspicious that they can cause the process to be terminated immediately, without any in-memory scan.

To evade this type of detection, the process needs to behave as normal as possible. In-memory scan can be circumvanted with memory encryption.

4. API hooking

It's a technique used to monitor the code execution in real time for malicious behaviors. It works by intercepting commonly abused API functions and analyzing the parameters of their invocation in real time. It allows the security solutions to see de-obfuscated and decrypted content passed to the API functions. EDRs just overwrites the API functions in loaded DLLs which they want to be monitored.

5. Manual analysis

Ultimately, malware can always be reversed manually. There is no ultimate solution to evade manual analysis. There are only techniques to make the analyst's life more difficult: anti-reversing, anti-VM and anti-debugger techniques.