Using Memory Management to Detect and Extract Illegitimate Code for Malware Analysis Carsten Willems 1, Thorsten Holz 1, Felix Freiling 2 1 Ruhr-University Bochum, Germany 2 University of Erlangen, Germany Annual Computer Security Applications Conference (ACSAC) 2012 左昌國 ADLab, NCU-CSIE

Introduction Related Work Model and Definitions Approach Implementation Application to the Analysis of PDF Documents Detection Evaluation Extraction Evaluation Conclusion Outline 2

Exploiting Software The ultimate aim is to perform malicious computation Executing illegitimate code (a.k.a. shellcode) Countermeasures in OSs Data Execution Prevention (DEP) Address Space Layout Randomization (ASLR) Above countermeasures do not help in the analysis of illegitimate code Introduction 3

CWXDetector A tool for the analysis of malware for the Windows OS Dynamic analysis for detecting and extracting illegitimate code Using memory management techniques For analysis CWXDetector is not meant to protect a system But to monitor and analyze the illegitimate code Introduction 4

Limitations Dynamic analysis Could get incomplete results Incapable to detect malicious code embedded in arbitrary data (only those being executed) Malicious computation does not always imply the existence of illegitimate code Return Oriented Programming (ROP) JIT-spraying Introduction 5

Preventive Measures Detection of Illegitimate Code Extraction of Illegitimate Code Related Work 6

Microsoft EMET tool (Enhanced Mitigation Experience Toolkit) emet-v3.aspx emet-v3.aspx CFI Cons: Self-modifying or dynamically created code No assistance to further analysis Related Work – Preventive Measures 7

Static signatures “sled component” searching Heuristics-based approaches Cons: Not generic Have to be extended when new anti-detection come up Related Work – Detection of Illegitimate Code 8

OllyBone OmniUnpack Renovo Cons: OllyBone: debugger-driven malware analysis OmniUnpack: rely on signatures, FP Renovo: requiring emulation environment Related Work – Extraction of Illegitimate Code 9

Attacker Model A remote attacker provides some malicious piece of data to exploit a vulnerability application Resulting in the execution of shellcode Not single staged full-ROP/JIT-sprayed attacks Illegitimate code (ILC) Code that is not legitimate (would not execute if functions properly) Model and Definitions 10

Enforcing an Invariant When a vulnerability is exploited, the control flow is redirected to one of the following locations: ILC on the stack (buffer overflow) ILC in the heap (heap-spraying) ILC in a static data area (exploiting a static data buffer) The approach enforces the following invariant All ILC resides in non-executable memory  all execution attempts of ILC will result in page fault exception Approach 11

Trusted Files and Functions All existing files are trusted before analysis. All created or modified files are untrusted during later operation. Trusted memory modification functions Trusted callers Approach 12

Memory Protection Modifications Only trusted callers can allocate executable memory. Only trusted callers can modify existing memory to being executable. Only trusted files can be mapped into executable memory.  all attempts that violate these rules are intercepted (hooked) and the resulting memory regions becomes non-executable. Exception: If the target memory belongs to a mapped trusted file (writable), the executable right has to be removed. Enforcing W ^ X Approach 13

Custom Page Fault Handler When page fault triggered Check if it is related to the system If so, Dumping memory Modifying the memory to executable and continuing Multi Version Dumping Different versions of each executed page may be created Compare each dumped record Approach 14

CWXDetector For x86 version of Windows XP Memory Function Hooks NtAllocateVirtualMemory NtProtectVirtualMemory NtMapViewOfSection Checking the Caller Trace the user-mode call stack Custom Page Fault Handler Hook MmAccessFault Check if the fault was cause by execute operation Check if the fault address resides in user-space Additional NtCreateFile NtCreateProcess Implementation 15

32bit Windows XP SP2 Adobe Acrobat Reader versions 6.0.0, 7.0.0, 7.0.7, 8.1.1, 8.1.2, 8.1.6, 9.0.0, 9.2.0, and Foxit Reader version installed the customized page fault handler and the system hooks 2. started the particular viewer application 3. disabled DEP for the viewer application 4. opened the PDF document 5. enforced the invariant: new allocated memory and modified memory Application to the Analysis of PDF Documents 16

6. if the execution of ILC was detected, dumped the memory page to a file, created a log entry, and modified the related PTE to being executable. Check the dumped file for patterns  marked as “PATTERN” 7. if a new process was created by the PDF viewer  marked as “PROCESS” and prevent the process from spawning 8. if a dialog window was shown  marked as “DIALOG” and simulated a user input to close the window 9. “CRASH” 10. time out “NOTHING” Application to the Analysis of PDF Documents 17

The log file contains information about: All attempts to allocate executable memory invoked by untrusted callers All attempts to modify existing memory to being executable invoked by untrusted callers All attempts to execute memory that contains ILC All created files All created processes All shown user dialog windows Application to the Analysis of PDF Documents 18

Every PDF file ended up with a tuple (d, c) d: whether illegal code was detected c: {PATTERN, CRASH, PROCESS, DIALOG, NOTHING} (d, c) > (d’, c’) iff either d had detected ILC and d’ not, or (if d = d’) c > c’ PATTERN > CRASH > PROCESS > DIALOG > NOTHING Application to the Analysis of PDF Documents 19

Determining Trusted Callers Identifying all the functions from all trusted files that are used to produce executable memory Loader-related function in ntdll.dll Tested benign PDF documents and manually inspected the function calls Application to the Analysis of PDF Documents 20

Benign PDF Sampleset Retrieved URL of the TOP 5000 sites from alexa Queried Google for the first 10 PDF documents on each site Using tool pdfid, selected all documents which contained JavaScript, OpenActions or some other extended PDF features Uniformly picked random samples from other files to 7,218 samples The benign set on the system No ILC execution detected  FP 0% Detection Evaluation 21

Malicious PDF Sampleset 7,278 known malicious PDF documents form a well-known AV vendor all their valid PDF samples from Jan Detection Evaluation 22

Detection Evaluation 23

15 “CRASH”: All performed invalid memory accesses (before executing ILC) 33 “PROCESS”: Using regular build-in features to create IE or OE with special crafted parameters 295 “DIALOG”: Invalid embedded JavaScript code Social engineering 154 “NOTHING”: Required specific environment Detection Evaluation 24

Detection Evaluation 25

Detection Evaluation 26

Adobe Acrobat Reader ,869 samples Quality: Valid x86 instructions (code ratio) Contained strings Partitions Extraction Evaluation 27

Analyzed all 4,869 PDF samples and got 2 versions of partitions Initial partition Final partition Code ratio determined by IDA Pro Valid strings Extraction Evaluation 28

Extraction Evaluation 29

Initail partitions 7,807 strings 1,866 URLs Final partitions 8,676 strings 2,280 URLs Extraction Evaluation 30

A generic and automatic method to detect and extract illegitimate code Effective in supporting malware analysis Also good detection rates Conclusions 31