1 Malware Analysis and Instrumentation Andrew Bernat and Kevin Roundy Paradyn Project Paradyn / Dyninst Week Madison, Wisconsin May 2-4, 2011

Forensic analysts need help Malware Analysis and Instrumentation 2 90% of malware resists analysis [1]  Malware attacks cost billions of dollars annually [2]  65% of users feel effect of cyber crime [3]  69% cybercrimes are resolved [3]  28 days on average to resolve a cybercrime [3] [1] McAfee [2] Computer Economics [3] Norton a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e c0 73 1c a d8 6a d0 56 4b fe af 40 0c b6 f f5 07 b c 85 a5 94 2b 20 fd 5b 95 Malware Binary

Malware Analysis and Instrumentation 3 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e c0 73 1c a d8 6a d0 56 4b fe af 40 0c b6 f f5 07 b c 85 a5 94 2b 20 fd 5b 95 Malware Binary Binary code identification Control- and data-flow analysis Instrumentation Effectiveness on malware The needed toolbox Forensic analysts need help

Malware Analysis and Instrumentation Dyninst Dyninst is a toolbox for analysts 4 program binary 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e c0 73 1c a d8 6a d0 56 4b fe af 40 0c b6 f f5 07 b Dyninst CFG loop, block, function, instruction instrument- ation function replace- ment call stack walking forward & backward slices loop analysis process control library injection symbol table reading, writing binary rewriting machine language parsing Control flow analyzer Instrumenter Data flow analyzer

Analysis tool Dyninst Dyninst is a toolbox for analysts Malware Analysis and Instrumentation Mutator  Specifies instrumentation  Gets callbacks for runtime events  Builds high-level analysis program binary 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e c0 73 1c a d8 6a d0 56 4b fe af 40 0c b6 f f5 07 b Dyninst Control flow analyzer Instrumenter Data flow analyzer CFG 5 loop, block, function, instruction instrument- ation function replace- ment call stack walking forward & backward slices loop analysis process control library injection symbol table reading, writing binary rewriting machine language parsing

Analysis tool Dyninst is a toolbox for analysts Malware Analysis and Instrumentation 6 Analysis of network communications Code visualizations Time bomb detection and analysis Identification of stolen data Reports on anti- analysis techniques printf(…) counter++ if (pred) callback(…) getTarget(insn) Code snippets Mutator  Specifies instrumentation  Gets callbacks for runtime events  Builds high-level analysis program binary 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e c0 73 1c a d8 6a d0 56 4b fe af 40 0c b6 f f5 07 b CFG Dyninst Control flow analyzer Instrumenter Data flow analyzer

Analysis tool Dyninst Dyninst on malware Malware Analysis and Instrumentation 7 printf(…) counter++ if (pred) callback(…) getTarget(insn) Code snippets Mutator  Specifies instrumentation  Gets callbacks for runtime events  Builds high-level analysis Malware defeats static analysis & is sensitive to instrument- ation malware binary 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e c0 73 1c a d8 6a d0 56 4b fe af 40 0c b6 f f5 07 b CFG Analysis of network communications Code visualizations Time bomb detection and analysis Identification of stolen data Reports on anti- analysis techniques Analysis of network communications Code visualizations Time bomb detection and analysis Identification of stolen data Reports on anti- analysis techniques Control flow analyzer Instrumenter Data flow analyzer

Analysis tool Dyninst Control flow analyzer Instrument- er Data flow analyzer Dyninst on malware Malware Analysis and Instrumentation 8 printf(…) counter++ if (pred) callback(…) getTarget(insn) Code snippets Mutator  Specifies instrumentation  Gets callbacks for runtime events  Builds high-level analysis Malware defeats static analysis & is sensitive to instrument- ation malware binary 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e c0 73 1c a d8 6a d0 56 4b fe af 40 0c b6 f f5 07 b CFG SR-Dyninst static-dynamic analysis Analysis of network communications Code visualizations Time bomb detection and analysis Identification of stolen data Reports on anti- analysis techniques Control flow analyzer Sensitivity Resistant Instrumenter Data flow analyzer

Outline Malware Analysis and Instrumentation 9 Anti-analysis tricks Hybrid static-dynamic analysis Sensitivity resistance Results H.A. Anti S.R. Res. 9

PC-sensitive code Obfuscated control flow Unpacked code Overwritten code Anti-patching Address-space probing PC-sensitive code call-pop pairs, return-address manipulation, call-stack tampering & probing Anti-analysis tricks Malware Analysis and Instrumentation 10 Obfuscated control flow indirect control flow, stack tampering, overlapping code, signal-based ctrl flow Unpacked code all-at-once, block-, loop-, function-at-a-time, to empty or allocated space Overwritten code single operand or opcode, whole instruction, function, code section, buffer Anti-patching checksum whole regions, probe for patches, use code as data, move stack ptr Anti Address-space probing scans & probes of locations that should be un-allocated Anti-analysis Anti-instrumentation

a0b0c0d e80300 e9eb045d4555c3 CALLJMP 40d00a459dd4f7 JMPPOPINCPUSHRET 40d00eebp anti-patching storm worm Obfuscated control flow Malware Analysis and Instrumentation 11 obfuscated control flow 40d002 address-space probing unpacked code overwritten code obfuscated control flow Entry Point pc-sensitive code Anti

storm worm Unpacked code Malware Analysis and Instrumentation 12 Entry Point 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e c0 73 1c a d8 6a d0 56 4b fe af 40 0c b6 f f5 07 b c 85 a5 94 2b 20 fd 5b 95 e7 c a d9 83 a1 37 1b 2f b c 22 8e obfuscated control flow unpacked code obfuscated control flow Anti 12 anti-patching address-space probing overwritten code pc-sensitive code

Overwritten code Malware Analysis and Instrumentation 13 Upack packer obfuscated control flow overwritten code obfuscated control flow Anti 13 anti-patching address-space probing pc-sensitive code unpacked code 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e c0 73 1c a d8 6a d0 56 4b fe af 40 0c b6 f f5 07 b c 85 a5 94 2b 20 fd 79 5e c0 73 1c a d8 5b 95 e7 c a d9 83 a1 37 1b 2f b c 22 8e Entry Point

PC Sensitive code Malware Analysis and Instrumentation 14 obfuscated control flow overwritten code obfuscated control flow Anti 14 anti-patching address-space probing pc-sensitive code unpacked code Local Data Access call pop esi add esi, eax mov ebx, ptr[esi] data Use call to get current PC Pop PC into register Construct pointer and dereference e.g., ASProtect

anti-patching obfuscated control flow Anti-patching Malware Analysis and Instrumentation 15 checksum routine protected code xor eax, eax cmp eax,.chksum jne.fail e.g., PECompact Checksumming detects instrumentation [Aucsmith 96] add eax, ptr[ebx] add 4, ebx cmp ebx, 0x41000 jne.loop jmp instrument- ation is detected passfail calculate checksum of protected region compare to expected value Anti 15 address-space probing unpacked code overwritten code pc-sensitive code

Address-space probing Malware Analysis and Instrumentation 16 obfuscated control flow overwritten code obfuscated control flow Anti 16 anti-patching address-space probing pc-sensitive code unpacked code segv_handler() { ptr += PAGESIZE; goto RESTART: } int *ptr = 0; sigaction(SIGSEGV, segv_handler); while(1) { RESTART: *ptr; ptr += PAGESIZE; } data code instrumentation Memory Scan

Malware Analysis and Instrumentation 17 Code discovery algorithm Hybrid algorithm: ? ? Parse from known entry points Instrument control flow that may lead to new code Resume execution H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

Malware Analysis and Instrumentation 18 Code discovery algorithm ? Parse from known entry points Instrument control flow that may lead to new code Resume execution ? Hybrid algorithm: H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

Malware Analysis and Instrumentation 19 Code discovery algorithm ? Parse from known entry points Instrument control flow that may lead to new code Resume execution ? Hybrid algorithm: H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

Malware Analysis and Instrumentation 20 Code discovery algorithm ? Parse from known entry points Instrument control flow that may lead to new code Resume execution ? Hybrid algorithm: H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

Malware Analysis and Instrumentation 21 Code discovery algorithm Parse from known entry points Instrument control flow that may lead to new code Resume execution ? Hybrid algorithm: H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

Malware Analysis and Instrumentation 22 Instrumentation-based discovery H.A. Invalid control transfers Indirect control transfers Exception-based control transfers push eax ret call Invalid Region call ptr[eax] ? jmp eax ? xor eax, eax mov ebx, ptr[eax] Exception Handler

Overwritten code discovery Malware Analysis and Instrumentation 23 Dyninst write RWX 23 H.A. RWX

Update after overwrite 1.Handle overwrite signal a)instrument write loop exits b)copy overwritten page c)restore write permissions d)resume execution 2.Update CFG when writes end a)remove overwritten and unreachable blocks b)parse at entry points to overwritten regions c)remove write permissions d)resume execution Overwritten code discovery Malware Analysis and Instrumentation 24 Dyninst R-X code write handler CFG update routine write Update after overwrite 1.Handle overwrite signal a)instrument write loop exits b)copy overwritten page c)restore write permissions d)resume execution 2.Update CFG when writes end a)remove overwritten and unreachable blocks b)parse at entry points to overwritten regions c)remove write permissions d)resume execution cb RWX cb R-X 24 H.A.

Dyninst Overwritten code discovery Malware Analysis and Instrumentation 25 Update after overwrite 1.Handle overwrite signal a)instrument write loop exits b)copy overwritten page c)restore write permissions d)resume execution 2.Update CFG when writes end a)remove overwritten and unreachable blocks b)parse at entry points to overwritten regions c)remove write permissions d)resume execution R-X RWX code write handler CFG update routine cb write cb 25 H.A.

PC-sensitivity analysis Malware Analysis and Instrumentation 26 SR-Dyninst S.R. call... data... pop esi add esi, eax mov ebx, ptr[esi]... process main: reloc_main: push jmp 0 pop esi add esi, eax mov ebx, ptr[esi]... Relocate Analyze

Anti-anti patching Malware Analysis and Instrumentation 27 S.R. checksum routine xor eax, eax cmp eax,.chksum jne.fail add eax, ptr[ebx] add 4, ebx cmp ebx, 0x41000 jne.loop passfail data code instrumentation patch add 4, ebx cmp ebx, 0x41000 jne.loop emulate (add eax, ptr[ebx]) restore state save state jmp shadow memory

Address-space scanning Malware Analysis and Instrumentation 28 S.R. scan routine xor eax, eax call chk_mem mov ptr[eax], ebx add 4, eax cmp eax, 0x0 jne.loop passfail data code instrumentation patch add 4, eax cmp ebx, 0x0 jne.loop emulate (mov ptr[eax], ebx) restore state save state jmp segv_handler... dyn_segv_handler...

Dyninst SR- Dyninst x x √ √ √ x √ √ √ √ √ √ yes Malware Analysis and Instrumentation 29 The packers we’re studying [1] Packer (r)evolution. Panda Research, Two-month average Feb-March Packer Malware market share [1] 0.13%MEW 0.17%WinUPack 0.33%Yoda's Protector 0.37%Armadillo 0.43%Asprotect 1.26%FSG 1.29%Aspack 1.74%nPack 2.08%Upack 2.59%PECompact 2.95%Themida 4.06%EXECryptor 6.21%PolyEnE 9.45%UPX 0.89%Nspack Res. Self- modifying yes Anti instru- mentation yes Obfuscated yes √ √ √ anti-debugging techniques

 Reduced relocation overhead despite emulation  Better handling of program features  Exceptions  Indirect control flow Malware Analysis and Instrumentation 30 Improved Dyninst overhead Res.

Malware Analysis and Instrumentation 31 Conclusion SR-Dyninst gives you  All the benefits of Dyninst on malware  Safer instrumentation on normal binaries Ongoing work  Anti-debugger techniques  More descriptive CFGs  Automated defensive-mode activation  SR-Dyninst in next Dyninst release