1. RAID 2010 Hybrid Analysis and Control of Malware Barton P. Miller bart@cs.wisc.edu 1 Hybrid Analysis of Program Binaries 1 Kevin A. Roundy roundy@cs.wisc.edu Computer Science Department

2. RAID 2010 2 Need for forensic analysis  Malware attacks cost billions of dollars annually [1]  65% of users feel effect of cyber crime [2]  28 days to resolve an average cybercrime [2] 90% of malware resists analysis [3] 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b fe 92 malware binary Our approach  analyze code before executing it  CFG-based interface for instrumentation  bring malware under analyst’s control [1] Computer Economics. 2007 [2] Norton. 2010 [3] McAfee. 2008

3. RAID 2010 malware binary 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21 Malware analysis factory Hybrid Analysis of Program Binaries 3 SD-Dyninst code coverage instrumentation network call instrumentation Stack trace at 1 st network communication Control flow graph showing code coverage Defensive tactics report  unpacked code  overwritten code  control flow obfuscations Trace of Win API calls

4. RAID 2010 storm worm Obfuscated control flow Hybrid Analysis of Program Binaries 4 Entry Point obfuscated control flow 030405060708090a0b0c0d e80300 e9eb045d4555c3 CALLJMP 40d00a459dd4f7 JMPPOPINCPUSHRET 40d00eebp 40d002 CALL ptr[eax] ? XOR eax,eax MOV ecx,*[eax] exceptionhandler ? handler-based ctrl flow unpacked code overwritten code obfuscated control flow handler-based ctrl flow

5. RAID 2010 storm worm Unpacked code Hybrid Analysis of Program Binaries 5 Entry Point 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21 0c 85 a5 94 2b 20 fd 5b 95 e7 c2 16 90 14 8a 14 26 60 d9 83 a1 37 1b 2f b9 51 84 02 1c 22 8e 63 01 obfuscated control flow handler-based ctrl flow unpacked code overwritten code

6. RAID 2010 Overwritten code Hybrid Analysis of Program Binaries 6 Upack packer 7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e 80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21 0c 85 a5 94 2b 20 fd 5b 95 e7 c2 16 90 14 8a 14 26 60 d9 83 a1 37 1b 2f b9 51 84 02 1c 22 8e 63 01 Entry Point obfuscated control flow handler-based ctrl flow unpacked code overwritten code

7. RAID 2010 Factory results for Conficker A 7 initial bootstrap code packed payload Hybrid Analysis of Program Binaries

8. RAID 2010 Hybrid Analysis of Program Binaries Factory results for Conficker A 8 API func non executed block static block unpacked block

9. RAID 2010 Factory results for Conficker A 9 Hybrid Analysis of Program Binaries Stack-walk of Conficker’s communications thread Frame pc=0x7c901231 func: DbgBreakPoint at 7x901230[Win DLL] Frame pc=0x10003c83 func: DYNbreakPoint at 0x100003c70[instrument.] Frame pc=0x100016f7 func: DYNstopThread at 0x100001670[instrument.] Frame pc=0x71ab2dc0 func: select at 0x71ab2dc0[Win DLL] Frame pc=0x401f34 func: nosym1f058 at 0x41f058[Conficker] Instrument select and perform a stack-walk

10. RAID 2010 Outline Hybrid Analysis of Program Binaries 10 R.W. Par. Related work Hybrid analysis algorithm Parsing Dynamic analysis components Results D.A. H.A. Res.

11. RAID 2010 Non-Defensive Binary Analysis 11 Hybrid Analysis of Program Binaries program binary Process Dynamic instrumenter Static tool static code CFG un-controlled executionpre-execution R.W.  parsing  value-set analysis  binary slicing e.g., Dyninst, CodeSurfer-x86  CFG-based API for instrument- ation e.g., ATOM, Vulcan (static) Dyninst (dynamic)

12. RAID 2010 Static tool analysis resistant binary Hybrid Analysis of Program Binaries 12 obfuscated code static code un-controlled execution Dynamic instrumenter dynamic code Process pre-execution CFG R.W. Non-Defensive Binary Analysis  parsing  value-set analysis  binary slicing e.g., Dyninst, CodeSurfer-x86  CFG-based API for instrument- ation e.g., ATOM, Vulcan (static) Dyninst (dynamic)

13. RAID 2010 un-controlled execution analysis resistant binary Dynamic instrumenter 13 Hybrid Analysis of Program Binaries obfuscated code static code dynamic code Process pre-execution post-execution analysis CFG Trace analysis Trace R.W. Non-Defensive Binary Analysis  Instruction- filter based API for instrument- ation e.g.: PIN, Valgrind, DynamoRIO, DIOTA e.g.: Madou et al. 2005 Quist, Liebrock. 2009

14. RAID 2010 un-controlled execution Our approach 14 Hybrid Analysis of Program Binaries SD-Dyninst obfuscated code static code analysis resistant binary Parser pre-execution Dynamic instrumenter Parser (source,dest) CFG dynamic code Process R.W.  CFG-based API for instrument- ation

15. RAID 2010 Outline 15 Hybrid Analysis of Program Binaries Related work Hybrid analysis algorithm Parsing Dynamic analysis components Results D.A. Res. R.W. P. H.A.

16. RAID 2010 Code discovery algorithm 16 Hybrid Analysis of Program Binaries Hybrid algorithm: ? ? Parse from known entry points Instrument control flow that may lead to new code Resume execution H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

17. RAID 2010 Code discovery algorithm 17 Hybrid Analysis of Program Binaries ? Parse from known entry points Instrument control flow that may lead to new code Resume execution ? Hybrid algorithm: H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

18. RAID 2010 Code discovery algorithm 18 Hybrid Analysis of Program Binaries ? Parse from known entry points Instrument control flow that may lead to new code Resume execution ? Hybrid algorithm: H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

19. RAID 2010 Code discovery algorithm 19 Hybrid Analysis of Program Binaries ? Parse from known entry points Instrument control flow that may lead to new code Resume execution ? Hybrid algorithm: H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

20. RAID 2010 Code discovery algorithm 20 Hybrid Analysis of Program Binaries Parse from known entry points Instrument control flow that may lead to new code Resume execution ? Hybrid algorithm: H.A. instrumentexceptionoverwrite CALL ptr[eax] DIV eax, 0

21. RAID 2010 Outline 21 Hybrid Analysis of Program Binaries Related work Hybrid analysis algorithm Parsing Dynamic analysis components Results D.A. H.A. Res. R.W. P.

22. RAID 2010  Standard control-flow traversal [1]  start from known entry points  follow control flow to find code  New conservative assumption  un-analyzed calls (pointer-based) may not return  New stack tamper detection  backwards slice at return instruction call 40d00a pop ebp inc ebp push ebp ret garbage 22 Hybrid Analysis of Program Binaries Accurate parsing P. [1] Sites et al., Binary Translation. 1993.

23. RAID 2010 Outline 23 Hybrid Analysis of Program Binaries Related work Hybrid analysis algorithm Parsing Dynamic analysis components Results H.A. Res. R.W. P. D.A.

24. RAID 2010 24 Invalid control transfers Indirect jumps/calls Abnormal return instructions push eax ret call 401000 Invalid Region call ptr [eax] ? jmp eax ? Instrumentation-based discovery D.A. Hybrid Analysis of Program Binaries

25. RAID 2010 ? call ptr[eax] findTarget (ptr[eax]) SD-Dyninst process findTarget (ptr[eax]) new target 0x402d8a resume execution call ptr[eax] Instrumentation-based discovery D.A. 25 Hybrid Analysis of Program Binaries

26. RAID 2010 26 SD-Dyninst Overwritten code discovery Overwrite Detection Possible strategies  Check each executed instruction for changes [1]  Monitor writes to code Page-level write detection [2]  Remove write permissions from code pages  Write to code causes exception  Handle exception [1] Royal et al. PolyUnpack. ACSAC ’06 [2] Maebe, De Bosschere. AADEBUG ’03 code write handler write RWE R E RWER E D.A. Hybrid Analysis of Program Binaries

27. RAID 2010 Hybrid Analysis of Program Binaries 27 write SD-Dyninst Overwritten code discovery When to update Cases to consider  large incremental overwrites  writes to data  writes to own page R E code write handler CFG update routine D.A.

28. RAID 2010 Hybrid Analysis of Program Binaries 28 SD-Dyninst Overwritten code discovery When to update Cases to consider  large incremental overwrites  writes to data  writes to own page Delaying the update  until write routine terminates R E CFG update routine code write handler D.A. write

29. RAID 2010 Delayed updates Two components 1.Handle overwrite signal a)instrument write loop b)copy overwritten page c)restore write permissions 2.Update CFG when writes end a)remove overwritten and unreachable blocks b)parse at entry points to overwritten regions c)remove write permissions Hybrid Analysis of Program Binaries 29 SD-Dyninst Overwritten code discovery R E code write handler CFG update routine D.A. write Delayed updates Two components 1.Handle overwrite signal a)instrument write loop b)copy overwritten page c)restore write permissions 2.Update CFG when writes end a)remove overwritten and unreachable blocks b)parse at entry points to overwritten regions c)remove write permissions cb RWE cb R E

30. RAID 2010 Hybrid Analysis of Program Binaries 30 SD-Dyninst Overwritten code discovery Delayed updates Two components 1.Handle overwrite signal a)instrument write loop b)copy overwritten page c)restore write permissions 2.Update CFG when writes end a)remove overwritten and unreachable blocks b)parse at entry points to overwritten regions c)remove write permissions R E RWE code write handler CFG update routine cb D.A. write cb

31. RAID 2010 Exception State eip 401002... eip 402d8a 31 xoreax,eax movecx,*[eax] pusheax... Operating System Handler-based CF obfuscations [1] [1] Popov, Debray, Andrews. Usenix 2007. Danekhar. http://www.codeproject.com/KB/system/inject2exe.aspx 2005.http://www.codeproject.com/KB/system/inject2exe.aspx Monitored Program D.A. access violation handler … mov *[ebp+10],eax mov 402d8a,edx mov edx,*[eax+b8] Hybrid Analysis of Program Binaries

32. RAID 2010 Exception State eip 401002... eip 402d8a 32 xoreax,eax movecx,*[eax] pusheax... Operating System [1] Popov, Debray, Andrews. Usenix 2007. Danekhar. http://www.codeproject.com/KB/system/inject2exe.aspx 2005.http://www.codeproject.com/KB/system/inject2exe.aspx Monitored Program D.A. access violation handler … mov *[ebp+10],eax mov 402d8a,edx mov edx,*[eax+b8] Resolving handler-based CF access violation handler … mov *[ebp+10],eax mov 402d8a,edx mov edx,*[eax+b8] SD-Dyninst instrument exit analyze code at new target Hybrid Analysis of Program Binaries

33. RAID 2010 33 Outline Related work Hybrid analysis algorithm Parsing Dynamic analysis components Results R.W. P. D.A. Res. H.A. Hybrid Analysis of Program Binaries

34. RAID 2010 yes 34 Fully analyzed packed programs Packer Malware market share [1] 0.13%MEW 0.17%WinUPack 0.33%Yoda's Protector 0.37%Armadillo 0.43%Asprotect 1.26%FSG 1.29%Aspack 1.74%nPack 2.08%Upack 2.59%PECompact 2.95%Themida 4.06%EXECryptor 6.21%PolyEnE 9.45%UPX 0.89%Nspack Res. Self check- summing yes Self- modifying yes Exception- based ctrl yes Obfuscated yes [1] Packer (r)evolution. Panda Research, 2008. Two-month average Feb-March 2008.

35. RAID 2010 Self-checksumming techniques Hybrid Analysis of Program Binaries [1] Packer (r)evolution. Panda Research, 2008. Two-month average Feb- March 2008. Fully analyzed packed programs Packer Malware market share [1] 0.13%MEW 0.17%WinUPack 0.33%Yoda's Protector 0.37%Armadillo 0.43%Asprotect 1.26%FSG 1.29%Aspack 1.74%nPack 2.08%Upack 2.59%PECompact 2.95%Themida 4.06%EXECryptor 6.21%PolyEnE 9.45%UPX SD- Dyninst yes 0.89%Nspackyes Time to unpack 3.9 23.6 1.4 4.4 1.5 23.5 3.2 1.2 0.5 2.7 uninstrumented times are about.02 secs unoptimized overwrite detection expensive overwrite detection Res. 35

36. RAID 2010 Instrumentation costs 36 Hybrid Analysis of Program Binaries Res. Packer Pre-payload execution timeInstrumented locations SD- DyninstRenovo Saffron Intel-PIN Ether Unpack SD- DyninstRenovo Saffron Intel-PIN UPX0.552.77.662,2784,526 Aspack4.45fail18.7342,0454,141 FSG1.681.431.11418,82231,854 WinUpack23.6823.567.82318,82632,945 MEW4.06fail150.52221,18635,466

37. RAID 2010 Conclusion 37 Hybrid Analysis of Program Binaries  Analysis before execution allows for  Understanding & control of before execution  Selective monitoring  Build-your-own analysis factory  Ongoing work  Handling self-checksumming code  Releasing Dyninst w/ SD-Dyninst inside http://www.paradyn.org/