1. Stealthy Malware Detection Through VMM-based “Out-of-the-Box” Semantic View Reconstruction CCS’07, Alexandria, VA, Oct 29 – Nov 2, 2007 Xuxian Jiang, Xinyuan Wang, Dongyan Xu George Mason University Purdue University

2.  Internet malware remains a top threat  Malware: viruses, worms, rootkits, spyware, bots… Motivation

3.  Recent Trend on Rootkits Source: McAfee Avert Lab Report (April 2006) 400% growth Q1 of 2005 700% growth Viruses/worms/bots, PUPs, …

4. Existing Defenses ( e.g., Anti-Virus Software )  Running inside the monitored system  Advantages  They can see everything (e.g., files, processes,…)  Disadvantages  Once compromised by advanced stealthy malware, they may not see anything! VirusScanFirefox IE OS Kernel …

5. Existing Defenses  Key observation  Both anti-virus software and vulnerable software are running inside the same system  Hard to guarantee tamper-resistance  Solution: “Out-of-the-box” defense Firefox IE OS Kernel … VirusScan Virtual Machine Monitor (VMM)

6. The “Semantic-Gap” Challenge  What we can observe?  Low-level states  Memory pages, disk blocks,…  Low-level events  Privileged instructions,  Interrupts, I/O access, …  What we want to observe?  High-level states w/ semantic info.  Files, processes,…  high-level events w/ semantic info.  System calls, context switches, … Virtual Machine Monitor (e.g., VMware, Xen, QEMU) Guest OS Semantic Gap VirusScan

7. Main Contribution  VMwatcher: A systematic approach to bridge the semantic gap  Reconstructing semantic objects and events from low-level VMM observations Firefox IE OS Kernel … Virtual Machine Monitor (VMM) VMwatcher Capability I: “Out-of-the-box” execution of commodity anti-malware software Capability I: “Out-of-the-box” execution of commodity anti-malware software Capability II: View comparison-based stealthy malware detection Capability II: View comparison-based stealthy malware detection

8. VMwatcher: Bridging the Semantic Gap  Step 1: Procuring low-level VM states and events  Disk blocks, memory pages, registers, …  Traps, interrupts, …  Step 2: Reconstructing high-level semantic view  Files, directories, processes, and kernel modules,…  System calls, context switches, … VM Introspection Guest View Casting

9. Step 1: VM Introspection Raw VMM Observations Virtual Machines (VMs) VMware Academic Program VM Disk Image VM Hardware State (e.g., registers) VM Physical Memory VM-related low-level events (e.g., interrupts)

10. Step 2: Guest View Casting Virtual Machine Monitor (VMM) Guest OS Disk Key observation: The guest OS already contains all necessary semantic definitions of data structures as well as functionalities to construct the semantic view VMwatcher Semantic Gap VirusScan Cross-view

11. Guest View Casting Raw VMM Observations Casted Guest Functions & Data Structures Reconstructed Semantic View Device drivers, file system drivers Memory translation, task_struct, mm_struct CR3, MSR_SYSENTER_CS, MSR_SYSENTER_EIP/ESP Event semantics Syscalls, Context switches,.... Event-specific arguments… VM Disk Image VM Hardware State (e.g., registers) VM Physical Memory VM-related low-level events (e.g., interrupts) Demo clip (3.5mins): http://www.ise.gmu.edu/~xjiang/http://www.ise.gmu.edu/~xjiang/

12. Guest View Casting on Memory State (Linux) Process List Process Memory Layout

13. Guest Memory Addressing  Traditional memory addressing  Given a VA, MMU translates VA to PA  OSes used to map with known PA  Linux: VA 0xc0000000 == PA 0x0  Windows: VA 0x80000000 == PA 0x0  VM complicates the translation  Guest virtual -> guest physical  Guest physical -> host physical VM Introspection Reverse Address Translation Emulated Address Translation

14. Evaluation  Effectiveness  Cross-view malware detection  Exp. I: Cross-view detection on volatile state  Exp. II: Cross-view detection on persistent state  Exp. III: Cross-view detection on both volatile and persistent state  Out-of-the-box execution of commodity anti- malware software  Exp. IV: Symantec AntiVirus  Exp. V: Windows Defender  Performance  Difference between internal scanning & external scanning

15.  Experiment Setup  Guest VM: Windows XP (SP2)  Windows Fu Rootkit  Host OS: Scientific Linux 4.4  VMM: VMware Server 1.0.1 Exp. I: Cross-view detection on volatile memory state “Inside-the-box” view VMwatcher view Diff

16.  Experiment Setup  Guest VM: A Redhat 7.2-based honeypot  Linux SHv4 rootkit  Host OS: Windows XP (SP2)  VMM: VMware Server 1.0.1 Exp. II: Cross-view detection on persistent disk state “Inside-the-box” view VMwatcher view Diff

18. Experiment (IV)  Experiment Setup  Both guest OS and host OS run Windows XP (SP2)  VMM: VMware Server 1.0.1  Running Symantec AntiVirus Twice  Outside  Inside Hacker Defender NTRootkit

19. External Scanning Result Internal Scanning Result Diff

20. Performance  Internal scanning time vs. external scanning time Internal scanning takes longer to complete !

21. Related Work  Enhancing security with virtualization ( Livewire[Garfinkel03], IntroVirt[Joshi05], HyperSpector[Kourai05] )  Focusing on targeted attacks with specialized IDSes  Cross-view detection ( Strider GhostBuster[Wang05], RootkitRevealer/ Blacklight/IceSword/… )  Either destroying the volatile state or obtaining two internal views  Secure monitors  CoPilot [Petroni04], Terra [Garfinkel03], sHype [Sailer05], SecVisor [Perrig07],TRANGO,…

22. Conclusions  VMwatcher – A systematic approach that bridges the semantic gap and enables two unique malware detection capabilities:  Cross-view malware detection  “Out-of-the-box” execution of commodity anti- malware software

23. Thank you! For more information: Email: xjiang@ise.gmu.edu@ise.gmu.edu URL: http://www.ise.gmu.edu/~xjianghttp://www.ise.gmu.edu/~xj