EUROSEC 2011 Gábor Pék, Boldizsár Bencsáth and Levente Buttyán Laboratory of Cryptography and Systems Security Budapest University of Technology and Economics nEther: IN-GUEST DETECTION OF OUT-OF-THE-GUEST MALWARE ANALYSERS

Gábor Pék, CrySyS Lab. Short Summary We successfully achieved  In-guest detection of an out-of-the-guest malware analysis framework (Ether)  In-guest timing attack  Detection based on CPUID information  Detecting hardware assisted virtualization (can be a bit of information for analysis )  Detection based on errata in Intel CPUs 9/16/2014 2

Gábor Pék, CrySyS Lab. Goals in Malware Analysis  Analyser: dissecting and figuring out the operations of the analysed program  Author of the malware: thwarting the analysis of the code and hiding its real intents, operations, execution 9/16/2014 3

Gábor Pék, CrySyS Lab. What is Malware Analysis?  Analysing malware  Static (entire program, thwarting disassemblers)  Dynamic (one control path)  we focus on this  Two types of dynamic analysis: Native and Virtualization based  Main tricks of detecting dynamic analyzers  Timing information  Special data structures, e.g., PEB  Single-step debugging (trap flag)  Exception handling 9/16/2014 4

Gábor Pék, CrySyS Lab. HW Assisted Virtualization  New and higher CPU privilege level (Ring -1)  Native instruction execution  Intel VT  VMX root mode for VMM/Hypervisor  VMX non-root mode for guest OS  VMX transitions: VM Exit / VM Entry  Rich feature set and control of operation  Xen, KVM 9/16/2014 5

Gábor Pék, CrySyS Lab. Ether – Malware analysis via HW Virtualization Extensions  Transparent, out-of-the-guest malware analysis platform based on Xen and Intel VT  Transparency of Ether: the malware cannot detect Ether  Transparency requirements as of the Ether paper:  Higher privilege of analyser environment  No non-privileged side effects  Same instruction execution semantics X  Identical exception handling  Identical notion of time X 9/16/2014 6

Gábor Pék, CrySyS Lab. Advantages of Ether and challenges to detect it  No in-guest memory presence  Hide of changes made on CPU registers  Memory protection: modifies only shadow page tables  Privileged instruction handling  No instruction emulation  Controlling timing (e.g., RDTSC instruction) 9/16/2014 7

Gábor Pék, CrySyS Lab. Contributions 9/16/  Design and implementation of an application framework to detect Ether based on multiple feature tests  Feature tests for Ether and Intel VT  A practical in-guest timing attack against Ether  Detecting Ether via CPUID information  Detection of HW assisted virtualization utilizing CPU errata

Gábor Pék, CrySyS Lab. System Overview 9/16/2014 9

Gábor Pék, CrySyS Lab. Various uses of RDTSC 9/16/  Different behaviour of sensitive instructions (e.g., RDTSC) in VMX non-root mode Guest OS rdtsc Normal operation TSC rdtsc faketime (FT) Operation of Ether rdtsc FT+Y VM Entry VM Exit … … CPU Virtual CPU Virtual CPU Guest OS

Gábor Pék, CrySyS Lab. Facts about Ether  Alleged operation: FT = TSC, Y = TSC_OFFSET  but TSC_OFFSET is disabled  Real operation: Monotonic increase of FT for every RDTSC call (Y =1)  There can be external RDTSC calls during an analysis  The TSC difference between two RDTSCs of the analysed program = # of RDTSCs of the Guest during analysis (~9-171) 9/16/

Gábor Pék, CrySyS Lab. Practical implementation of in-guest timing attack  Call an RDTSC and store it  Create a loop of non-sensitive instructions (e.g., nop )  Call an RDTSC and compare it with the stored value ( diff ) 9/16/ if (diff < length of the loop) Ether is present else Ether is not present

Gábor Pék, CrySyS Lab. CPUID information  CPUID instruction: processor identifcation and feature information  Allegedly: Ether has no in-memory presence  Reality: The TSC bit returned by CPUID is unset under Ether  Other bits of information  PAE and PSE are disabled 9/16/

Gábor Pék, CrySyS Lab. CPU Errata  Design deficiencies of CPUs  Some of them are unpredictable  Cause unexpected system behaviour  Several have ”No Fix ” status  Xen creates virtualized CPUs for privileged instructions  We have an erratum using MSRs (AH4)  The access of MSRs are privileged  VM exit  Errata are not emulated by virtual CPUs  Bingo, we have a new feature test 9/16/

Gábor Pék, CrySyS Lab. Detecting Intel VT 9/16/ Erratum AH4Number of updates # of testsNativeXenXen + Ether

Gábor Pék, CrySyS Lab. Future Work  Fundamentality of these problems  Updating the theoretical model and practical implementation of Ether  Finding more feature tests against other out- of-the-guest approaches (e.g., Azure)  Proving that perfect transparency has practical limitations 9/16/

Gábor Pék, CrySyS Lab. Thanks for Your Attention! Questions ? CrySyS Lab. Budapest University of Technology and Economics 9/16/

Gábor Pék, CrySyS Lab. Dynamic Analysis  Using Virtualization  Pure software virtualiztaion (QEMU, BOCHS)  Sandboxing environments (Anubis, CWSandbox)  Myriads of detection vectors  Timing information (e.g., Racing)  In-guest memory scans  Invalid instructions  Novel approaches were required 9/16/