1. Private & Confidential Property of COSEINC

2. Who am I? Senior Security Researcher at COSEINC One of the developers of BluePill, a hardware- based virtualization rootkit. Creator of one of the most effective methods to detect virtualization rootkits. Experience with systems programming (kernel, device drivers) and reverse engineering for x86/x64 architectures. Private & Confidential Property of COSEINC

3. 1.Review of the implementation methods for virtualization of the x86 architecture. 2.Show the complexity of using hardware supported virtualization instructions to implement virtual machines. 3.Present a framework that makes easy the task of creation of hypervisors. 4.Applications of the framework 5.Security aspects Private & Confidential Property of COSEINC

4. X86 VIRTUALIZATION The COSEINC Hypervisor Framework Private & Confidential Property of COSEINC

5. Private & Confidential Property of COSEINC

6. IA-32 processor VMM Windows guest Linux guest Linux guest 1.Type I (native) The VMM runs directly on the host’s hardware. Hardware resources controlled by the VMM. Examples: VMware ESX, Microsoft Hyper-V Private & Confidential Property of COSEINC

7. IA-32 processor VMM Windows guest Linux guest Linux guest Operating System Type II – Hosted The VMM runs as an application. Hardware resources controlled by the host OS. The COSEINC hypervisor framework creates a Type-II VMM. Examples: VMware Workstation. Private & Confidential Property of COSEINC

8. When the guest VM uses the same Instruction Set Architecture (ISA) of the host machine, the guest instructions can be executed in 2 ways: – Emulation – Direct native execution Private & Confidential Property of COSEINC

9. The VMM must read and interpret each guest instruction Can be implemented using code interpretation or binary translation Performance penalty Private & Confidential Property of COSEINC

10. The guest instructions are executed directly on the CPU. Great performance. Some instructions still need to be emulated. How to decide which instructions can be used for direct native execution? Private & Confidential Property of COSEINC

11. Private & Confidential Property of COSEINC

12. Private & Confidential Property of COSEINC

13. Private & Confidential Property of COSEINC

14. Private & Confidential Property of COSEINC

15. Private & Confidential Property of COSEINC

16. Innocuous instructions are instructions which doesn’t change or affect system configuration or resources. A efficient VMM allows the direct execution of innocuous instructions. Examples: – mov eax, 00204012h – shr ebx, 03 – xor eax, eax – cmp ebx, ecx Private & Confidential Property of COSEINC

17. Private & Confidential Property of COSEINC

18. Private & Confidential Property of COSEINC

19. INNOCUOUS INSTRUCTIONS KERNEL code cmp eax, ebx jnz 8c0dab00 xor edx, edx mov eax, 030h cmp eax, ebx jnz 8c0dab00 xor edx, edx mov eax, 030h wrmsr cmp eax, 020Fh jnz 08000bc00 shr eax, 8 cmp eax, 020Fh jnz 08000bc00 shr eax, 8 SENSITIVE INSTRUCTION Private & Confidential Property of COSEINC

20. Set CPL to RING 3 and execute the code directly on the cpu KERNEL code cmp eax, ebx jnz 8c0dab00 xor edx, edx mov eax, 030h cmp eax, ebx jnz 8c0dab00 xor edx, edx mov eax, 030h wrmsr cmp eax, 020Fh jnz 08000bc00 shr eax, 8 cmp eax, 020Fh jnz 08000bc00 shr eax, 8 #GENERAL PROTECTION FAULT VMM trap handler routine (emulation) VMM trap handler routine (emulation) Private & Confidential Property of COSEINC

21. Virtualization of guest instruction would be very easy if all sensitive instructions generates a fault in ring 3. There are sensitive but non-privileged instructions in the x86 architecture! A sensitive non-privileged instruction will not generate an exception in ring 3! Private & Confidential Property of COSEINC

22. POPFD instruction writes a DWORD value in the EFLAGS register. It’s a sensitive instruction because it can be used to set the IF flag. The IF (Interrupt Flag) controls the hardware external interrupt mechanism. Private & Confidential Property of COSEINC

23. Problem: Executing POPFD in ring3 will not generate a fault! The CPU just ignores the IF flag modification attempt. How to virtualize sensitive non-privileged instructions? Private & Confidential Property of COSEINC

24. How VMware Player VMM is able to prevent direct execution of non-privileged instructions? VMware Player is a Type II VMM The hypervisor is stored as a PE resource inside the vmware-vmx.exe executable. ELF executable loaded directly inside the Windows kernel memory by the vmx86.sys device driver Private & Confidential Property of COSEINC

25. vmplayer.exe vmware-vmx.exe USER MODE KERNEL MODE vmx86.sys ntoskrnl.exe Private & Confidential Property of COSEINC Vmware Hypervisor ELF executable stored as a PE resource

26. Solution: Scan all the guest code instructions and search for non-privileged instructions. Replace the non-privileged instructions by a privileged instruction. VMM handles the faults and emulates the execution of the non-privileded instruction. Private & Confidential Property of COSEINC

27. 1.Review x86 virtualization implementation methods. 2.Show how to use the Intel VT® to implement virtual machines. 3.Present a framework to make easy the task of creation of hypervisors. 4.Applications of the framework 5.Security and detection discussion Private & Confidential Property of COSEINC

28. The COSEINC Hypervisor Framework Private & Confidential Property of COSEINC

29. Virtualizable ISA – If all sensitive instructions of some ISA are privileged, the processor is considered to be ‘virtualizable’ [3] IA-32 is obviously not-virtualizable. New instruction sets created by Intel and AMD – Intel Virtual Machine eXtensions (VMX) – AMD Secure Virtual Machine (SVM) Private & Confidential Property of COSEINC

30. Presentation focus on Intel VMX. AMD SVM concepts are very similar. New form of processor operation: the ‘VMX operation mode’ VMX mode – activated by the VMXON instruction. Private & Confidential Property of COSEINC

31. VMXON fails if virtualization is locked. Locked by default in the BIOS for security reasons Ring -1. There’s no more need to move kernel guest code from ring 0 to ring 3. Guest kernel code can run directly in ring 0. Private & Confidential Property of COSEINC

32. 2 types of VMX operation: – VMX root operation – VMX non-root operation VMX root operation – New instructions available (VMX instructions) – Used by the VMM (hypervisor) VMX non-root operation – Restricted mode of operation – Certain instructions and events are intercepted to facilitate virtualization. Private & Confidential Property of COSEINC

33. Transitions between VMX root operation and VMX non-root operation are called ‘VMX transitions’ Transition from the VMM to the guest: VM- ENTRY. Transition from the Guest VM to the VMM: VMEXIT Private & Confidential Property of COSEINC

34. Hypervisor (vmx root operation) Hypervisor (vmx root operation) VIRTUAL MACHINE (vmx non-root operation) VIRTUAL MACHINE (vmx non-root operation) VM-ENTRY – vmresume/vmlaunch VM-EXIT event interception Private & Confidential Property of COSEINC

35. Creating a VMM with Intel VT® - first steps Detection of Intel VMX instruction support. – CPUID Enable VMX (CR4) – VMXE bit Check status of the Lock bit (rdmsr) – More about in the security section Setup of the VMXON region Private & Confidential Property of COSEINC

36. Creating a VMM with Intel VT® - first steps Enable VMX instructions (VMXON) Create and configure the VMCS region of each guest VM. Launch the guest VM with VMLAUNCH instruction Wait for VM-exit events Private & Confidential Property of COSEINC

37. VMCS Virtual Machine Control Structure Most important vmx data structure One VMCS for each Virtual Machine and for each CPU core. It controls the behavior of VMX transitions Private & Confidential Property of COSEINC

38. VMCS VMM must not access the VMCS directly. Read and write access to the VMCS via VMREAD and VMWRITE instructions. Internal structure undocumented but reverse engineering it is easy. Private & Confidential Property of COSEINC

39. VMXON and VMCS areas VMXON region CPU A VMXON region CPU A VM Linux VM Linux VM Windows VM Windows VMXON region CPU B VMXON region CPU B VMCS #1A VMCS #2A VMCS #1B VMCS #2B CPU ACPU B Private & Confidential Property of COSEINC

40. VMCS logical groups Guest-state area Host-state area VM-execution control fields VM-exit control fields VM-entry control fields VM-exit information fields 4K-aligned physical address 6 logical areas Private & Confidential Property of COSEINC

41. Guest-state area Area of the VMCS where guest context information is stored. On #VMEXIT, guest processor state is saved in this area. On VMENTRY this information is loaded. Register state: – Control Registers – Debug Registers – RSP, RIP, RFLAGS – LDTR, GDTR, IDTR – Segment selectors – Model Specific Registers Private & Confidential Property of COSEINC

42. Guest-state area Non-register state – Activity State – Interruptibility state – VMCS link pointer For future expansions Private & Confidential Property of COSEINC

43. Host-state area Contains information about the host (VMM) Processor stated is loaded from this area after each #VMEXIT Registers: – RIP (Entry-point address of the hypervisor routine responsible for handling #VMEXIT events) – RSP, RFLAGS – MSR Private & Confidential Property of COSEINC

44. VM-execution control fields Controls how the VM will be executed. The instructions that the hypervisor wants to intercept are specified in these control fields. – Example: HLT, INVLPG, MWAIT, RDPMC, RDTSC, MOV-DR Exception bitmap – Bitmap which controls interception of CPU interrupts like page faults, debug exceptions, #GP,... I/O bitmap – Can be used to control interception of I/O ports MSR bitmap – Interception of Model Specific Registers Some instructions wil unconditionally result in VMEXIT Private & Confidential Property of COSEINC

45. VM-entry control fields Controls the behavior of VM entries. Includes information about SMM, debug registers and some MSRs. Guest Event Injection: – It’s possible to inject virtual interrupt or exception in the guest – Types of interrupts allowed: External, NMI, Hardware exceptions, software interrupt. Private & Confidential Property of COSEINC

46. VM-exit fields `VM-exit control fields` which controls the behavior of VM exits. VM-exit information fields: – Read-only fields with information about the most recent VM exit – Exit reason – Exit qualification Private & Confidential Property of COSEINC

47. Interception After configuring the VMCS, the hypervisor can launch the virtual machine and wait for a VMEXIT event. When a instruction is intercepted in the guest, the processor will: – Save the VM-exit reason information in the VMCS – Save guest context information – Load the host-state area – Transfer control to the hypervisor Private & Confidential Property of COSEINC

48. VMX ROOT-MODE RING 0 VMX ROOT-MODE RING 0 VMM #VMEXIT event handler VMX NON-ROOT RING 0 VMX NON-ROOT RING 0 mov eax, 23 inc edx xor ebx, edx sub ecx cmp eax, 1 jnz c080df00 retn mov eax, 23 inc edx xor ebx, edx sub ecx cmp eax, 1 jnz c080df00 retn mov cr3, ebx #VMEXIT VMLAUNCH #VMRESUME Private & Confidential Property of COSEINC

49. 1.Review x86 virtualization implementation methods. 2.Show how to use the Intel VT® to implement virtual machines. 3.Present a framework to make easy the task of creation of hypervisors. 4.Applications of the framework 5.Security and detection discussion Private & Confidential Property of COSEINC

50. Creating a VMM using these new hardware virtualization ISA is complex – More complex features always comming: EPT for nested paging Very hard to find and to fix bugs No debugger Intel VT error codes not very useful – Code 33 = “VM-entry failure due to invalid guest state” – What’s exactly invalid in the guest state? – More than 40 suspects! Private & Confidential Property of COSEINC

51. The COSEINC Hypervisor Framework, referred from now as just the ‘framework’, enables you to easily create a Hosted Virtual Machine Monitor (Type II VMM) using the Windows Operating System. Simple and easy-to-use API exported Abstraction over the different hardware virtualization instruction sets (VMX-SVM) Private & Confidential Property of COSEINC

52. 2 versions: – 32-bits Windows device driver – 64-bits Windows device driver API exported methods: – Export table – IOCTL codes for user-mode communication Initial version only for Windows, but porting to Mac/Linux should not be difficult. Release date: very soon! Private & Confidential Property of COSEINC

53. Features Automatic detection of the virtualization instruction sets. SMP support Evaluation of the lock bit Detailed error-status codes Plugin-like architecture Private & Confidential Property of COSEINC

54. Architecture Framework Ring 0 Operating System Kernel Operating System Kernel Ring -1 User applications Ring 3 Framework Client Private & Confidential Property of COSEINC

55. API The full documentation of the API will be released with the framework. Preliminary documentation. Subject to change. Function categories: – Virtual Machine management functions Creation and deletion of Virtual Machines. Executing and resuming a virtual machine. – Interception Events functions The framework call the registered client function callbacks. – Root guest VM. Private & Confidential Property of COSEINC

56. Virtual Machine management VMSTATUS CreateVirtualMachine ( IN VMINFO *vminfo ); This function creates a new virtual machine in the system. Fails if virtualization MSR is locked by the BIOS. Private & Confidential Property of COSEINC

57. VMINFO data structure Most important framework data structure Contains all the information needed to create and control a VM: – all the GUEST context information – GDT, LDT, Page Tables, Control Registers,... – Interception handler function callback address. – Contains Event Injection information – VMEXIT information Private & Confidential Property of COSEINC

58. GUEST_INFO Registers Segments Descriptor Tables Control Debug Model Specific CONTROL_INFO Interception Event Injection I/O Interrupts MSR Virtual Machine VMINFO data structure VMEXIT info Extra info Private & Confidential Property of COSEINC

59. Interception Event management VMSTATUS VirtualMachineExec ( IN VMINFO *vminfo ); This function controls the execution of the virtual machine. It can be called after the creation of the VM and to resume the execution of the VM after an intercept event. If the VMM must inject some event in the guest VM, the information is provided in the VMINFO data structure. Private & Confidential Property of COSEINC

60. x x VM creation and execution Framework Client (VMM plugin) Framework Client (VMM plugin) Framework CreateVirtualMachine( ) Intercept event handler Intercept Event Message VirtualMachineExec( ) Private & Confidential Property of COSEINC VM

61. VM Scheduler VM message handler Timer interrupt VM Event Router Virtual Machine VM Event Manager Hypervisor Framework – Client communication

62. Root guest VM One of the best features of the framework: – Automatic conversion of the host operating system into a virtual machine in runtime! This guest VM is called ‘root VM’ The creation of the root VM is optional and controlled by the api. Root VM is shared between all loaded plugins. Private & Confidential Property of COSEINC

63. 1.Review x86 virtualization implementation methods. 2.Show how to use the Intel VT® to implement virtual machines. 3.Present a framework to make easy the task of creation of hypervisors. 4.Applications of the framework 5.Security and detection discussion Private & Confidential Property of COSEINC

64. The COSEINC Hypervisor Framework Private & Confidential Property of COSEINC

65. Applications of the framework Specially useful for education and research purposes Can abe used to create any type of small and fast VM. Not only system VMs. The best features are available when using the root guest VM. Private & Confidential Property of COSEINC

66. Process VM Whole virtualization of a process or a thread is possible with the framework. Normally achieved by interception of system calls. Additional functions will be added to the API for better memory virtualization. No support for EPT in the first version. Private & Confidential Property of COSEINC

67. Syscall hooking A great number of system monitoring and security tools are implemented using system call hooking methods. Old Windows OS uses INT 2eh Linux and newer Windows OS uses SYSENTER instructions Private & Confidential Property of COSEINC

68. Syscall mechanism - illustration SYSENTER_EIP MSR SYSENTER_CS MSR mov edx, esp sysenter mov edx, esp sysenter mov ecx, 23h push 30h pop fs... mov ecx, 23h push 30h pop fs... nt!KiFastCallEntry Ntdll.dll Private & Confidential Property of COSEINC Windows OS syscall mechanism

69. Syscall hooking Syscall hooking methods includes: – Patching syscall handler – Patching of IDT table – Patching the SYSENTER Model Specific registers Private & Confidential Property of COSEINC

70. Syscall interception Syscall interception using the root guest VM No need to hook SSDT No need to patch/modify guest kernel code Virtualization of the SYSENTER MSR Plugin (framework) – VMINFO->ControlInfo->Interception->MSR Can also be applied to Linux guests Virtualized IDTR for old guest operating systems using INT xx instructions for syscall implementation. Private & Confidential Property of COSEINC

71. Instrumentation Instrumentation is also easy to implement using the Interruptibility controls in the VMCS. Performance registers are also virtualizable Tools: – Optimization tools – System statistics Private & Confidential Property of COSEINC

72. Nested virtualization The framework doesn’t provide support for nested virtualization But it is possible to add this feature via a VMM plugin. Also, a virtualization debugger could be implemented! Private & Confidential Property of COSEINC

73. 1.Review x86 virtualization implementation methods. 2.Show how to use the Intel VT® to implement virtual machines. 3.Present a framework to make easy the task of creation of hypervisors. 4.Applications of the framework 5.Security and detection discussion Private & Confidential Property of COSEINC

74. 64-bits The framework and the plugins must be digitally signed to run in 64-bit versions of Windows. Private & Confidential Property of COSEINC

75. MSR IA32_FEATURE_CONTROL (Index 3Ah) Controls: – SMX – Safer Mode eXtensions Disabled by default in the BIOS Private & Confidential Property of COSEINC

76. “There is no software-visible bit whose setting indicates whether a logical processor is in VMX non-root operation. This fact may allow a VMM to prevent guest software from determining that it is running in a virtual machine.” – Intel manual 3 – 19.3 VMX transitions are cpu-expensive operations. Thousand of cycles just for a simple VMEXIT. SyScan 2007 – Detecting BluePill Private & Confidential Property of COSEINC

77. QUESTIONS?

78. THANK YOU FOR YOUR TIME!

79. 1.John Scott Robin and Cynthia E. Irvine (2000). "Analysis of the Intel Pentium's Ability to Support a Secure Virtual Machine Monitor". Proc. 9th USENIX Security Symposium.Analysis of the Intel Pentium's Ability to Support a Secure Virtual Machine Monitor 2.Virtual Machines: Versatile Platforms for System and Processes – Jim Smith, Ravi Nair – Morgan Kaufmann - 2005 3.Intel manuals (www.intel.com)www.intel.com