Study on “Secure In-VM Monitoring Using Hardware Virtualization” Qiang.Guan Dependable Computing System Lab New Mexico Tech

1 Contents  Background & Requirements  Secure In-VM monitoring  Implementation  Experiment evaluation  Overhead,…..

2 Background  Rootkits vs security tools  Rootkit: a software program or coordinated set of programs designed to gain control over a computer system or network of computing systems without being detected.softwareprogram  Security tools: antivirus, intrusion detection system, security reference monitoring

3 Two approaches  In-VM & Out-of-VM A: application Dp:system data Cp:system code Cm:monitor code Dm:monitor data K: event hook H; handler to event R: Response to event Dk: data about the event

4 Two monitoring modes Passive vs Active Passive: Cm analyze Cp+Dp Active: include hooks and handlers Monitoring component EventHookHandlerSys Routine Dk R

5 Out-of-VM vs In-VM  Out-of-VM  Pro: provides security ( isolation system from monitor )  Con: cannot provides performance  In-VM  Pro: provides performance (low overhead )  Con: cannot provides security

6 Performance requirements  The overhead (changing privilege between kernel level and hypervisor)  Fast invocation  Read/write in native speed.  In-VM support performance  Out-of VM cannot, why?  Hypervisor is invoked

7 Security requirements  Requirements  Isolate Cm&Dm from Cp&Dp (integrity of Cm&Dm)  Designed point for switching into Cm (switch is neat)  K H is one-to-one mapping  Monitor is not alterable (H is dependent)  Out-of-VM support performance  In-VM cannot, why?  In the same VM environment

8 Secure In-VM  A In-VM to satisfy the security requirements

9 Secure In-VM  A In-VM to satisfy the security requirements New elements

10 Features of SIM  “One-way view” design of memory mapping.  Entry and Exit gate  Transferring execution between system address space and security monitoring space.  Invocation checker Kernel-level Monitor

11 Virtual memory mapping

12 Virtual memory mapping Code and data of SIM is invisible to user address space

13 Virtual memory mapping The entry and exit gate is unchangeable for system space (1to1 policy)

14 Virtual memory mapping Kernel code will not be executed while executing in security monitoring (to make sure all the code in monitoring space is trusted)

15 Implementation  Initialization  To reserve the virtual address ranges for entry and exit gates  To create the SIM virtual space  To load security monitor application (as part of the kernel driver)  To create the link between two space (hook and handler)

16 Experimental evaluation  Test objects  SIM vs Out-of VM (why? Why not In-VM)  Test routine  Monitor Invocation Overhead  Security Application case study  Process creation monitoring  System call tracing

17 Monitor Invocation Overhead  Out-of-VM: null event handler that return immediately

18 Monitor Invocation Overhead  Out-of-VM: null event handler that return immediately  SIM: handler only calls the corresponding exit gate.

19 Result of overhead  10 times faster in avg time  More centralized from std dev

20 Summary  Contradiction  Security monitor vs untrusted guest vm  Basic mode  In-VM & out-of VM  SIM  Performance and security  Based on In-VM and appending security issues.  Result (overhead)  SIM is 10 times better than out-of-VM