A Secure System-wide Process Scheduling across Virtual Machines Hidekazu Tadokoro (Tokyo Institute of Technology) Kenichi Kourai (Kyushu Institute of Technology) Shigeru Chiba (Tokyo Institute of Technology) 1

Scheduling Problem across VMs  Server consolidation using virtual machines(VMs)  To improve the resource utilization  VMs make it difficult to execute processes as administrators intend  Guest OSes schedule only their processes  A low-priority process in a VM may interfere with a high-priority in other VMs 2 Hardware VMM VM OS Indexing WEB OS

System-wide Process Scheduler  Necessary for scheduling processes across VMs  It can suppress the execution of less important process  Because it knows important processes among all VMs  E.g. it can run the file indexing process only when the whole system is idle 3 Indexing VMM system-wide scheduler check VMs are idlerun indexing VM

Issue: Difficult to Implement  Implementing a system-wide process scheduler in the VMM is unsuitable  VMM cannot recognize the process  Processes are abstraction of OSes  Passing information of processes to VMM requires modification of guest Oses  Modification of guest OSes is often unacceptable 4 ???? VMM ???? semantics gap what process is running? VM 1) Guest-aware VM scheduling [Euro-Par’08 Kim et al.] 2) ask grain scheduling [HPCC’08 Kinebuchi et al.] 1), 2)

Issue: Vulnerable to a DoS Attack  A process in a compromised VM can prevent processes in other VMs through the scheduler  E.g. a busy loop process can easily stop the file indexing process in other VMs  The indexing is configured to run at idle time 5 Indexing VMM VM malicious loop system-wide scheduler never run VMs are NOT idle

Monarch Scheduler  A system-wide process scheduler in the VMM  manipulate internal data in guest OSes for process scheduling  recognize the process  Hybrid scheduling to mitigate a DoS attack  Periodically switches between system-wide process scheduling and original scheduling 6 Indexing VMM VM WEB Monarch Scheduler change scheduling

Process Scheduling by the VMM  VMM monitors and manipulates the run queue and the process structure in guest OSes  Suspending a process  Remove from the run queue  Rewrite its state to stop spontaneously  Resuming a process  Insert it into a run queue 7 Monarch Scheduler process modify memory run queue VM

Hybrid Scheduling  To guarantee some CPU time to every process  Periodically switches two modes  Controlled mode: performs system-wide scheduling  Autonomous mode: stops system-wide scheduling  VMM and guest OSes are perform their own original scheduling 8 switch Monarch Scheduler malicious loop indexing VM controlled VM stop Monarch Scheduler malicious loop indexing VM autonomous run freely

Implementation  We implemented in Xen  Supported guest OS is Linux 2.6 (x86_64)  Scheduler is invoked by timer interrupts in VMM  Pause a DomainU  To prevent conflict between the Monarch scheduler and the guest OS  Get the CPU time of each process  Schedule when the controlled mode 9 Xen Monarch Scheduler process run queue DomainU interruptschedule

Accessing Kernel Data  The Monarch scheduler accesses the internal data of guest OSes based on their information  Obtain debug information from kernel image in advance  Translate virtual addresses of domainU into machine addresses of the VMM at run time  Page tables of guest OSes  P2M tables 10 virtual address Xen VMM DomU P2M table machine memory page table kernel image

Finding process structures  The Monarch scheduler traverses a process list  Every process structure is linked to the list  The starting point is init_task  The address of init_task is invariant in each kernel image 11 init_task Linux kernel

Finding Run Queues  The Monarch scheduler finds a run queue for each v-CPU  The address is unknown until boot of the guest OS  The number of v-CPUs is not determined until boot  The starting point is GS register of each v-CPU  The GS points x8664_pda, which contains a pointer to a run queue 12 struct x8664_pda { task_t* current; ulong data_offset; …}; x8664_pda run queue Linux memory data_offset + PER_CPU_RUNQUEUES GS register

Guaranteeing Consistency  The Monarch scheduler checks a lock of the data structure  To guarantee that the guest is not accessing the data whenever the Monarch scheduler accesses it  Acquiring the lock is not needed  The domain is paused 13 schedule() { spin_lock(runqueue); RUN QUEUE OPERATION spin_unlock(runqueue); } scheduler of Linux OS Monarch Scheduler runqueue spinlock unlock checklock

Monitoring Process Time  The Monarch scheduler records the execution time of each process  It tracks the switches of virtual address spaces  By trapping modification of the CR3 register  It binds virtual address spaces to processes  By using process information in guest Oses  Time recorded by guest OSes is inaccurate 14 Monarch Scheduler CR3 process track change of CR3 bind CR3 to process

Experiments  Examining overheads  Scheduling overheads  Monitoring overheads  Performance degradation  Examining the scheduling behavior  System-wide idle-time scheduling  Hybrid scheduling with the idle-time scheduling  Examining the impact of update the guest OS 15 Core 2 Duo 2.4 GHz Memory 6GB Xen Dom0: Linux DomU: Linux (1GB)

Scheduling Overheads  Time for traversing the process list  Change the number of processes in one VM  Change the number of VMs with fixed number of processes  Traversing time is negligible in the schedule  36ns/proc  880ns/VM 16

Monitoring Overheads 17  Time for recording the execution time of processes with CR3  The total number of context switches per second  Overhead is negligible Time to record (us/context switch) Number of context switches (/sec) Overhead(%) Boot time Steady state

Performance Degradation  Throughput and response time of lighttpd  Changing scheduling interval  Only traversing the process list  Changing the number of processes  Slightly degraded when the interval is 10ms 18 Throughput Response time

System-wide Idle-time Scheduling  Examining that the Monarch scheduler correctly archives the idle-time scheduling  Stop HyperEstraier whenever lighttpd runs  The Monarch scheduler archived the policy  HyperEstraier degrades lighttpd without scheduling 19 Xen VMM lighttpd Hyper Estraier VM2VM1 run only at idle time without scheduler with scheduler

Hybrid Scheduling  Examining the effectiveness of hybrid scheduling  Changing the ratio of the autonomous mode  The indexing process was executed according to the ratio of autonomous mode  A steep rise of CPU utilization when more than 80% 20

Impact of Updating the Guest OS  How much the Monarch scheduler has to be modified when the Linux kernel is updated  Inspected 33 versions of the Linux kernel VersionChangeDifficulty Internal structure of spinlock_tEasy runqueue is renamed to rqEasy Process scheduler changed from O(1) to CFS Hard but possible The way to calculate the address of a run queue Easy

Related Work  Guest-aware VM scheduling [Euro-Par’08 Kim et al.]  Guest OSes notify the VMM of their highest priority  Modification of guest OSes is required  Task grain scheduling [HPCC’08 Kinebuchi et al.]  Guest OSes notify L4 of priorities of all processes  Not suitable for Xen due to frequent VM switches  Task-aware VM scheduling [VEE’09 Kim et al.]  Using gray-box knowledge  Not for process scheduling 22

Conclusion  Monarch scheduler  A secure system-wide process scheduler running in the VMM  monitor the execution of processes  change the scheduling behavior of each guest OS  provide hybrid scheduling to mitigate a DoS attack  Future work  Completion of the support for Windows guest OS 23