1. Process Migration Checkpoint/Restart
ECI, July 2005

2. Process Migration Process migration benefits: Tool for load balancing
Data access locality
Improved system administration
Mobile computing

3. Process Migration Issues
Execution model: home, remote
Migrating virtual memory
Minimizing downtime
Cost of migration
Run time cost (home, remote)
Migration operation
Limitations of migration

4. Checkpoint / Restart Checkpoint/restart benefits:
Like migration plus …
Fault resilience
Fault recovery
High availability
Gang scheduling
Debugging, testing, developing
Security (honey-pot)

5. Checkpoint/restart goals
Transparency
Support parallel programs
Multi-process
Multi-node
Security
Minimize required state
Minimize required storage

6. CKPT: Application Level
Efficient
Non-preemptive
Lack of common API
Source code changes
Possible compiler support
Examples ?

7. CKPT: Library Level Library level Examples…
Typically use a signal handler (callback)
Common API
Restricts functionality (e.g., no IPC)
Relatively portable
Examples…

8. CKPT: Library (contd) Libckpt Condor Score, co-check
Memory exclusion, incremental, forked
Modify source code, link statically
Condor
Support memory mapping, shared libraries
Relink to special library (needs object file)
Score, co-check
Parallel applications
Modify communication layer

9. Implementation (contd)
Kernel level
Loadable kernel module vs. change kernel
Preemptive / cooperative
Access to entire process state
Complex, less portable
Examples: Sprite, Zap
Virtual machines
(soon)

10. Multi-process Checkpoint
Global state
A set of states from all processes
Consistent global state
If the state of A reflects a message received from B, then the state of B reflects sending
If the state of A reflect a message sent to B but not yet received, it must be part of the channel state

11. Consistent Global State

12. Multi-process Checkpoint
Uncoordinated checkpoint
Inspect data to find recovery line
Processes are independent, efficient
Domino effect, much storage

13. Multi-process Checkpoint
Coordinated checkpoint
Centrally managed
Blocking
All processes suspended
Flush communication channels
Non blocking
Delay in triggers may yield inconsistency

14. Multi-process Checkpoint
Communication-induced
Piggyback process checkpoint status and requests on messages
May require enforcing global checkpoint
Unpredictable checkpoint times

15. Multi-process Checkpoint
Summary:
Uncoordinated
Coordinated
Communication induced
Domino effect
Possible No
Management overhead
None
More
Less
Decision making
Local
Central
Local/central
Checkpoint data stored
All
Latest only
Several

16. Virtual Machines
“Any problem in computer science can be solved by another layer of indirection”
ECI, July 2005

17. What is a Virtual Machine ?
An indirection layer below the execution environment seen by applications and OS
Decouple architecture and user perceived behavior of SW and HW resources from their physical implementation
Provide a uniform view of the underlying resources
Multiplex multiple virtual systems on a single (physical) resource

18. VM History 1960’s – Hypervisors (mainframes) 1980-90’s – Obsolete
Time-share expensive hardware
No change to legacy software
’s – Obsolete
Proliferation of cheap hardware
Hardware support neglected
Later 1990’s – Reincarnation
For complex MPP lacking OS infrastructure
Today: Renaissance
Consolidation, isolation, reliability

19. VM Benefits Performance Security Server consolidation
Efficient HW utilization
Adaptive resource balancing
Checkpoint/restart and migration
Security
Simple (reduced complexity)
Encapsulation and isolation
Mediation

20. VM benefits (contd) Reliability And… Redundancy through replication
Disaster recovery
Deployment testing
And…
Quality of service
Transparent (for legacy SW)
Enhanced interoperability
Development & testing

21. Server utilization Cumulative usage of 28 servers: Memory CPU Disk
45% of RAM not used 99.9% of time
25% of RAM never used concurrently
CPU
85% of CPU not used 99.9% of time
81% of CPU never used concurrently
Disk
68% of storage space never used

22. Virtualization levels
HOST entity: encapsulates the guest
GUEST entity: managed by the host
Application programs
Libraries
API
Operating system
ABI
ISA
Hardware

23. Process & System VM Application Process virtual machine Hardware OS
VMM
Application
Application OS OS
VMM
Virtual
machine
Hardware

24. VM at different levels HW level OS level Programming language level
VMware, Xen, Denali, Virtual PC, UML
OS level
Virtual Servers, BSD Jail, Zap
Programming language level
Java, .NET
Network
VLAN, VPN

25. VM Taxonomy
Process VM - virtual platform that exists solely to support the process
Unix
Emulators (interpreters)
Dynamic binary translators
Optimize by block translation and caching
Java – “compile once run everywhere”
Intermediate machine code
Optimize by native compilation on-the-fly

26. VM Taxonomy (contd)
System VM - complete persistent system environment providing access to virtual hardware
Classic - bare HW
Hosted VM
Easy install and maintenance
Leverage native services of underlying OS
Multiprocessor virtualization

27. Hardware Virtualization
Challenges to build virtual machines
Performance isolation
Scheduling priority
Memory demand
Network traffic
Disk Access
Support for various OS platforms
Small performance overhead

28. Lack of Hardware Support
Ring aliasing
Non-faulting access to privileged state
Does the guest see the right state ?
Address space compression
Where does the VMM reside ?
Impact on transitions
Traps, SYSENTER, SYSEXIT
Interrupts masking
Hidden state

29. Now What ? Hardware extensions Software virtualization
Change semantics to support VM
Intel, AMD
Software virtualization
Translate code to emulate desired behavior
VMware
Paravirtualization
Xen, Denali

30. Hardware Extensions for VM
Root mode
Runs VMM
Like ring-0 before
Non-Root mode
Runs guest OS
Less privileged
Mask of events to trap

31. VMware Hardware virtualization Design goals: CPU, memory, I/O
Suspend/resume
Live migration
Design goals:
Compatibility
Performance
Simplicity

32. VMware: CPU Virtualization
Execute guest on bare hardware while retaining control by the VMM
Traps privileged ops & emulates their action
Challenge: lack of HW support
POPF and read access to privileged state
Solution: fast binary translation
Only kernel mode code
Eliminate unnecessary traps

33. VMware: Memory Virtualization
Shadow page tables
Challenges:
Inefficient page replacement
Oversized due to replication
Solutions:
Ballooning
Content based sharing

34. VMware: I/O Virtualization
Challenge: wide variety of devices and interfaces
Solution:
Hosted architecture
Trap through the VMM
Export special devices

35. Xen: Paravirtualization
Provide some exposure to the underlying hardware
Better performance
Must modify OS to adapt
No modifications to applications

36. Xen (contd) Downgrade privilege of guest OS
Guest registers syscall and page-fault handlers with Xen
Partial access to page tables
Fast handlers for most exceptions
Expose set of simple device abstractions

37. Xen (contd) The cost of porting an OS to Xen: Privileged instructions
Page table access
Network driver
Block device driver
<2% of code-base

38. Denali Lightweight protection domains Changes:
Minimalistic method geared for performance
Changes:
Idle loops - avoid busy wait
Interrupt queueing - save context switch
Interrupt semantics – “just”/”recent”
No virtual memory (!)
No BIOS – no legacy “crap”
Generic I/O devices

39. Virtual Machine Migration
Optimizations:
Reduce memory state before snapshot
ballooning
Reduce total cost by incremental updates
COW hierarchy
Reduce start-up time by paging on-demand
Reduce transfer time relying on common data
Use hash functions to identify common blocks

40. Virtual Machine Migration
Minimizing down time
Reduce size of VM state
Pre-copy static parts (or..)
Demand-copy static parts
Hot-copy dynamic parts

41. OS Virtualization Confine applications in containers Advantages:
Fine granularity
Low overhead
Easier maintenance
Challenges
Transparency
Correctness
Extend OS:
Modify kernel, loadable module, library

42. Isolation – BSD Jail
Create an isolated existing environment via software means.
Uses chroot (private root per jail)
Processes in a jail are isolated from files, processes, or network services in other jails.
A jail can be restricted to a single IP address.

43. Specialized Virtualization – Linux VServer
Hosting (consolidation)
Experimentation
Education (do you trust students … ?)
Personal security box
Manage several "versions“
Applications
Virtual servers
Per user firewall
Fail over servers
Honey-pots

44. Specialized Virtualization – Linux VServer
Isolation
Processes, file system, IPC, network, super user capabilities
Kernel patch
Add a “context” tag per process/resource
syscalls to handle contexts (irreversible)
Challenges
Capture all holes (indirect access !)
Efficient storage

45. General Virtualization – Zap
Virtualization for isolation
POD – PrOcess Domain
Private namespace
Virtualization for migration
Decouple process from OS
Capture state and reconstruct state

46. Zap – virtualization Process environment File system Network
Interpose on system calls
File system
Rely on “chroot” environment
Network
Per protocol methods
Challenges
Race conditions (smp)
Life-span of objects
Fast translation

47. Zap – Migration Checkpoint – outside process context
Capture process tree
Capture pod state
Capture per-process state
Restart – inside process context
Restore process tree
Restore processes
Example issues
Sharing
Deleted files