1. From Application to Appliance
Predrag Buncic

2. Overview Virtual Machines Appliances Demo Technology Forecast
Hardware, software & virtual software appliance
Demo
Technology Forecast
Clouds over Grid
Conclusions
CERN, 30/03/

3. Back to the future… Some time ago… Then, things changed.. Today
We had an application
statically linked
running in a VM
prefect isolation
Strict application boundary
Then, things changed..
Unix workstations, PC, commodity computing
Memory was still expensive
Shared libraries, dynamical linking, plugins
Fuzzy application boundary
IBM-VM 360 mainframe, 1988
Today
Memory and disk space is cheap
Virtual Machines running on commodity hardware on Open Source OS are promising to deliver what we lost some time ago
CERN, 30/03/

4. Why (again) Virtual Machines?
The system infrastructure can evolve independently from the evolution of the application
Re-creating application boundary
Now we can Start, Stop, Pause, Migrate VM
Software running inside a VM can not negatively affect the execution of another VM
VMs can provide a perfect process and file sandboxing
The application can (re)use a lot of code which was previously is system/kernel domain
Fancy user space file systems via kernel modules
CERN, 30/03/

5. Virtual Software Appliance = Application + Virtual Machine + Simple UI
Virtual Appliances
This talk is about VM running your favourite application on your laptop or desktop
Virtual Software Appliance = Application + Virtual Machine + Simple UI
Virtual Software Appliance is a lightweight Virtual Machine image that combines
minimal operating environment
specialized application functionality
These appliances are designed to run under one or more of the various virtualization technologies, such as
VMware , Xen, Parallels, Microsoft Virtual PC, QEMU, User mode Linux, CoLinux, Virtual Iron…
Virtual Software Appliances also aim to eliminate the issues related to deployment in a traditional server environment
complex configuration
maintenance
CERN, 30/03/

6. rPath: Software Appliance Company
CERN, 30/03/

7. rPath Technologies Conary rPath Linux rBuilder
the underlying package management technology for rPath Linux
rPath provides instructions to package software from various popular sources and package technologies
rPath Linux
Linux distribution created as the basic operating system for Conary-based appliances
rBuilder
rPath's tool for building appliances
eliminates unnecessary components and provides only the software needed by the applications on your appliance.
rPath Appliance Platform (rAP)
The rPath Appliance Platform (rAP) is rPath's extensible tool providing a web-based user interface for maintaining appliances
rMake
rMake is rPath's tool for allowing full clean rebuilds of the software used in an appliance when significant modifications are made to the core operating system components
CERN, 30/03/

8. Practical exercise: AliEn Appliance
External
Dependencies
AliEn
busybox
(system tools)
ggbox
System devices
Kernel + =
Grid Appliance
CERN, 30/03/

9. /coda -> /opt/alien, /opt/packages
Coda File System
CODA is a distributed file system with its origin in AFS2. It is freely available under a liberal license has several features not found elsewhere:
disconnected operation for mobile computing
high performance through client side persistent caching and server replication
security model for authentication, encryption and access control
continued operation during partial network failures in server network
network bandwidth adaptation
well defined semantics of sharing, even in the presence of network failures
/coda -> /opt/alien, /opt/packages
CERN, 30/03/

10. AliEnX
AliEn Linux – minimal guest OS capable of running AliEn services and hosting Grid applications
Built using rPath tools (rBuilder and Conary package manager)
Very similar to what we were trying to do with AliEn BITS
CERN, 30/03/

11. AliEn Appliance AliEn Appliance Version 0.4
x86 Mountable Filesystem (Xen Virtual Appliance)
x86_64 Mountable Filesystem (Xen Virtual Appliance)
x86 VMware (R) ESX Server Virtual Appliance
x86 Installable CD/DVD
x86_64 Parallels, QEMU (Raw Hard Disk)
x86 Parallels, QEMU (Raw Hard Disk)
Already usable as User Interface
Generic, can be customized for other purposes
To do:
Run Grid Jobs in VM
Prototyping together with Globus developers at Teraport cluster at University of Chicago
Start and manage VMs using Globus Workspace Service
CERN, 30/03/

12. Benchmarks Test machine 1 GB RAM 3 GHz Pentium D
100 particles, standard AliRoot setup
Xen 3.0.3
Native
Simulation
193 s
191.5 s
Reconstruction
52 s
51 s
CERN, 30/03/

13. Download history
CERN, 30/03/

14. Demo time…
CERN, 30/03/

15. AliEn Way: Grid Layered Cake
Large “physical grid(s)”
Reliably execute jobs, store, retrieve and move files
Individual V.O. have at given point in time access to a subset of physical resources
Using standard tools to submit jobs (Job Agents) to physical grid layer, V.O. creates ‘upper’ middleware layer, an overlay grid tailored to V.O needs but on smaller scale
V.O has identity and can handle interactions with physical layer on users behalf
Individual users interacting with V.O middleware will typically see a subset of the resources available to the entire VO
Each session will have certain number of resources allocated
Virtual Cluster (User layer)
Virtual Grid (V.O. layer)
Physical Grid (Common layer)
Reducing scale to achieve scalability
CERN, 30/03/

16. Cloud over Grid SQS + S3 + EC2 Physical Grid (Common)
Simple Queue Service (SQS) offers a reliable, highly scalable hosted queue for storing messages as they travel between computers
Simple Storage Service (S3) provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web
Elastic Computing Cloud (EC2) is a web service that provides re-sizable compute capacity
CERN, 30/03/

17. rPath + Amazon
Software developers use rBuilder to build an Amazon Machine Image (AMI) that is stored using the Amazon Simple Storage Service (Amazon S3).
With a single click, rBuilder and rBuilder Online users can boot their software appliances on Amazon EC2.
User has a complete control of instances
1.7Ghz x86 processor
1.75GB of RAM
60GB of local disk
250Mb/s of network bandwidth.
User can
load them with their unique software appliance image
manage their network's access permissions
and run their image using as many or few systems as they desire.
CERN, 30/03/

18. $ ? $$$ vs ??? Everything has the price… How much would it cost?
$0.15 per GB-Month of storage used
$0.10 per instance-hour consumed
$0.20 per GB of data transferred in/out
How much would it cost?
Store 10PB/year
Use 40k CPUs for processing $ ?
We have faster network
We have 40k CPUs
We have the storage
Why don’t we have 99.99% service availability?
storage
1.8
$18,000,000.00 in
0.2
$2,000,000.00
out
cpu
0.1
$35,040,000.00
$57,040,000.00
CERN, 30/03/

19. What did they do differently?
Storage
Scale as an advantage
Adding nodes to the system increases, not decreases, its availability, speed, throughput, capacity, and robustness
P2P model
99.99% data availability
There can be no single points of failure
Redundant data copies on spinning media
Computing
Elastic
EC2 enables you to increase or decrease capacity within minutes, not hours or days
Integrated
Designed for use with S3
Secure
Uniform
Uses VM technology to provision uniform representation of computational resources to end user
Security and messaging
Web (portal) and Web services provide entry points into the system
But internally, system uses reliable message queues
Event driven rather than service oriented implementation
CERN, 30/03/

20. Use cases for Virtual Machines
Grid
Sandbox environment for job execution on WN
Enhanced site security
VO box
Enhanced Scalability
User Interfaces
Separation of Grid and system environment
Reducing Grid initiation threshold
Specialized environments
PROOF/CAF
process migration
kernel modules to enable fancy user space file systems
P2P like object sharing and caching
Training setups
Make sure that everyone has the same environment when they walk in training room
Testing environments
Easy to setup, saving time and money
CERN, 30/03/

21. Conclusions
Virtual Machine are coming back as viable technology with lots of potential benefits for users and resource providers
The technology is maturing quickly
The business is catching up fast…
… and overtaking us even faster
If we do not want to end up paying a big money to big business we have to learn from them just as they learned from us
lots of technology behind the clouds is the same Open Source software under our fingertips
Are VMs going to solve all our problems?
No.
Do they have a potential to make our life simpler and computing less expensive?
Yes.
CERN, 30/03/