SODA : Service-On-Demand Architecture for Application Service Hosting Utility Platforms Dongyan Xu, Xuxian Jiang Lab FRIENDS (For Research In Emerging Network and Distributed Services) Department of Computer Sciences Center for Education and Research in Information Assurance and Security (CERIAS) Purdue University

Outline  Motivations and goals  Related work  Research components of SODA  Summary and on-going work

Motivations  Vision of utility computing  Computation utility  Storage utility  Application service hosting  Conference management  e-Campaign  Digital government  Serving the underserved communities  IT function shadowing for disaster recovery  Virtual enterprise, collaboratory, and community

Our Goal  To build a value-added application service hosting platform based on shared infrastructure, achieving:  On-demand creation and provisioning  Virtualization  Isolation  Protection  Accountability  Privacy

 Utility computing architectures  VERITAS, HP UDC, IBM Oceano  Grid platforms  Computation: Globus, Condor, Legion, NetSolve, Harness, Cactus  Storage and data: SRB, NeST, Data Grid, OceanStore  Shared infrastructure  PlanetLab, Emulab  Active services  Active Service Grid, Berkeley Active Service Framework, CANS (NYU), Darwin, WebOS Related Work

 Resource isolation  GARA, QLinux (UMass), Virtual service (UMich), Resource Container, Cluster Reserves (Rice)  Virtualization technologies  Virtual super computer (aggregation): NOW, HPVM  Virtual OS, isolation kernel (slicing): VMWare, Xen (Cambridge), Denali (UW), UML, UMLinux, Virtual Private Server (Ensim)  Grid computing on VM: Virtuoso (Northwestern), Entropia  Virtual cluster: Cluster-on-Demand (Duke) Related Work

SODA  Service-On-Demand Architecture for application service hosting utility platforms  Research components of SODA  General architecture  Protection, intrusion detection, logging  Confined and VM-based overlay  Market-driven planning and management

Outline  Research components of SODA:  General architecture  Security and protection  Confined VM-based overlay  ‘Property’ planning and management

Detailed Information  Xuxian Jiang, Dongyan Xu, "SODA: a Service-On-Demand Architecture for Application Service Hosting Utility Platforms", Proceedings of The 12th IEEE International Symposium on High Performance Distributed Computing (HPDC-12), Seattle, WA, June 2003."SODA: a Service-On-Demand Architecture for Application Service Hosting Utility Platforms"HPDC-12

Overview of SODA SODA Host (physical) AS AS’ Virtual service node

Virtualization: Key Technique  Two-level OS structure  Host OS  Guest OS  Strong isolation  Administration isolation  Installation isolation  Fault / attack Isolation  Recovery, migration, and forensics  Virtual service node  Application service (AS)  Guest OS  Internetworking enabled One SODA host Host OS … Guest OS AS 1 AS n

SODA Master SODA Agent Host OS Guest OS Service S SODA Daemon Host OS Guest OS Service S SODA Daemon Host OS Guest OS Service S’ SODA Daemon Guest OS Service S’ Service Switch for S Service Switch for S’ Service Requests From Clients Service Creation Requests From ASP Virtual service node

On the Same SODA Host WWW service Honeypot

Host OS and Guest OS  Guest OS: based on User-Mode Linux (UML), an open-source virtual OS ( different from UMLinux and VServer )  By Jeff Dike,  Running in user space of host OS  Separate kernel address space  Physical memory usage limit  Host OS: Linux (linux , enhanced)  CPU fair share scheduler (for CPU isolation between virtual service nodes)

Experiment: CPU Isolation Original Linux SchedulerEnhanced Linux Scheduler VM 1 : CPU-intensive VM 2 : IO-intensive VM 3 : Web

On-Demand Service Priming  Performed by SODA Daemon  Customization of guest OS (“cook to order” )  Active service image downloading  Automatic bootstrapping of virtual service node

Service Bootstrapping Time Linux Configuration Image size Time (seattle) Time (tacoma) Rootfs_tomrtbt_ MB2.0 sec.3.0 sec. Rootfs_base_ MB3.0 sec.4.0 sec. Root_fs_lfs_ MB4.0 sec.16.0 sec. Root_fs.rh-7.2- server.pristine MB22.0 sec.42.0 sec.

Slow-Down (w/o optimization) 1,368 37,004 gettimeofday 1,200 27,044 munmap 1,208 27,864 mmap 1,084 26,904 dup2 1,064 26,648 geteuid 1,208 27,276 getpid Linux UMLSystem call System call level (clock cycles) Application level

Outline  Research components of SODA:  General architecture  Security and protection  Confined VM-based overlay  ‘Property’ planning and management

Detailed Information  Xuxian Jiang, Dongyan Xu, Rudolf Eigenmann, "Protection Mechanisms for Application Service Hosting Platforms", Proceedings of IEEE/ACM Int'l Symposium on Cluster Computing and the Grid (CCGrid 2004), Chicago, IL, April 2004."Protection Mechanisms for Application Service Hosting Platforms"CCGrid 2004  Xuxian Jiang, Dongyan Xu, "Collapsar: A VM-Based Architecture for Network Attack Detention Center", to appear in Proceedings of the 13 th USENIX Security Symposium (Security '04), San Diego, CA, August 2004."Collapsar: A VM-Based Architecture for Network Attack Detention Center"Security '04

Security and Protection  Virtual switching and firewalling  IDS in guest OS kernel  Untamperable logging (‘blackbox’-ing) Host OS … Guest OS AS 1 AS n

Virtual Switching and Firewalling Virtual machine (with IP addr.) SODA host (Invisible on Internet) Guest OS Host OSFirewall

Kernort: IDS in Guest OS Kernel  Problems with traditional IDS  Encrypted traffic (e.g. ssh) makes NIDS less effective  App-level IDS process will be “killed”, once a machine is compromised  Log may be tampered with  Fail-open  Related projects  Backtracker (Michigan)  VMM-based retrospection (Stanford)  Forensix (OHSU)  ESP (Purdue CERIAS)  Open-source projects: Snort, Saint Jude

Kernort  VM-based IDS  Deployed in each VM  Inside guest OS kernel: a unique vista point  Customizable without affecting host OS  Clearer view  Untamperable logging (saved to SODA host)  Renewable signature (read from SODA host)  Fail-close instead of fail-open

Kernort: IDS in Guest OS Kernel Guest OS

Kernort  Components  Kernort sensor  Event-driven (system call and packet reception)  Renewable signature set  Matching against a small signature set (“Top 20 most wanted”)  Kernort blackbox  Untamperable logging  Privacy preservation of ASes  Analyzer  Exhaustive signature matching  Detection of complex attack patterns  Session replay

Kernort Virtual machine Host OS Kernort (shaded areas: logs)

Real-Time Alert

Session Re-play

Impact on Performance

Outline  Research components of SODA:  General architecture  Security and protection  Confined VM-based overlay  ‘Property’ planning and management

Detailed Information  Xuxian Jiang, Dongyan Xu, "vBET: a VM-Based Emulation Testbed", Proceedings of ACM Workshop on Models, Methods and Tools for Reproducible Network Research (MoMeTools, in conjunction with ACM SIGCOMM 2003), Karlsruhe, Germany, August 2003."vBET: a VM-Based Emulation Testbed"MoMeToolsACM SIGCOMM 2003  Xuxian Jiang, Dongyan Xu, "VIOLIN: Virtual Internetworking on OverLay INfrastructure", Department of Computer Sciences Technical Report CSD TR , Purdue University, July 2003."VIOLIN: Virtual Internetworking on OverLay INfrastructure"  Xuxian Jiang, Dongyan Xu, “A Middleware Architecture for Confined Virtual Machine Overlays", in preparation, March 2004.“A Middleware Architecture for Confined Virtual Machine Overlays"

Traditional Overlay Network  Problems with traditional overlays:  Open for attacks  Attacks from the outside (i.e. Internet) against overlay nodes  Attacks from an overlay node against the outside  Difficult to manage  An overlay across multiple administration domains  A host participate in multiple overlays  Difficult to enforce overlay topology and traffic volume  VPN does not solve the problems

Traditional Overlay Network Firewall

VM-based Overlay  The case for VM-based overlay  Multiple overlays on shared infrastructure  On-demand creation  Confinement and isolation  VM introduces new network administration complexity  “What is this new machine that has suddenly appeared in my domain?”  “Where is the machine that was in my domain yesterday?”  “How much network connectivity should a VM have?”  “How many IP addresses for VMs?”

Confined VM-based Overlay  In addition to VM, we need VN for VMs  VN: a highly overloaded term (VPN, X-bone…)  What is new: Confined and VM-based overlays  Applications  Multi-institutional collaborations  Philanthropic (volunteer) computing systems  Network emulations

Confined VM-based Overlay Firewall VM ≤1Mbps ≤2Mbps Virtual infrastructure

Key Properties  Confined overlay topology and traffic  No attack possible from inside the overlay to the outside world  Virtual IP address space  No need for application modification and re-compilation

A More Generic Picture VIOLIN: Virtual Internetworking on OverLay INfrastructure

vBET: an Example of Confined Overlays on Demand  An education tool for network and distributed system emulation  Fidelity-preserving setup  Maneuverable network entities  Real-world network software  Strict confinement (network security experiment)  Flexible configuration  Not constrained by device/port availability  No manual cable re-wiring or hardware setup  Simultaneous experiments  Cost-effective

vBETvBET Features  Can be deployed in n ≥ 1 vBET servers  Efficient startup and tear-down of emulated entities  Strong network virtualization  IP address space  Virtual routers, switches, firewalls, end-hosts, links  Communications confined by virtual topology  Dynamic addition, deletion, migration, configuration of network entities

vBET GUI

Sample Emulation: OSPF Routing

Emulation of OSPF Routing Demo video clip at:

Sample Emulation: Distributed Firewalls

Screenshot

Sample Emulation: Chord P2P Network

Screenshot

Outline  Research components of SODA:  General architecture  Security and protection  Confined VM-based overlay  ‘Property’ planning and management

Property Planning and Management  Tenant selection:  Among a set of potential tenants (ASes), which ones to host? (for maximum revenue, resource utilization, security…)  SODA provider selection:  Among a set of SODA providers, which one should be chosen to host an AS?

 Examples of bad planning:  Many PDA transcoding ASes in an area with a small PDA user population  AS not requiring client registration and log-in (potential DDoS attacks)  Majority of ASes exhibiting similar demand characteristics such as: Property Planning and Management Load Time

Property Planning and Management  AS profiling  Resource requirement  Security/authentication  Demand characteristics  Market analysis  Competing ASes, market size/growth/expected share  ASes correlation (“80% of clients requesting AS X also request AS Y” )  Trading/pricing of SODA machine slices

Property Planning and Management  Forming alliance of SODA providers

Property Planning and Management  Forming alliance of SODA providers

Summary  Virtualization: a key enabling technology in realizing utility computing vision  Hosting utility is more complex than computation utility (host – tenants – clients)  SODA achieves:  On-demand service creation  Service virtualization, isolation and confinement  Protection, accountability, privacy  Overlay isolation and confinement

Ongoing Work  VM/service migration, shadowing, recovery  Service profiling, accounting, auditing (resources, security)  Market-driven planning, provisioning, and management (SODA ecology)  Deployment and evaluation (Purdue Bindley Bioscience Center)

Thank you. For more information: {dxu, AOL keywords “Purdue SODA Friends”