1. https://portal.futuregrid.org Virtual Appliances CTS Conference 2011 Philadelphia May23 2011 Geoffrey Fox gcf@indiana.edu http://www.infomall.org https://portal.futuregrid.orghttp://www.infomall.orghttps://portal.futuregrid.org Director, Digital Science Center, Pervasive Technology Institute Associate Dean for Research and Graduate Studies, School of Informatics and Computing Indiana University Bloomington

2. Exploit electronic infrastructure to enhance learning Several quite old approaches are critical and dominant – “Just a bunch of web pages” aka digital library – Video conferencing – Shared material as in Webex, Adobe Connect Note asynchronous interaction via Twitter, Blackboard, Google docs etc. much easier (and successful) than synchronous (Polycom, access grid, Webex) approaches Interactive web learning environments such as www.whyville.net www.whyville.net Virtual worlds such as Second Life have not taken off but some think this will change as performance of clients and networks are improving dramatically (VRML failed ~1999) Must move to an environment consistent with world view of current students aka the “Twitter University”

3. C 4 Continuous Collaborative Computational Cloud C4C4 I N T E L I G L E N C E Motivating Issues job / education mismatch Higher Ed rigidity Interdisciplinary work Engineering v Science, Little v. Big science Modeling & Simulation C(DE)SE C 4 Intelligent Economy C 4 Intelligent People C 4 Intelligent Society NSF Educate “Net Generation” Re-educate pre “Net Generation” in Science and Engineering Exploiting and developing C 4 C 4 Curricula, programs C 4 Experiences (delivery mechanism) C 4 REUs, Internships, Fellowships Computational Thinking Internet & Cyberinfrastructure Higher Education 2020 CDESE is Computational and Data- enabled Science and Engineering

4. Educational appliances One component of C 4 A flexible, extensible platform for hands-on, lab- oriented education (on FutureGrid) Need to support appliances representing clusters of resources Virtual machines + social/virtual networking to create sandboxed modules – Virtual “Grid” appliances: self-contained, pre-packaged execution environments – Group VPNs: simple management of virtual clusters by students and educators

5. Why use Virtualization? Traditional ways of delivering hands-on training and education in parallel/distributed computing have non-trivial dependences on the environment Difficult to replicate same environment on different resources (e.g. HPC clusters, desktops) Difficult to cope with changes in the environment (e.g. software upgrades) Virtualization technologies remove key software dependences through a layer of indirection

6. Appliance Infrastructure - guiding principles Fidelity: activities should use full-fledged, executable software: education/training modules – Learn using the proper tools Reproducibility: Creators of content should be able to install, configure, and test their modules once, and be assured of the same functional behavior regardless of where the module is deployed – Incentive to invest effort in developing, testing and documenting new modules

7. Appliance Infrastructure - guiding principles Deployability: Students and users should be able to deploy modules in a simple manner, and in a variety of resources – Reduce barriers to entry; avoid dependences upon a particular infrastructure Community-oriented: Modules should be simple to share, discover, reuse, and expand – Create conditions for “viral” growth

8. Towards this vision in FutureGrid Executable modules – virtual appliances – Deployable on FutureGrid resources – Deployable on other cloud platforms, as well as virtualized desktops Community sharing – Web 2.0 portal, appliance image repositories – An aggregation hub for executable modules and documentation

9. https://portal.futuregrid.org What is a virtual appliance? An appliance that packages software and configuration needed for a particular purpose into a virtual machine “ image ” The virtual appliance has no hardware – just software and configuration The image is a (big) file It can be instantiated on hardware 9

10. Virtual appliance example Linux, Java, Hadoop, configuration scripts copy instantiate Hadoop image A Hadoop worker Another Hadoop worker Repeat… Virtualization Layer

11. https://portal.futuregrid.org What about the network? Multiple Web servers might be completely independent from each other Parallel processing: workers are not – Need to communicate and coordinate with each other – Each worker needs an IP address, uses TCP/IP sockets Cluster middleware stacks assume a collection of machines, typically on a LAN (Local Area Network) 11

12. Virtual cluster appliances Virtual appliance + virtual network copy instantiate Hadoop + Virtual Network A Hadoop worker Another Hadoop worker Repeat… Virtual machine Virtual network

13. https://portal.futuregrid.org Virtual cluster appliances Virtual appliance + virtual network 13 copy instantiate MPI + Virtual Network An MPI node Another MPI node Repeat… Virtual machine Virtual network

14. https://portal.futuregrid.org 14 Background Virtual appliances – Encapsulate software environment in image Virtual disk file(s) and virtual hardware configuration The Grid appliance – Encapsulates cluster software environments Current examples: Condor, MPI, Hadoop – Homogeneous images at each node – Virtual LAN connecting nodes to form a cluster – Deploy within or across domains

15. https://portal.futuregrid.org Grid appliance in a nutshell Plug-and-play clusters with a pre-configured software environment – Linux + (Hadoop, Condor, MPI, …) – Scripts for zero-configuration –“ Virtual machine ” appliance; open-source software runs on Linux, Windows, Mac Hands-on examples, bootstrap infrastructure, and zero-configuration software – you ’ re off to a quick start 15

16. https://portal.futuregrid.org Grid appliance in a nutshell Creating an equivalent Grid on your own resources, or on cloud providers, is also easy Deploy image on FutureGrid, Amazon EC2 Copy the same appliance to clusters, PC labs Simple deployment and management of ad-hoc clusters – Opportunistic computing – Testing, evaluation – Education, training 16

17. https://portal.futuregrid.org Virtual Clusters in FutureGrid 17 Nimbus Eucalyptus Appliance image Education Training

18. https://portal.futuregrid.org Social virtual private networks Education/training: deploy your own cluster! 18 copy instantiate MPI + Virtual Network An MPI worker Another MPI worker Repeat… Virtual machine Group VPN GroupVPN Credentials (from Web site) Virtual IP - DHCP 10.10.1.1 Virtual IP - DHCP 10.10.1.2

19. https://portal.futuregrid.org Where to go from here? You can download Grid appliances and run on your own resources You can create private virtual clusters and manage groups of users You can customize appliances with other middleware, create images, and share with other users Tutorials available at FutureGrid.org More information on Grid appliances also available at Grid- appliance.org Contact Renato Figueiredo renato@acis.ufl.edu for more information about appliancesrenato@acis.ufl.edu 19