Eucalyptus on FutureGrid: A case for Eucalyptus 3 Sharif Islam, Javier Diaz, Geoffrey Fox Gregor von Laszewski Indiana University

Abstract In this talk we will be presenting an overview of Eucalyptus used by FutureGrid users. We will provide our experience of running Eucalyptus 2 over multiple years. We conducted performance experiments essential to our community motivating us to switch to Eucalyptus 3. Our experiments are based on running many virtual machines in parallel by the same user in order to coordinate large scale scientific calculations. Gregor von Laszewski,

Bio Gregor von Laszewski was exposed to parallel computers since Currently, he is the Assistant Director for Cloud Computing at the Community Grids Lab at Indiana University and the Software Architect of FutureGrid. He holds a PhD in Computer Science from Syracuse University. He worked in the past for GMD (Germany), NASA, and Argonne National Laboratory. His current interest are in Cloud Computing and works on "rain".

Statistics FutureGrid in general o 920 users o 220 projects FutureGrid Eucalyptus o 285 eucalyptus o number of projects we do not track for eucalyptus Images in Eucalyptus o 120 customized images  (mostly ubuntu and centos images) Gregor von Laszewski,

FutureGrid Project and Technology Requests Gregor von Laszewski,

Normalized Project and Technology Requests in % of total projects Gregor von Laszewski,

Projects... using Eucalyptus Generation of genetic sequencing (Indiana University) Investigate data provenance via MapReduce (Indiana University) Integrating heterogeneous sensor, data and computational resources deployed over a wide area (Indiana University) Distributed systems Graduate Course (Indiana University) Gregor von Laszewski,

Projects... using Eucalyptus (cont.) STAMPEDE: Synthesized Tools for Archiving, Monitoring Performance and Enhanced DEbugging (LBLN) SAGA: Simple API for Grid Applications - Louisiana State University Eucalyptus Usage Metrics Analysis (IU) Virtual Cluster Generation for Clouds (IU) Bare-metal and other Cloud Performance Comparison (IU) Cloud Monitoring (SDSC/IU)

Selected Resources Gregor von Laszewski,

Services on FutureGrid Hardware

Rain (term coined by us) Dynamic provisioning of o HPC services o Virtual machines Image Management o Image templates that run on HPC and clouds o Include authentication and authorization to our user management Resource Management o Fabric Weaving and Cloud Shifting Gregor von Laszewski,

How do we know something is wrong? We run user level tests to identify issues Tests and results are displayed in a dashboard o (red/green) indicates broken and working tests Tests are displayed in historical context to see if something is wrong Gregor von Laszewski,

Tests: Time for VM instantiations (create) &xml=cloudPerf.xml&submit=re-graph results eucalyptus 2.0.3: 272s create 30s ping 30s ssh 0.44s destroy Gregor von Laszewski,

Tests: VM instantiations (ping, ssh) Gregor von Laszewski,

FutureGrid Cloud Metric Tool - What is your user doing? google:github futuregrid cloud metric Gregor von Laszewski,

Improvements between Eucalyptus 2 and 3 Gregor von Laszewski,

Eucalyptus Issues after OS upgrades o Switch from MANAGED-NOVLAN mode from MANAGED mode solved some network problems leading to downtime after a reboot of our systems. An OS upgrade had some adverse effect. o Fresh resinstalation of Xen was needed to solve network issues that occurred after an OS upgrade and Xen was updated. Gregor von Laszewski,

Eucalyptus (issues) Problem with instances o When multiple instances were launched some won't boot up properly. o Instances will remain in pending status forever. Error while communicating with Storage Controller. o Often times euca-describe-volumes won't show the currently created volumes even though the volume appears in the folder. After VmTypes were updated in the cluster configuration instances suddenly started to remain in pending status with o A full restart fixed the issue.

2.0.3 Issues (Cont. ) Memory/resource allocation issue: o This is partly due to lack of compute nodes and how xen handles memory. Eucalyptus will send instances to boot to a particular node when it is overloaded. As a result, even though the node will be scheduled but will fail eventually to boot ("xend.err 'Error creating domain: Not enough free memory"). When DEBUG is on it is very hard to find relevant information in the log. o We need DEBUG "on" to monitor system Gregor von Laszewski,

Eucalyptus We are excited about these changes: Improved handling of multiple instances o "Some instances were not able to access metadata services when multiple instances were launched at the same time." o "When multiple instances were launched at the same time, some remained in a pending state indefinitely." Launching instances after restart o "A fatal parsing error was sometimes reported in the cloud logs when you attempted to launch an instance immediately after restarting the cloud processes."

cont. Improved command line tools o euca-get-console-output with the Xen hypervisor User management o "LDAP and Active Directory Integration." o User management from command line tools o "New and unique identity management allows group- based access control of the resources managed by Eucalyptus." Fewer restarts o Change of VM types (ram, disk) in the cluster configuration does not cause any connectivity issues and does not require a restart.

FutureGrid Software for Clouds Create o Virtual clusters on demand o Hadoop clusters on demand Compare o Cloud and HPC performance o Scalability studies of Cloud infrastructures Configure o Cloud Shifting and Fabric weaving  Move resources between Cloud infrastructures  Deploy Cloud infrastructure on demand

Scalability Tests Study the scalability by instantiating as many virtual machines (VM) at the same time as possible (success if all the machines have ssh access) Our results is the time that takes to have access to all the VMs Performance of Eucalyptus 3 o Tests performed on Sierra o We had 15 physical machines Performance of Eucalyptus 2, OpenStack, OpenNebula and HPC o Tests performed on India o We had 111 physical machines for HPC o We had up to 80 physical machines for Cloud

Results Eucalyptus 3 and 2 Com = Commercial OS = Open Source Gregor von Laszewski,

Results Eucalyptus 3, Eucalyptus 2 and OpenStack Cactus Gregor von Laszewski,

Results all tests Gregor von Laszewski,

Scalability Test: Conclusions The scalability and reliability has been significantly improved in Eucalyptus 3 The time to instantiate VMs has been reduced Very few errors when instantiating more than 16 VMs at the same time o VM does not get IP (Status: pending) The -m option of euca-run-instances works much better than in Eucalyptus 2 and OpenStack o In the tests results the instances were created 10 at a time o Additional tests with Eucalyptus 3 were performed instantiating all VMs with the same euca-run-instances command without problems We will run larger tests once our machine becomes more empty

Next Steps Discontinue use of Eucalyptus 2 Switch the users to Eucalyptus 3 Continue with our Projects o Rain Infrastructure o Measure usage Apply for projects: THANKS! To the Eucalyptus support team that makes all the difference for a deployed Eucalyptus environment and letting us use Eucalyptus 3. Gregor von Laszewski,