Information Initiative Center, Hokkaido University North 11, West 5, Sapporo , Japan Tel, Fax: General Cluster Deployment Steps Robust Deployment Tool: Considerations 2,3 Layered Development Approach Candidate Technologies for Lower Layers StarCluster 4 + CloudInit.d 2 Chef (+ Knife) 5  Middleware to handle disparate cloud APIs.  Coordination of interdependent launches.  Creation of machine images on different clouds.  Software installation and configuration management.  Policy-driven monitoring, repair, and maintenance of VMs and network links (i.e., Autonomic functions). 1.Node Configuration: Add the IP address of each node to the hosts file of all nodes and associate each IP address with a hostname. 2.User Specification: Define a user to run jobs. 3.Passwordless SSH Setup: Setup passwordless SSH on all nodes so that the master can run commands on the compute nodes. 4.Network Share Setup: Install and setup shared data space (usually NFS) so that shares can be accessed locally on each node to submit and receive jobs. 5.Daemon Setup: Setup a daemon to take care of the running jobs on the nodes. CPAL CIL CPSL Cloud Independent Layer {Higher level services} Cloud Provider Agnostic Layer {Clusters, VPCs, Etc.} Creates a common context Cloud Provider Specific Layer {APIs, CPS-Related Activities} Multi-Cloud Tools {Autonomic Functions, API Mgmt., etc.} * Facilitates a Modular Approach and Structured Development.  Cloud Provider Specific Layer (CPSL)  CloudInit.d {Utility for launching, controlling, and monitoring cloud applications}  Knife (Chef) {Command line interface to the Chef server}  Puppet {Open source configuration management tool}  Cloud Provider Agnostic Layer (CPAL)  StarCluster {Open-source utility that builds, configures, and manages AWS EC2 clusters }  Chef {Open-source systems integration framework for automating the cloud}  Rocks {Open-source Linux cluster distribution for building computational clusters} At Hokkaido University, we are developing an inter-cloud manager called the Simple Heterogeneous INter-CLoud Manager (SHINCLOM) 1 that will enable academic inter-cloud federation and facilitate and manage virtual private academic clouds and clusters via a single sign-on web portal. SHINCLOM will be able to deploy and manage virtual private clouds consisting of virtual machines and virtual private networks across heterogeneous cloud systems controlled by disparate cloud managers and providers. We are currently investigating possible methods of implementing a versatile, robust cluster deployment tool in SHINCLOM using already-existing utilities/technologies.  Launch Plan with services & run levels  Write once, deploy many times  Coordinates interdependent launches and configurations  Deploy on various clouds (uses LibCloud).  Test-based monitoring & policy-driven repair  Paramiko provides secure remote shell access to all instances in cluster.  Config File contains all required settings.  Has Security group for SSH access, Passwordless SSH, Shared disk volume (NFS), etc.  Simple cluster startup: starcluster start mycluster StarCluster CloudInit.d Chef Clusters using Environments & Databags †  Chef Environments facilitates logical infrastructure partitioning.  Dedicated logical group can be created for a cluster.  Chef Databags store information in JSON.  The cluster’s databag can be updated in real time from the nodes. No Need To Reinvent the Wheel!!! Architecture Operation References 1.SHINCLOM, 2.J. Bresnahan, t al., “Managing Appliance Launches in Infrastructure Clouds,” Teragrid 2011, Salt Lake City, UT, July G. Juve and E. Deelman, “Wrangler: Virtual Cluster Provisioning for the Cloud,” Proc. HPDC '11, pp , StarCluster, 5.M. Walls, “Managing Distributed Systems with Chef,” RiCon 2012.