Javier Diaz, Gregor von Laszewski, Fugang Wang and Geoffrey Fox Abstract Image Management and Universal Image Registration for Cloud and HPC Infrastructures Javier Diaz, Gregor von Laszewski, Fugang Wang and Geoffrey Fox Community Grids Lab Pervasive Technology Institute Indiana University https://portal.futuregrid.org

Motivation FutureGrid (FG) is a testbed providing users with grid, cloud, and high performance computing resources One of the goals of FutureGrid is to provide a testbed to perform experiments in a reproducible way among different infrastructures We need mechanism to ease the use of these infrastructures FG Image Management framework allows users to easily create customized environments by placing suitable images onto the FG resources . For that to happen, we need to https://portal.futuregrid.org

Introduction I Image management is a key component in any modern compute infrastructure (virtualized or non-virtualized) Processes part of the image management life-cycle: Image management is a key component in any modern compute infrastructure, regardless if used for virtualized or non- virtualized resources. We distinguish a number of important processes that are part of the life-cycle management of images. image creation and customization. Advanced users demand creation and customization of images to fit their particular needs. sharing the images via a repository. Once we have created an image, we have to store it into a repository. As there are significant differences on how images are managed between IaaS and bare-metal it is necessary to provide an image repository in which we store abstract images that get further modified in the registration process. Then, we need to register the image in the infrastructure where we want to use it. The registration is typically provided by the underlying infrastructure. However, the images need to be adapted to the particularities of each infrastructure. Moreover, we need to provide a significant toolset to expose registration functionality in bare-metal to non-administrators. image instantiation. Once the image is registered with the infrastructure, it can be instantiated by the user in the infrastructure. Depending on the infrastructure, users will be able to create VMs or provision bare-metal machines. http://futuregrid.org

Introduction II Targeting multiple infrastructures amplifies the need for mechanisms to ease these image management processes We have identified two mechanisms Introduce standards and best practices to interface with the infrastructure (OVF, OCCI, Amazon EC2) Provide tools that interface with these standards and expose the functionality to the users while hiding the underlying complexities Otherwise, only the most experienced users will be able to manage images for multiple infrastructures under great investment of time Since we target multiple infrastructures, it amplifies the need for mechanism to ease these image management processes We have identified two mechanisms that ease this processes. OVF(open virtualization format) OCCI (open cloud computing interface) … Without these mechanisms, only the most experienced users will be able to manage them under great investment of time https://portal.futuregrid.org

FutureGrid Image Management Framework Framework provides users with the tools needed to ease image management across infrastructures Users choose the software stacks of their images and the infrastructure/s Targets end-to-end workflow of the image life-cycle Create, store, register and deploy images for both virtualized and non-virtualized resources in a transparent way Allows users to have access to bare-metal provisioning (departure from typical HPC centers) Users are not locked into a specific computational environment offered typically by HPC centers Create a framework that provides the tools needed to ease image management across infrastructures by abstracting users from the details. Users can choose the software they want to have in their images and the infrastructure or infrastructures where they want to register their images. … It is important to highlight that this framework allows users to have …. Departure from typical HPC centers where the system administrators decide the OS and the software that you can have in bare-metal. However, this framework allow FG users to have the OS and software they want in bare-metal. https://portal.futuregrid.org

Architectural Overview This picture represents the architecture of our framework. -we have devised a component for each one of the processes supporting the image life-cycle. This includes an image generation component to create images following user requirements, an image repository component to store, catalog and share images, an image registration component for preparing, uploading and registering images into specific infrastructures such as HPC or different clouds. Once you have registered an image in a particular infrastructure, users can proceed to instantiate VMs or dynamic provision bare-metal machines with the image. Users can access via a python API, a REST service, a convenient command line shell ------------------ One important feature in our design is how we are not simply storing an image but rather focusing on the way an image is created through abstract templating. Thus, it is possible at any time to regenerate an image based on the template describing the software stack and services for a given image. This enables us also to optimize the storage needs for users to manage many images. Instead of storing each image individually, we could just store the template or a pedigree of templates used to generate the images. To aid storage reduction, our design includes data to assist in measuring usage and performance. This data can be used to purge rarely used images, while they can be recreated on- demand by leveraging the use of templating. Moreover, the use of abstract image templating will allow us to automatically generate images for a variety of hypervisors and hardware platforms on-demand. Autonomous services could be added to reduce the time needed to create images or deploy them in advance. Reusing images among groups of users and the introduction of a cache as part of the image generation will reduce the memory footprint or avoid the generation all together if an image with the same properties is already available. https://portal.futuregrid.org

Image Generation Creates images according to user’s specifications: OS type and version Architecture Software Packages Software installation may be aided by Chef Images are not aimed to any specific infrastructure Image stored in Repository or returned to user This picture represent the workflow of the image generation. After the user introduce the requirements, the image generation service searches into the image repository to identify a base image to be cloned. A base image only contains the OS and minimum required software. If the image generation service finds such an image, it just needs to install the software required by the user and store de image. In the case that it does not find a base image, it create such an image from scratch. This is done using the tools to bootstrap images provided by the different OSes, such as yum for CentOS and deboostrap for Ubuntu. To deal with different OSes and architectures, we use cloud technologies. Consequently, an image is created with all user specified packages inside a VM instantiated on-demand. Therefore, multiple users can create multiple images for different operating systems concurrently; obviously, this approach provides us with great flexibility, architecture independence, and high scalability. Currently, we use OpenNebula to support this process. ------------------ First, the image generation tool searches into the image repository to identify a base image to be cloned, and if there is no good candidate, the base image is created from scratch. Once we have a base image, the image generation tool installs the software required by the user. One feature of our design is to either create images from scratch or by cloning already created base images we locate in our repository. In the first case, images are created using the tools to bootstrap images provided by the different OSes, such as yum for CentOS and deboostrap for Ubuntu. To deal with different OSes and architectures, we use cloud technologies. Consequently, an image is created with all user specified packages inside a VM instantiated on-demand. Therefore, multiple users can create multiple images for different operating systems concurrently; obviously, this approach provides us with great flexibility, architecture independence, and high scalability. Currently, we use OpenNebula to support this process. We can speed-up the process of generating an image by not starting from scratch but by using an image already stored in the repository. We have tagged such candidate images in the repository as base images. Consequently, modifications include installation or update of the packages that the user requires. Our design can utilize either VMs or a physical machine to chroot into the image to conduct this step. https://portal.futuregrid.org

Image Repository Service to query, store, and update images Unique interface to store various kind of images for different systems Images are augmented with some metadata which is maintained in a searchable catalog Keep data related with the usage to assist performance monitoring and accounting Independent from the storage back-end. It supports a variety of them and new plugins can be easily created https://portal.futuregrid.org

Image Metadata User Metadata Field Name Description imgId Image’s unique identifier owner os Operating system description Description of the image tag Image’s keywords vmType Virtual machine type imgType Aim of the image permission Access permission imgStatus Status of the image createdDate Upload date lastAccess Last time the image was accessed accessCount # times the image has been accessed size Size of the image Field Name Description userId User’s unique identifier fsCap Disk max usage (quota) fsUsed Disk space used lastLogin Last time user used the framework status Active, pending, disable role Admin, User ownedimg # of owned images These tables collects the metadata associated with images and users. This includes information about properties of the images, the access permission by users and the usage. -------- Access permissions allow the image owner to determine who has access to the image. The simplest types of sharing include private to owner, shared with the public or shared with a set of people defined by a group/project. Usage information is available as part of the metadata to allow information about usage to be recorded. This includes how many times an image was accessed and by whom.

Image Registration I Adapts and registers images into specific infrastructures Two main infrastructures types are considered to adapt the image: HPC: Create network bootable images that can run in bare-metal machines (xCAT/Moab) Cloud: Convert the images in VM disks and enable VM’s contextualization for the selected cloud Once the image has been created and stored into the repository, we need to register it into the targeted infrastructure before we can instantiate it. This is performed by the image registration component. https://portal.futuregrid.org

Image Registration II User specifies where to register the image Optionally, user can select kernel from a catalog Decides if an image is secure enough to be registered The process of registering an image only needs to be done once per infrastructure This figure represents the image registration workflow. Users only need to specify the infrastructure where they want to register their images. Moreover, advance users can also select the kernel they want to use. With this information the image registration service adapt the image for the selected infrastructure. Then, it perform some security checks to determine if the image is secure enough to be registered. Finally, it is registered in the infrastructure. The process of registering an image only needs to be done once per infrastructure. Therefore, after registering an image in a particular infrastructure, it can be used anytime to instantiate as many VMs or in case of HPC as many physical machines as available to meet the users requirements. ------ Customizes images (network IP, DNS, file system table, kernel modules, etc) for specific infrastructures and and register them Security checks -Remove ssh keys from root user, disable password for root, prevent ssh with password.. - Plan to create a platform for instantiating the images in a controlled environment such as a VM with limited network access and perform tests to verify the integrity of the image, detect vulnerabilities and possible malicious software https://portal.futuregrid.org

Tests Results obtained from the Analysis of the Image Management Framework https://portal.futuregrid.org

Methodology Software deployed on the FutureGrid India cluster Intel Xeon X5570 servers with 24GB of memory Single drive 500GB with 7200RPMm 3Gb/s Interconnection network of 1Gb Ethernet Software Client is in India’s login node Image Generation supported by OpenNebula Image Repository supported by Cumulus (store images) and MongoDB (store metadata) HPC supported by xCAT, Moab and Torque Performed different tests to evaluate the Image Generation and the Image Registration tools Image Generation supported by OpenNebula to allow us to create images for different OSes and architectures. This also provides scalability by simply adding more compute nodes to OpenNebula. So, when we create an image from SCRATCH, OpenNebula starts the appropriate VM and create the image inside. https://portal.futuregrid.org

Scalability of Image Generation I Concurrent requests to create CentOS images from scratch Increasing number of OpenNebula compute nodes to scale First, we studied the scalability of the image generator tool. We performed experiments varying the number of concurrent image generation requests (from one to eight) to create CentOS images from scratch. As part of these experiments, we increased the number of OpenNebula compute nodes from one to four to highlight the scalability of the service, which is represented by each one of the lines in the graph. We observe the overall performance using a single OpenNebula compute node is good as the minimum average wallclock time to create an image with this setup is around six minutes for a single request. When using the same node to handle eight requests, we see an overall time of 16 minutes. To reduce this time, we can increase the number of compute nodes to distribute the workload on other nodes. Finally, we observe a performance degradation when we generate more than two images per compute node. ------ OpenNebula is used to provide scalability and independence from the OS. Each time that we receive an image creation request, we boot a VM with the appropriate OS and create the image in that VM. Minimum walltime to create an image is around 6 minutes when we have a single request In the worst scenario (one compute node and eight requests) the walltime is only 16 minutes Performance degrades when we generate more than two images per compute node http://futuregrid.org

Scalability of Image Generation II Analyze how the time is spent within the image creation process Only one OpenNebula compute node to better analyze the behavior of each step of the process Concurrent requests to create CentOS and Ubuntu images Image creation performed from scratch and reusing a base image from the repository we analyzed the time spent within the image creation process. For these tests we used a single OpenNebula compute node in order to better analyze the behavior of the different steps involved in the image generation process. https://portal.futuregrid.org

Create Image from Scratch Takes Up to 69% CentOS the figures use a stacked bar chart to depict the time spent in each phase of this process including . This (1) boot the VM, (2) generate the image and install user’s software, (3) compress the image, and (4) upload the image to the repository. we observe the virtualization layer (blue section) introduces significant overhead in the process, which is higher in the case of centos. Next, the orange section represent the time to generate and install the user’s software. Here, generating the image from scratch is the most time consuming part. This is up to 69% for Centos and up to 83% in the case of Ubuntu. For this reason, we considered to manage base images that can be used to save time during the image creation process. Ubuntu Takes Up to 83% https://portal.futuregrid.org

Create Image from Base Image Reduces 55-67% CentOS Here, we show the results of creating images using a base image stored in the image repository. So, first, we retrieve the base image and uncompress it, then we install the software required by the user and finally we compress the image and upload to the repository. By using a base image, the time spent in the creation process is reduced dramatically. In particular, we have reduced the overall time of this process up to an 80% in the case of 1 ubuntu image. There are two reasons: there is no need to create the base image every time, and we do not need to use the virtualization layer since the base image already has the desired OS and only needs to be upgraded with the software requested by the user. ------------- This means that the time to create an image has been reduced from six minutes to less than two minutes for a single request. For the worst case, we reduced the time from 16 minutes to less than nine minutes, where we used one processors handling eight concurrent requests. Ubuntu Reduces 62-80% https://portal.futuregrid.org

Scalability of Image Registration Register the same CentOS image in different infrastructures: OpenStack (Cactus version configured with KVM hypervisor) Eucalyptus (2.03 version configured with XEN hypervisor) HPC (netboot image using xCAT and Moab) Concurrent registrations in Eucalytpus and Openstack Only one request at a time is allowed for HPC registration (modifies important parts of the HPC system) Next, we analyzed the behavior of the image registration processes. For that, we registered the same CentOS image in different infrastructures, namely OpenStack (version Cactus configured with KVM hypervisor), Eucalyptus (v2.03 configured with XEN hypervisor), and HPC (Moab v6.0.3, xCAT 2, Torque v2.5.5). For Eucalyptus and Open- Stack, we utilized concurrent registrations. In contrast, our service to register images in the HPC infrastructure only processes a single request at a time because it modifies critical parts of our HPC infrastructure. Therefore, at this time it must be performed in an atomic section. https://portal.futuregrid.org

Register Images on Cloud Eucalyptus the figures use a stacked bar chart to depict the time spent in each phase of this process including (1) customization of the image, (2) retrieval of the image after customization, and (3) the upload and registration of the image to a cloud infrastructure. We observe the time needed to customize the image (1) increases with the number of concurrent requests. This phase includes uncompressing the image to prepare the image to be uploaded and registered to the infrastructure. Therefore, when we process several images at the same time, this phase takes longer. To prevent resource starvation, we have introduced a parameter that limits the maximum number of request that are processed concurrently. We observe that the time to register an image in OpenStack is higher than in Eucalyptus. This is based on two factors. First, in OpenStack, we need to include certain software to allow OpenStack to contextualize the VM (included in (1)). Second, the process to upload the image to the OpenStack cloud takes longer than in Eucalyptus as we can see in the yellow section. As part of this process, both frameworks compress and split the image in smaller tgz files that are uploaded to the IaaS server. The difference is that OpenStack uncompresses and changes the format of the image in the server side as part of this process, while Eucalyptus seems to maintain the original compressed version. ------------ Time needed to deploy an image in OpenStack is higher than in Eucalyptus OpenStack we need to include certain software to allow OpenStack to contextualize the VM during the instantiation time The process to upload the image to the OpenStack cloud takes longer than in Eucalyptus OpenStack uncompress the image in the server side as part of this process, while Eucalyptus seem to maintain the compress version of the images Occasionally OpenStack fails to upload some images when we perform several concurrent requests. These images get stuck in the untarring status and do not evolve to the available one. We suspect that it may be due to some problem with the messaging queue system of our OpenStack deployment, but we could not confirm that because the logs were not very helpful. OpenStack http://futuregrid.org

Register Image on HPC Finally, we show the results obtained from registering images in our HPC infrastructure utilizing Moab/xCAT. We distinguish the following phases (see Section IV-C) : (1) retrieve the image from the repository, (2) uncompress the image, (3) retrieve kernels and update the xCAT tables, and (4) package the image. We observe that the overall process only takes about two minutes The most time consuming parts are uncompress the image (orange part) and execute the xCAT packimage command Creates a tar/cpio image that will be use to netboot bare-metal machines ------------- The scalability of this service is limited because the packimage has to be executed in the machine where xCAT server is installed https://portal.futuregrid.org

Conclusions I We have introduced the FG user-controlled image management framework to handle images for different infrastructures Framework abstracts the details of each underlying system Users can easily create and manage customized environments within FG Replicate software stack on the supported cloud and bare-metal infrastructures we have presented the FutureGrid user controlled image management framework as a revolutionary way to handle images for different infrastructures spanning virtualized and non-virtualized resources. Eucalyptus, OpenStack, Nimbus, OpenNebula, as well as bare-metal infrastructures https://portal.futuregrid.org

Conclusions II Image management results show a linear increase in response to concurrent requests Image Generation Create image from scratch in only 6 min and using a base image in less than 2 min Scale by adding more nodes to the cloud Support different OS and arch due to virtualization Image Registration registers images in any supported infrastructure in less than 3 min Image Repository supports perfectly the rest of the framework with a negligible overhead Indirectly we have also seen that our image repository shows excellent behavior in our use cases and introduces only negligible overhead to the overall processes. https://portal.futuregrid.org

Ongoing Work Integrate a messaging queue system (like RabbitMQ or ZeroMQ) to process user’s requests in an asynchronous way Develop a portal interface On-demand resource re-allocation between infrastructures (usage, user’s requests) https://portal.futuregrid.org

Thank for your attention!! Contact info: Javier Diaz: javier.diazmontes@gmail.com Gregor Laszewski: laszewski@gmail.com http://futuregrid.github.com/rain/ https://portal.futuregrid.org https://portal.futuregrid.org