1. Javier Diaz, Gregor von Laszewski, Fugang Wang and Geoffrey Fox
Abstract Image Management and Universal Image Registration for Cloud and HPC Infrastructures
Javier Diaz, Gregor von Laszewski, Fugang Wang and Geoffrey Fox
Community Grids Lab
Pervasive Technology Institute
Indiana University

2. Motivation
FutureGrid (FG) is a testbed providing users with grid, cloud, and high performance computing resources
One of the goals of FutureGrid is to provide a testbed to perform experiments in a reproducible way among different infrastructures
We need mechanism to ease the use of these infrastructures
FG Image Management framework allows users to easily create customized environments by placing suitable images onto the FG resources .
For that to happen, we need to

3. Introduction I
Image management is a key component in any modern compute infrastructure (virtualized or non-virtualized)
Processes part of the image management life-cycle:
Image management is a key component in any modern compute infrastructure, regardless if used for virtualized or non- virtualized resources. We distinguish a number of important processes that are part of the life-cycle management of images.
image creation and customization. Advanced users demand creation and customization of images to fit their particular needs.
sharing the images via a repository. Once we have created an image, we have to store it into a repository. As there are significant differences on how images are managed between IaaS and bare-metal it is necessary to provide an image repository in which we store abstract images that get further modified in the registration process.
Then, we need to register the image in the infrastructure where we want to use it. The registration is typically provided by the underlying infrastructure. However, the images need to be adapted to the particularities of each infrastructure. Moreover, we need to provide a significant toolset to expose registration functionality in bare-metal to non-administrators.
image instantiation. Once the image is registered with the infrastructure, it can be instantiated by the user in the infrastructure. Depending on the infrastructure, users will be able to create VMs or provision bare-metal machines.

4. Introduction II
Targeting multiple infrastructures amplifies the need for mechanisms to ease these image management processes
We have identified two mechanisms
Introduce standards and best practices to interface with the infrastructure (OVF, OCCI, Amazon EC2)
Provide tools that interface with these standards and expose the functionality to the users while hiding the underlying complexities
Otherwise, only the most experienced users will be able to manage images for multiple infrastructures under great investment of time
Since we target multiple infrastructures, it amplifies the need for mechanism to ease these image management processes
We have identified two mechanisms that ease this processes.
OVF(open virtualization format)
OCCI (open cloud computing interface) …
Without these mechanisms, only the most experienced users will be able to manage them under great investment of time

5. FutureGrid Image Management Framework
Framework provides users with the tools needed to ease image management across infrastructures
Users choose the software stacks of their images and the infrastructure/s
Targets end-to-end workflow of the image life-cycle
Create, store, register and deploy images for both virtualized and non-virtualized resources in a transparent way
Allows users to have access to bare-metal provisioning (departure from typical HPC centers)
Users are not locked into a specific computational environment offered typically by HPC centers
Create a framework that provides the tools needed to ease image management across infrastructures by abstracting users from the details.
Users can choose the software they want to have in their images and the infrastructure or infrastructures where they want to register their images. …
It is important to highlight that this framework allows users to have ….
Departure from typical HPC centers where the system administrators decide the OS and the software that you can have in bare-metal. However, this framework allow FG users to have the OS and software they want in bare-metal.

6. Architectural Overview
This picture represents the architecture of our framework.
-we have devised a component for each one of the processes supporting the image life-cycle. This includes an
image generation component to create images following user requirements,
an image repository component to store, catalog and share images,
an image registration component for preparing, uploading and registering images into specific infrastructures such as HPC or different clouds. Once you have registered an image in a particular infrastructure, users can proceed to instantiate VMs or dynamic provision bare-metal machines with the image.
Users can access via a python API, a REST service, a convenient command line shell
One important feature in our design is how we are not simply storing an image but rather focusing on the way an image is created through abstract templating. Thus, it is possible at any time to regenerate an image based on the template describing the software stack and services for a given image. This enables us also to optimize the storage needs for users to manage many images. Instead of storing each image individually, we could just store the template or a pedigree of templates used to generate the images.
To aid storage reduction, our design includes data to assist in measuring usage and performance. This data can be used to purge rarely used images, while they can be recreated on- demand by leveraging the use of templating. Moreover, the use of abstract image templating will allow us to automatically generate images for a variety of hypervisors and hardware platforms on-demand. Autonomous services could be added to reduce the time needed to create images or deploy them in advance. Reusing images among groups of users and the introduction of a cache as part of the image generation will reduce the memory footprint or avoid the generation all together if an image with the same properties is already available.

7. Image Generation Creates images according to user’s specifications:
OS type and version
Architecture
Software Packages
Software installation may be aided by Chef
Images are not aimed to any specific infrastructure
Image stored in Repository or returned to user
This picture represent the workflow of the image generation. After the user introduce the requirements, the image generation service searches into the image repository to identify a base image to be cloned. A base image only contains the OS and minimum required software. If the image generation service finds such an image, it just needs to install the software required by the user and store de image.
In the case that it does not find a base image, it create such an image from scratch. This is done using the tools to bootstrap images provided by the different OSes, such as yum for CentOS and deboostrap for Ubuntu. To deal with different OSes and architectures, we use cloud technologies. Consequently, an image is created with all user specified packages inside a VM instantiated on-demand. Therefore, multiple users can create multiple images for different operating systems concurrently; obviously, this approach provides us with great flexibility, architecture independence, and high scalability. Currently, we use OpenNebula to support this process.
First, the image generation tool searches into the image repository to identify a base image to be cloned, and if there is no good candidate, the base image is created from scratch. Once we have a base image, the image generation tool installs the software required by the user.
One feature of our design is to either create images from scratch or by cloning already created base images we locate in our repository. In the first case, images are created using the tools to bootstrap images provided by the different OSes, such as yum for CentOS and deboostrap for Ubuntu. To deal with different OSes and architectures, we use cloud technologies. Consequently, an image is created with all user specified packages inside a VM instantiated on-demand. Therefore, multiple users can create multiple images for different operating systems concurrently; obviously, this approach provides us with great flexibility, architecture independence, and high scalability. Currently, we use OpenNebula to support this process.
We can speed-up the process of generating an image by not starting from scratch but by using an image already stored in the repository. We have tagged such candidate images in the repository as base images. Consequently, modifications include installation or update of the packages that the user requires. Our design can utilize either VMs or a physical machine to chroot into the image to conduct this step.

8. Image Repository Service to query, store, and update images
Unique interface to store various kind of images for different systems
Images are augmented with some metadata which is maintained in a searchable catalog
Keep data related with the usage to assist performance monitoring and accounting
Independent from the storage back-end. It supports a variety of them and new plugins can be easily created

9. Image Metadata User Metadata
Field Name
Description
imgId
Image’s unique identifier
owner os
Operating system
description
Description of the image
tag
Image’s keywords
vmType
Virtual machine type
imgType
Aim of the image
permission
Access permission
imgStatus
Status of the image
createdDate
Upload date
lastAccess
Last time the image was accessed
accessCount
# times the image has been accessed
size
Size of the image
Field Name
Description
userId
User’s unique identifier
fsCap
Disk max usage (quota)
fsUsed
Disk space used
lastLogin
Last time user used the framework
status
Active, pending, disable
role
Admin, User
ownedimg
# of owned images
These tables collects the metadata associated with images and users. This includes information about properties of the images, the access permission by users and the usage.
Access permissions allow the image owner to determine who has access to the image. The simplest types of sharing include private to owner, shared with the public or shared with a set of people defined by a group/project. Usage information is available as part of the metadata to allow information about usage to be recorded. This includes how many times an image was accessed and by whom.

10. Image Registration I
Adapts and registers images into specific infrastructures
Two main infrastructures types are considered to adapt the image:
HPC: Create network bootable images that can run in bare-metal machines (xCAT/Moab)
Cloud: Convert the images in VM disks and enable VM’s contextualization for the selected cloud
Once the image has been created and stored into the repository, we need to register it into the targeted infrastructure before we can instantiate it. This is performed by the image registration component.

11. Image Registration II
User specifies where to register the image
Optionally, user can select kernel from a catalog
Decides if an image is secure enough to be registered
The process of registering an image only needs to be done once per infrastructure
This figure represents the image registration workflow. Users only need to specify the infrastructure where they want to register their images. Moreover, advance users can also select the kernel they want to use. With this information the image registration service adapt the image for the selected infrastructure. Then, it perform some security checks to determine if the image is secure enough to be registered. Finally, it is registered in the infrastructure.
The process of registering an image only needs to be done once per infrastructure. Therefore, after registering an image in a particular infrastructure, it can be used anytime to instantiate as many VMs or in case of HPC as many physical machines as available to meet the users requirements.
------
Customizes images (network IP, DNS, file system table, kernel modules, etc) for specific infrastructures and and register them
Security checks
-Remove ssh keys from root user, disable password for root, prevent ssh with password..
- Plan to create a platform for instantiating the images in a controlled environment such as a VM with limited network access and perform tests to verify the integrity of the image, detect vulnerabilities and possible malicious software

12. Tests Results obtained from the Analysis of the Image Management Framework

13. Methodology Software deployed on the FutureGrid India cluster
Intel Xeon X5570 servers with 24GB of memory
Single drive 500GB with 7200RPMm 3Gb/s
Interconnection network of 1Gb Ethernet
Software Client is in India’s login node
Image Generation supported by OpenNebula
Image Repository supported by Cumulus (store images) and MongoDB (store metadata)
HPC supported by xCAT, Moab and Torque
Performed different tests to evaluate the Image Generation and the Image Registration tools
Image Generation supported by OpenNebula to allow us to create images for different OSes and architectures. This also provides scalability by simply adding more compute nodes to OpenNebula. So, when we create an image from SCRATCH, OpenNebula starts the appropriate VM and create the image inside.

14. Scalability of Image Generation I
Concurrent requests to create CentOS images from scratch
Increasing number of OpenNebula compute nodes to scale
First, we studied the scalability of the image generator tool. We performed experiments varying the number of concurrent image generation requests (from one to eight) to create CentOS images from scratch. As part of these experiments, we increased the number of OpenNebula compute nodes from one to four to highlight the scalability of the service, which is represented by each one of the lines in the graph.
We observe the overall performance using a single OpenNebula compute node is good as the minimum average wallclock time to create an image with this setup is around six minutes for a single request. When using the same node to handle eight requests, we see an overall time of 16 minutes. To reduce this time, we can increase the number of compute nodes to distribute the workload on other nodes. Finally, we observe a performance degradation when we generate more than two images per compute node.
------
OpenNebula is used to provide scalability and independence from the OS. Each time that we receive an image creation request, we boot a VM with the appropriate OS and create the image in that VM.
Minimum walltime to create an image is around 6 minutes when we have a single request
In the worst scenario (one compute node and eight requests) the walltime is only 16 minutes
Performance degrades when we generate more than two images per compute node

15. Scalability of Image Generation II
Analyze how the time is spent within the image creation process
Only one OpenNebula compute node to better analyze the behavior of each step of the process
Concurrent requests to create CentOS and Ubuntu images
Image creation performed from scratch and reusing a base image from the repository
we analyzed the time spent within the image creation process. For these tests we used a single OpenNebula compute node in order to better analyze the behavior of the different steps involved in the image generation process.

16. Create Image from Scratch
Takes Up to 69%
CentOS
the figures use a stacked bar chart to depict the time spent in each phase of this process including . This (1) boot the VM, (2) generate the image and install user’s software, (3) compress the image, and (4) upload the image to the repository.
we observe the virtualization layer (blue section) introduces significant overhead in the process, which is higher in the case of centos. Next, the orange section represent the time to generate and install the user’s software. Here, generating the image from scratch is the most time consuming part. This is up to 69% for Centos and up to 83% in the case of Ubuntu.
For this reason, we considered to manage base images that can be used to save time during the image creation process.
Ubuntu
Takes Up to 83%

17. Create Image from Base Image
Reduces 55-67%
CentOS
Here, we show the results of creating images using a base image stored in the image repository. So, first, we retrieve the base image and uncompress it, then we install the software required by the user and finally we compress the image and upload to the repository.
By using a base image, the time spent in the creation process is reduced dramatically. In particular, we have reduced the overall time of this process up to an 80% in the case of 1 ubuntu image.
There are two reasons: there is no need to create the base image every time, and we do not need to use the virtualization layer since the base image already has the desired OS and only needs to be upgraded with the software requested by the user.
This means that the time to create an image has been reduced from six minutes to less than two minutes for a single request. For the worst case, we reduced the time from 16 minutes to less than nine minutes, where we used one processors handling eight concurrent requests.
Ubuntu
Reduces 62-80%

18. Scalability of Image Registration
Register the same CentOS image in different infrastructures:
OpenStack (Cactus version configured with KVM hypervisor)
Eucalyptus (2.03 version configured with XEN hypervisor)
HPC (netboot image using xCAT and Moab)
Concurrent registrations in Eucalytpus and Openstack
Only one request at a time is allowed for HPC registration (modifies important parts of the HPC system)
Next, we analyzed the behavior of the image registration processes. For that, we registered the same CentOS image in different infrastructures, namely OpenStack (version Cactus configured with KVM hypervisor), Eucalyptus (v2.03 configured with XEN hypervisor), and HPC (Moab v6.0.3, xCAT 2, Torque v2.5.5). For Eucalyptus and Open- Stack, we utilized concurrent registrations. In contrast, our service to register images in the HPC infrastructure only processes a single request at a time because it modifies critical parts of our HPC infrastructure. Therefore, at this time it must be performed in an atomic section.

19. Register Images on Cloud
Eucalyptus
the figures use a stacked bar chart to depict the time spent in each phase of this process including (1) customization of the image, (2) retrieval of the image after customization, and (3) the upload and registration of the image to a cloud infrastructure.
We observe the time needed to customize the image (1) increases with the number of concurrent requests. This phase includes uncompressing the image to prepare the image to be uploaded and registered to the infrastructure. Therefore, when we process several images at the same time, this phase takes longer. To prevent resource starvation, we have introduced a parameter that limits the maximum number of request that are processed concurrently.
We observe that the time to register an image in OpenStack is higher than in Eucalyptus. This is based on two factors. First, in OpenStack, we need to include certain software to allow OpenStack to contextualize the VM (included in (1)). Second, the process to upload the image to the OpenStack cloud takes longer than in Eucalyptus as we can see in the yellow section. As part of this process, both frameworks compress and split the image in smaller tgz files that are uploaded to the IaaS server. The difference is that OpenStack uncompresses and changes the format of the image in the server side as part of this process, while Eucalyptus seems to maintain the original compressed version.
Time needed to deploy an image in OpenStack is higher than in Eucalyptus
OpenStack we need to include certain software to allow OpenStack to contextualize the VM during the instantiation time
The process to upload the image to the OpenStack cloud takes longer than in Eucalyptus
OpenStack uncompress the image in the server side as part of this process, while Eucalyptus seem to maintain the compress version of the images
Occasionally OpenStack fails to upload some images when we perform several concurrent requests. These images get stuck in the untarring status and do not evolve to the available one. We suspect that it may be due to some problem with the messaging queue system of our OpenStack deployment, but we could not confirm that because the logs were not very helpful.
OpenStack

20. Register Image on HPC
Finally, we show the results obtained from registering images in our HPC infrastructure utilizing Moab/xCAT. We distinguish the following phases (see Section IV-C) : (1) retrieve the image from the repository, (2) uncompress the image, (3) retrieve kernels and update the xCAT tables, and (4) package the image.
We observe that the overall process only takes about two minutes
The most time consuming parts are uncompress the image (orange part) and execute the xCAT packimage command
Creates a tar/cpio image that will be use to netboot bare-metal machines
The scalability of this service is limited because the packimage has to be executed in the machine where xCAT server is installed

21. Conclusions I
We have introduced the FG user-controlled image management framework to handle images for different infrastructures
Framework abstracts the details of each underlying system
Users can easily create and manage customized environments within FG
Replicate software stack on the supported cloud and bare-metal infrastructures
we have presented the FutureGrid user controlled image management framework as a revolutionary way to handle images for different infrastructures spanning virtualized and non-virtualized resources. Eucalyptus, OpenStack, Nimbus, OpenNebula, as well as bare-metal infrastructures

22. Conclusions II
Image management results show a linear increase in response to concurrent requests
Image Generation
Create image from scratch in only 6 min and using a base image in less than 2 min
Scale by adding more nodes to the cloud
Support different OS and arch due to virtualization
Image Registration registers images in any supported infrastructure in less than 3 min
Image Repository supports perfectly the rest of the framework with a negligible overhead
Indirectly we have also seen that our image repository shows excellent behavior in our use cases and introduces only negligible overhead to the overall processes.

23. Ongoing Work
Integrate a messaging queue system (like RabbitMQ or ZeroMQ) to process user’s requests in an asynchronous way
Develop a portal interface
On-demand resource re-allocation between infrastructures (usage, user’s requests)

24. Thank for your attention!!
Contact info:
Javier Diaz:
Gregor Laszewski: