Image is Everything: Dynamic HPC VM Repositories using Murano J. Michael Lowe (jomlowe@iu.edu) Robert Budden (rbudden@psc.edu) Jetstream System Engineer Senior Grid Cluster Systems Developer IU High Performance Systems Pittsburgh Supercomputing Center Jeremy Fischer (jeremy@iu.edu) Senior Technical Advisor / Technical Outreach Indiana University UITS Research Technologies Bob intro and Bridges Mike intro and Jetstream Bob data exacell Mike XSEDE and ACI funded by the National Science Foundation Award #ACI-1445604 Jetstream Award #ACI-1445606 Bridges Award #ACI-1261721 Data Exacell

Why Dynamic VM Repositories? The problem(s): Traditional image development is a never ending task Driven by security patches, updated versions, software requests Time constraints of admins and support staff Slow accumulation of lots of old images if you don’t stay vigilant Difficult to keep consistent over time Size Large of storage for duplicate rarely accessed bits Time consuming and difficult to version and move from place to place Bob

Why Dynamic VM Repositories? Security and provenance Who is really making your image? What are they really doing? Image sanitation Reproducibility Cargo cult SAAS Transparency in installation and configuration Mike

Why Dynamic VM Repositories? Advantages to dynamic image creation: Always up to date! Always consistent (i.e. the maintainer didn’t forget something) Users can still snapshot for specific workflows Sharing orchestration scripts for consistent VMs Solves moving images between clouds It's software, so all of the collaboration tools (I'm looking at you github) work like we expect them to Mike

Further motivations... Enhancing the XSEDE project: Developing a federated Keystone using XSEDE/Globus Authorization Developing a shared image repository for XSEDE Dynamic images would be more easily moved between XSEDE OpenStack resources Contributing to the OpenStack Community Contributing to the Science Working Group OS Community Bob

Putting it to use… Where to start with dynamic images? Base images (minimal install + development tools/libraries) – building blocks for further work Web portals: Building science gateways (e.g. SciGaP http://scigap.org/) Gateways with virtual clusters (SEAGrid project - deployed cluster) Gateways that offload to clusters like Bridges Galaxy instances Virtual Cluster Standalone Galaxy instances Bob

Ok, What do you really mean? Take any base image, use this YAML cloud-init config #cloud-config package_upgrade: true runcmd: - sysctl -w net.core.netdev_max_backlog=300000 - sysctl -w 'net.ipv4.tcp_wmem=4096 87380 16777216' - sysctl -w 'net.ipv4.tcp_rmem=4096 87380 16777216' - sysctl -w net.core.wmem_max=16777216 - sysctl -w net.core.rmem_max=16777216 With any distribution that has cloud-init, you will end up with an instance that is patched and tuned for 10GigE Mike

Ok, What do you really mean? Install packages #cloud-config packages: - "@Development Tools" - python-pip - cmake Mike

OK, what do you really mean? Run shell scripts #cloud-config write_files: - path: /tmp/linux-rootfs-resize.sh content: | #! /bin/bash cd /tmp git clone https://github.com/flegmatik/linux-rootfs-resize.git cd linux-rootfs-resize ./install permissions: '0744' runcmd: - /tmp/linux-rootfs-resize.sh Mike

When just one isn't enough Orchestration is needed for when you want to start more than once instance especially if they do different things Heat – OpenStack orchestration engine and template language Templates reuse cloud-init configs to start several instances that may have dependencies and differentiate them Again, software and text files, so collaboration tools work Bob

Why Murano? Murano lets you select more complex environments vs relatively straightforward Heat templates Easily package entire environments for users Allow researchers to quickly provision an environment on demand without needing to set up security groups or build networks Easily transportable / shared with other Murano-capable clouds and through the OS Community App Catalog Bob

Typical pre-Murano workflow User wants a cluster to plug into their web front end User gets credentials and logs into horizon User is overwhelmed and calls for help User gets help, creates vm, and network User calls for help User creates router having now read the documentation User recreates vm with security group and is happy Mike

Typical post-Murano workflow User wants a cluster to plug into their web front end  User gets credentials and logs into horizon User clicks on cluster application and selects quick start User is happy Mike

Lessons Learned It's relatively easy to work with cloud-init and nova boot or horizon It's an order of magnitude harder to write a good heat template It's an order of magnitude harder to write a good template and then package it for Murano Bob

Lessons Learned – Murano Specific You will need a world accessible rabbit mq and as of Mitaka it can't be a HA cluster Murano is still a moving target, be ware of older images with the agent already installed When things go wrong there are two extra layers to debug compared to good old fashioned nova boot, solid underpinnings are a must Mike

Lessons Learned – Murano Packaging When cloud-init is a hard sell, remember that it can also just be a shim for your favorite configuration management tools (not Murano specific) Many HPC site already using Puppet, Ansible, Salt, CFEngine, etc. Leverage the work already done Convert existing infrastructure to Murano Useful for Staff as well as Users Deploying ”Manged VMs” Bob

Jetstream Partners

Bridges Links XSEDE User Portal is required to actually login: https://portal.xsede.org The Data Exacell https://www.psc.edu/index.php/ research-programs/advanced-systems/ data-exacell User guide: http://psc.edu/bridges

Jetstream Links Jetstream: https://use.jetstream-cloud.org/ XSEDE User Portal is required to actually login: https://portal.xsede.org User guide: http://jetstream-cloud.org/training.php Paper describing Jetstream Jetstream: A self-provisioned, scalable science and engineering cloud environment http://github.com/jetstream-cloud/Jetstream-Salt-States