INDIGO-DataCloud RIA-653549 Catania Summit May 12, 2017, Catania Resource Virtualization Summary WP4.2 Storage Virtualisation WP4.3 Network Virtualisation WP4.1 Computing Virtualisation Marcus Hardt, KIT Patrick Fuhrmann, DESY Zdenek Sustr, CESNET Patrick Fuhrmann, DESY Dr. Alvaro Lopez Garcia aloga@ifca.unican.es Spanish National Research Council – CSIC INDIGO-DataCloud RIA-653549 Catania Summit

INDIGO-DataCloud RIA-653549 Catania Summit Reminder WP4 provides interoperability for infrastructures in the data , compute and networking area, consequently: We use standards wherever possible (till it hurts) We make sure that code is accepted upstream or provide plug-ins to allow customers to use unmodified main stream systems. Significantly reduce the dependendy on INDIGO TOSCA WaTTS Improved Schedulers Containers synchronization Containers support in Cloud Management OCCI CDMI Virtual Router Quality of Service in Storage OCCI in Networking INDIGO-DataCloud RIA-653549 Catania Summit

INDIGO-DataCloud RIA-653549 Catania Summit Milestones Month 23 Milestone Title MS16 All WP4 components provide the agreed AAI mechanisms MS20 Support of Batch Systems for containers MS27 Full Reference Implementation of Quality of Service in storage MS30 Cross Site access to storage systems MS31 Prototype implementation for local virtual networks Please find all INDIGO WP4 Milestones here : https://owncloud.indigo-datacloud.eu/index.php/s/JkGqeUa77BtDNPb INDIGO-DataCloud RIA-653549 Catania Summit

INDIGO-DataCloud RIA-653549 Catania Summit Our Demos Zdenek INDIGO-DataCloud RIA-653549 Catania Summit

INDIGO-DataCloud RIA-653549 Catania Summit WP4 Demos Inter-site Networking with Virtual Router CDMI Based Storage QoS Management Fair Share Scheduler Usage bDocker and uDocker Pre-emptible Instances WaTTS, the Token Translation Service INDIGO-DataCloud RIA-653549 Catania Summit

Contributions merged upstream Essential in the context of sustainability and acceptance. OpenStack NovaDocker Heat Translator (On the client side) and Tosca Parser OOI (OCCI for OpenStack) Cloud info provider has been merged into EGI upstream code. Keystone library OPIE: Pre-emptible Instance (in discussion in the open stack developers) OpenNebula OneDock rOCCI OpenID Connect OpenID Connect via KeyStone (in the client side, via browser) Apache: Lib Cloud authentication. Erlang library Java implementation for dCache. INDIGO-DataCloud RIA-653549 Catania Summit

WP4.1 Update on Computing Virtualisation RIA-653549 Alvaro Lopez Garcia aloga@ifca.unican.es Spanish National Research Council - CSIC

IaaS Design in Computing TOSCA Parser INDIGO-IAM OpenID Connect HEAT Translator Java-reposync Local Repository DockerHub INDIGO Cloud-info-provider Information system SSM (Accounting) GLUE JSON Cloud UR OCCI (1.2) TOSCA Scheduling FaSS, Synergy OpenStack Heat / IM (Orchestration) OpenNebula Docker Container Virtual Machines INDIGO-DataCloud RIA-653549 Catania Summit

INDIGO-DataCloud RIA-653549 Catania Summit Container Support Container support in Cloud Management Frameworks OpenStack (nova-docker) OpenNebula (ONEDocker). Container repository (DockerHub) synchronization Batch system container support uDocker MPI Support, run in parallel bDocker Almost there nxt INDIGO-DataCloud RIA-653549 Catania Summit

INDIGO-DataCloud RIA-653549 Catania Summit Improved Scheduling Synergy General purpose manager for OpenStack, implementing FairShare scheduling, based on user, group and ‘past usage’ information. SLURM “Multifactor Algorithm” External product interacting with OpenStack Compute. OpenNebula FairShare Scheduler (FaSS) Under development. Framework is finished (communication between FaSS and OpenNebula) The actual algorithm will be provided with an update to Release 2 Will be accepted as an OpenNebula plug-in. Partition Director Switches physical compute resources between a Batch System and a Cloud Infrastructure. Reassigning shares and quotas. Next step is to provided the PD for other batch systems, e.g. HTCondor. INDIGO-DataCloud RIA-653549 Catania Summit

Improved Scheduling (cont.) OPIE: OpenStack pre-emptible Instances Extension Pre-emptible instance model: interruptible by higher priority VMs. Technical prerequisite to support spot instance markets. Working on upstream integration. (Still in discussion) Advantages: Providers can use preemptible instances to get their resource utilization increased Users can leverage preemptible instances for fault tolerant tasks at a fraction of their normal price INDIGO-DataCloud RIA-653549 Catania Summit

INDIGO-DataCloud RIA-653549 Catania Summit Heat Translator aaS See yesterdays plenary from IBM HEAT HOT HEAT Translator TOSCA Template HEAT Translator HOT TOSCA Client INDIGO-DataCloud RIA-653549 Catania Summit

INDIGO-DataCloud RIA-653549 alkdsfjalsdfk New features in Computing Container persistency has been added to ONEDock The Heat Translator, the TOSCA parser and the Infrastructure Manager now support all INDIGO use cases There is a new OpenStack OCCI Interface (OOI) version, supporting OCCI 1.2
plus the networking part of the specification. The keystoneauth library for authentication (including support for OpenID Connect) in OpenStack is no longer released as a fork, as all the needed changes have been contributed upstream, e.g. the support for OpenID Connect. The OpenStack and Nova clients are not released anymore within the INDIGO release. INDIGO is releasing the corresponding pluggable modules for the clients instead. INDIGO-DataCloud RIA-653549 alkdsfjalsdfk

Outlook Improving OpenStack Keystone OpenID Connect support in collaboration with upstream developers. Better compliance with OpenID Connect specification. Simpler OpenStack CLI usage. Release Heat Translator as a Service. Finalizing direct submission of TOSCA templates to Heat.

WP4.2 Storage Virtualisation RIA-653549 Benjamin, Marcus, Bas From KIT Vladimir, Enrico, Andrea From INFN Marina, Paul, Patrick, Jürgen, Marina From DESY Staszek From PNSC

INDIGO-DataCloud RIA-653549 Catania Summit Status RDA (QoS and DLC working group) QoS WG meetings at all RDA Plenaries Group still not created, but requested SNIA INDIGO-DataCloud recognized as “Commercial Vendor” Suggested QoS Storage Attribute Extensions approved Implementation SNIA Reference Framework significantly improved and production ready Plugins for StoRM, GPFS, CEPH(S3) and dCache available INDIGO-DataCloud RIA-653549 Catania Summit

INDIGO-DataCloud RIA-653549 Catania Summit From the SNIA web page Shipping Commercial CDMI Servers Arsys CloudStorage (Powered by Scality) Indigo Project - Storage Quality of Service and Data Lifecycle Coho Data Compuverde Object Store Critical Path Messaging Platform (Powered by Scality) DDN WOS Mezeo MezeoCloud (Zimbra) NetApp StorageGRID 9 NetApp StorageGRID Webscale ProphetStor Scality Ring SGI OmniStor (Powered by Scality) Tarmin GridBank XOR Systems - Cloud Aqua   INDIGO-DataCloud RIA-653549 Catania Summit

Evaluation Deployment Implementation of CDMI extensions Including workflows, Staging Webservice at KIT continuously verifies 5 evaluation endpoints: Location Type Quality KIT GPFS, HPSS Disk, Tape BARI GPFS Disk CNAF GPFS, StoRM POZNAN CEPH, S3 DESY dCache INDIGO-DataCloud RIA-653549 Catania Summit

INDIGO-DataCloud RIA-653549 Catania Summit To the federated layer Collected Capabilities Access Latency Number of Copies Storage Lifetime Location Available Transition INDIGO-DataCloud RIA-653549 Catania Summit

INDIGO-DataCloud RIA-653549 Catania Summit Outlook Finally create the RDA working group, document our current findings and let the group agree on what we already did. Check if CDMI, in it’s current form, is appropriate for further development in QoS and DLC. Documenting the plug-in structure for others to add more components. Find a killer customer. INDIGO-DataCloud RIA-653549 Catania Summit

INDIGO-DataCloud RIA-653549 Catania Summit WaTTS WaTTS creates credentials for services that do not natively support OpenID Connect. Such services are for example ssh, S3 storage, OpenNebula or certificate authorities, CAs. OIDCC is now OpenID Connect certified client library Plenty of security enhancements Rcauth.eu support (Research and Collaboration Authentication CA Service for Europe) CLI interface now available INDIGO-DataCloud RIA-653549 Catania Summit

WP4.3 Update on Network Virtualisation RIA-653549 Zdenek Sustr CZNET

OCCI Layout for the network Clients Servers INDIGO-DataCloud RIA-653549 Catania Summit

Network Orchestration Network Orchestrator Wrapper Standalone component to orchestrate VXLAN-based networks in OpenNebula. Included in INDIGO Release 2. rOCCI-server backend for NOW to enable network management over OCCI. ROCCI-server + NOW translate OCCI requests, perform consistency checks with the knowledge of the site’s network infrastructure, and pass valid calls to OpenNebula. 24

More new features in OCCI IP Reservations – the ability to have a fixed IP address reserved and assign it to different VMs Security Groups – firewalls, i.e., setting allowed address and port range combinations per VM INDIGO has developed OCCI specifications for both features. An IP reservation is treated as a sub-class of OCCI Network with just one IP. A security group is a new type of OCCI Resource, that can be linked with any VM. Implementation in OOI (LIP) and rOCCI+NOW (CESNET) is now ongoing. INDIGO-DataCloud RIA-653549 Catania Summit

INDIGO-DataCloud RIA-653549 Inter-site Networking INDIGO Virtual Router – installed in a private virtual network, provides networking services to VMs in that network independently of the underlying cloud manager. Can set up OpenVPN connections to other INDIGO Virtual Routers, spanning a complex network across hybrid sites. The INDIGO Virtual router is now available in a preliminary release and has been demonstrated at this conference. INDIGO-DataCloud RIA-653549

INDIGO-DataCloud RIA-653549 Catania Summit Outlook Finish the implementation of IP Reservations and Security Groups in OOI and rOCCI+NOW Implement best practices in the deployment of the INDIGO Virtual Router Rely more deeply on the features of cloud-init and Ansible Define a suitable release vector for the INDIGO Virtual Router Neither RPM/DEB nor Docker are appropriate here INDIGO-DataCloud RIA-653549 Catania Summit