WP4.1 Computing Virtualisation Alvaro Lopez Garcia, CSICMarcus Hardt, KIT WP4.2 Storage Virtualisation WP4.3 Network Virtualisation Zdenek Sustr, CESNET Rehearsal Bologna, Oct 20. 1

Deliverables 2 NumberDeliverable TitleBeneficiaryDueStatus D 4.1Software Design DocumentDESY6Done D 4.2Detailed Work planDESY6Done D 4.3Status Report and updated Work plan DESY18Delayed D 4.4Network Orchestration and SDN evaluation CESNET18Done D 4.5Final Status ReportDESY30N.A.

Milestones 3 #Milestone Title 14Software design document for WP4 x 25Definition of QoS classes, policies and protocols for storage x 15Delivery of the first of components for WP4 storage and compute area using the agreed AAI mechanism beside X501. In Review 19Support for containers (OpenStack plus OCCI part and rep sync, plus IAM), including rep-sync arch. changes. x 22Implementation of spot instances. x 26First Reference Implementation prototype of QoS classes and policies. x 29First prototype implementation of a seamless solution to data access. x 33Evaluation of the first SDN orchestration. x 18Integration of Service Discovery and Monitoring. WP5

Non-INDIGO IaaS OpenStack OpenNebula Fair Share Scheduling Spot Instances Container Support The IaaS INDIGO Context 4 INDIGO-DataCloud General Architecture PaaS Orchestrator TOSCA Infrastructure Mgr TOSCA PaaS User Portals Mobile Apps Keppler Workflow Portals Remark: Storage and Network not yet included

WP4.1 Update on Computing Virtualisation Alvaro Lopez Garcia, CSIC RIA

Task Overview  Providing Support for Containers at the IaaS layer  Support inside selected Cloud Management Frameworks.  Using open-standards interfaces.  With automatic repository synchronization.  Improving on-demand compute capabilities at the local datacenters  Improving cloud scheduling. (mandatory for production systems)  Facilitating container execution in HPC and HTC systems.  Providing IaaS-level orchestration. 6

WP4.1 : Container Support  Container support in Cloud Management Frameworks  OpenStack (nova-docker) and OpenNebula (ONEDocker).  Maintenance and packaging of nova-docker -> bugfixes and improvements.  Container repository (DockerHub) synchronization  “java-syncrepos”  Batch system container support  uDocker  Running Docker containers in user-space, without Docker at all.  Bdocker  Manager for executing containers in batch-systems  Under development, not yet released INDIGO-1 7

WP4.1 status: IaaS scheduling  Synergy  General purpose manager for OpenStack, implementing FairShare scheduling.  External product interacting with OpenStack Compute.  OpenNebula FairShare Scheduler  Delayed, under development, not yet released.  OPIE: OpenStack Pre-emptible Instances Extension  Pluggable scheduler + API extensions.  Working on upstream integration.  Partition Director  Move (switch) physical resources between a batch system and a cloud infrastructure.  Reassigning shares and quotas.  Released in INDIGO-1 for LSF batch system INDIGO-1 8

WP4.1 status: standards support  OCCI : IaaS management standard and interface.  “ooi” : include OpenStack networking changes.  Both rOCCI (ONE) and “ooi” support containers transparently.  TOSCA: The Orchestration Standard.  “tosca-parser” and “heat-translator”: fixes for supporting custom types. This work has been contributed upstream.  Support in ONE through the Infrastructure Manager (IM).  Support in OpenStack through the “heat-translator”.  Discussions in adding TOSCA support into Heat. (OpenStack orchestration engine with its own orchestration language) 9

WP4.1: other components  Cloud Info Provider  Augmented to implement a JSON rendering.  Changes being merged into EGI's info provider.  One single product for both renderings.  OpenID Connect support (AAI)  ONE support through a TTS instance.  OpenStack Identity support for OpenID Connect support in place  INDIGO configuration guide and profile being elaborated.  Refactored CLI support, merged and released upstream.  Further improvements in the OpenID Connect support in the Identity component being proposed, implemented and merged. 10

Overall Architecture 11

WP4.2 Storage Virtualisation Marcus Hardt / KIT RIA

Cloud Storage Virtualisation 1. Object Storage File system translation 2. Identity Harmonisation 3. Standardisation 4. Implementation of CDMI extensions Four activities to Enable smart management of data on on the IaaS level. 13

Cloud Storage Virtualisation  Four activities to Enable smart management of data on a low level 1. Object Storage File system translation  POSIX-like access to a CEPH backend  Realised via (long) integration chains  CEPH S3 S3FS POSIX  Using a stack of open source implementations  CEPH CDMI  Via dCache 2. Identity Harmonisation  Support consistent access to data with linked accounts 14

Cloud Storage Virtualisation 3. Standardisation: RDA – Research Data Alliance to define the vocabulary  Working Group accepted: QoS-DataLC Definitions WG   Open platform to receive input from end-users  SKOS editor available to submit specific input   Allows communities to contribute to the definition of ’qualities’ SNIA to define the ‘wire’ protocol  Member of the SNIA working group on extensions to the CDMI-Standard  Storage-QoS management in a standardised way 15

Cloud Storage Virtualisation 4. Implementation of CDMI extensions  Built in Java/spring-boot, based on CDMI reference implementation  Understand endpoint storage qualities  Information system to provide details about storage quality  Initial attributes supported:  Access latency (estimated time to first byte)  Location of data (on a country-level granularity for legal purposes)  Number of copies of data  List of available storage qualities into which a migration is available  Modify storage quality  “Bring online” by moving between the available storage classes  Storage systems supported via plugins  TSM/StoRM, dCache, CEPH, HPSS 16

Pluggable CDMI Framework 17 Online CA Identity Harmonization Service CDMI Framework Server Token Translation Service X509 Plug-in Direct Plug-in Set-uid Plug-in INDIGO IAM Google PORTAL OIDC X509 Set UID StoRM Grid Systems dCache GPFS Lustre

 Implementation of CDMI extensions (cont.)  Webservice at KIT continuously verifying 5 evaluation endpoints:   KIT, BARI, CNAF, POZNAN, DESY Cloud Storage Virtualisation 18 Have to find a nicer picture

WP4.3 Update on Network Virtualisation Zdenek Sustr / CEZNET RIA

Virtual Networking in WP4  Main goal  Make virtual networks manageable through a standard interface (OCCI)  Fill in the gaps in functionality, mainly for OpenStack and OpenNebula-based sites  More goals  Survey available SDN options to enable spanning inter-site networks  Initiate/standardize/develop additional features, services or appliances as needed 20

Virtual Networks – Results  Networking functions in OpenStack OCCI Interface  OOI – currently in production on EGI sites  Networking functions and orchestration for OpenNebula (Currently in alpha version)  NOW (Network Orchestrator Wrapper)  standalone component to orchestrate VXLAN-based networks in OpenNebula  rOCCI-server backend for NOW to enable network management over OCCI  Evaluation of SDN capabilities available form vendors  Recommendations and possible approaches for federated environments such as EGI Federated Cloud  INDIGO Deliverable D4.4 21

Virtual Networks – Results  Outline/standardization of additional networking features (in progress)  IP address reservations over OCCI  Preview implementation ready for OpenStack/OCCI  SecurityGroup (IP filter) controls over OCCI  Ambition to push both as extensions to OCCI standard  Drafts ready internally 22

rOCCI client / server big picture 23

For something completely different: CHEP’16 contributions 24

INDIGO Contributions to CHEP 25  The INDIGO-DataCloud Authentication and Authorisation Infrastructure  Optimizing the resource usage in Cloud based environments: the Synergy approach  Storage Quality-of-Service in Cloud-based Scientific Environments: A Standardization Approach  Improved Cloud resource allocation: how INDIGO-Datacloud is overcoming the current limitations in Cloud schedulers  Unified data access to e-Infrastructure, Cloud and personal storage within INDIGO- DataCloud  dCache, managed Cloud Storage  Mesos exploitation within INDIGO-DataCloud for multi-site long-running services, computational workloads and advanced PaaS features

26  TOSCA-based orchestration of complex clusters at the IaaS level  Geographically distributed Batch System as a Service: the INDIGO- DataCloud approach exploiting HTCondor  Facilitating the deployment and exploitation of HEP Phenomenology codes using INDIGO-Datacloud tools  Optimizing the resource usage in Cloud based environments: the Synergy approach  Abstracting application deployment on Cloud infrastructures  A FairShare Scheduling Service for OpenNebula

27 The End