INDIGO-DataCloud RIA WP4.1 Computing Virtualisation Dr. Alvaro Lopez Garcia Spanish National Research Council – CSIC Marcus Hardt KIT WP4.2 Storage Virtualisation WP4.3 Network Virtualisation Zdenek Sustr CESNET Resource Virtualization May 6, 2016, Frascati

Summary WP4  Thanks to everyone contributing to WP4  Especially thanks to the the task leaders  And to Lukasz for his patience with me  General goals  Finalizing the BIG PICTURE of Indigo  Preparing for the release and agreeing on details  All activities arrived in good shape  Storage specific :  Very detailed discussions on QoS specifications (ongoing)  Working group between WP4 and WP5 needed INDIGO-DataCloud RIA alkdsfjalsdfk

WP4.1 Session Wednesday May 4, 14:00 Aula Direzione, Building 1 Alvaro Lopez Garcia RIA

Summary Session WP 4.1  Main achievements  Repository Synchronization back on track  IAM for OpenNebula solved  “Spot instance” now called “preemtible”  Final ‘first release’ schedule negotiated and agreed.  Container in user space (Jorge promises : nothing illegal) INDIGO-DataCloud RIA

Summary Session WP 4.1  WP Container support  OpenStack support is ready.  Packaging: CERN, INFN-PD, LIP, CNRS.  CNRS is working on packaging for Synergy  OpenNebula support is ready  INFN-TO is going to deploy and test it.  WP Repository sync  Will be done by Atos in time.  Access to OpenStack ok  Access to OpenNebula will be provided by INFN-TO INDIGO-DataCloud RIA

Summary Session WP 4.1 (cont)  WP OCCI support  Support is there for containers. Support for Spot instances will be added. Synergy does not require extensions.  OpenStack implementation has a bug regarding networking, handled (an probably solved) by WP4.3  WP Improved Scheduling  Synergy - Lisa: Adapting the code to Liberty and preparing the release. Will be ready for the 1st release.  FairShare Scheduling - Sara: The people working on this has left INFN, so we are progressing slowly. We expect to have this ready with *limited* functionality for the release. INDIGO-DataCloud RIA

Summary Session WP 4.1 (cont)  Partition Director - Stefano dal Pra  Discussion on the possibility on moving nodes between infrastructures.  Kilo, ok, others will be checked  Works for sure with public API’s. –  Checking QA and polishing code, removing hardcoded values, etc.  Preemptible instances – Alvaro  Implementation ready  Testbed available at IFCA, working on QA.  Peparing for the release.  Spot instances are now called preemptible Instances?  Information will be updated. INDIGO-DataCloud RIA

Summary Session WP 4.1 (cont)  WP Container support in batch systems  Working on the implementation of the support in batch systems, but we do have the possibility to run the container contents in userspace. USeful for sending jobs to the batch system and the Grid. - Alvaro: we do not have a hard deadline for this.  WP TOSCA Orchestration  TOSCA custom types support in IM ready and working.  TOCSA custom types in OpenStack not ready. Working on the testing of the custom types. Need to check with WP5 how to map the TOSCA attributes to actual resources at the sites (i.e. flavors, images, etc). INDIGO-DataCloud RIA

Summary Session WP 4.1 (cont)  WP4.X - IAM integration  IAM is working with OpenStack right now, this is just a configuration issue.  Sites to deploy this as soon as possible. Check that every other component works with it.  Components relying on Keystone should work out of the box.  OpenNebula support:  UPV or INFN-TO should have a look on how to integrate this with the TTS (token translation service). INDIGO-DataCloud RIA

Summary Session WP 4.1 (cont)  Release and release plans  Release of Documentation : should evolve with the software.  Although busy with Packaging : Provide some documentation with the first release.  Proposed date relevant to the release :  FF 30th May  Code Freeze 20th June  RC 23 th june  RC Testing 18th July  Release: 1st August INDIGO-DataCloud RIA

Summary Session WP 4.1 (cont)  Synergy Demo – Lisa  Interaction with nova through the vanilla nova API Alvaro:  Maintaining the "scheduling" status by intercepting the call before it arrives to the scheduler. Whenever we consider that the request is to be scheduled, we pass it again to the scheduler. INDIGO-DataCloud RIA

Summary Session WP 4.1 (cont)  Open issues  nova-docker packaging is pending (task assumed by CNRS).  Ensure that nova-docker is working with all components.  Currently:  OCCI: ok  Synergy: ok  preemptible: ok  TOSCA: not tested, but should work. –  What about IAM integration with Open Nebula? Likely solved  How are we mapping the TOSCA requirements into actual resources for each site?  For instance, in TOSCA you only specify an image as a set of key:value pairs (being the keys architecture, distribution, version, type), so this needs to be mapped to a concrete image ID at each site. How are we handing this? The heat translator needs to have this information available, currently this is hardcoded. Same applies to flavors.  Check with WP5 how are we handling this. INDIGO-DataCloud RIA alkdsfjalsdfk

WP4.2 Session Wednesday May 4, 14:00 Aula A1, Building 36 Marcus Hardt RIA

Summary Session WP 4.2 (cont) INDIGO-DataCloud RIA  Contributions  Benjamin on how we move-on with the SNIA CDMI Reference Implementation  Marina with a DEMO on how dCache:  Handles tape – disk transitions  REST full and GUI interaction for quering and migration triggering  How to render the CDMI for QoS  Maciej  Options to integrate Object Storages into the QoS efforts

Summary Session WP 4.2 (cont) INDIGO-DataCloud RIA alkdsfjalsdfk  CDMI reference Implementaion was improved and runs at 4 sites with static configuration  Agreed on a continuous test system (web page)  RESTful runs at DESY with real dCache integration  Next : CDMI rendering  Currently still discussions ongoing to define “Profile 1” to have something which customers will understand  Plans for Rrelease 1:  CDMI daemon with support for “Profile 1”  Read-only QoS Information  Support to distinguish Disk + Tape  Support to describe it

Genaral CDMI Approach INDIGO-DataCloud RIA CDMI Reference I/O Stack Meta Data Stack POSIX Layer (GPFS, CEPH-FS dCache) Capabilities via DOT-Files QoS Interceptor Backend Adapter System Specific Meta Data API = INDIGO Components ?

Possible Mapping between ‘capabilities’ and ‘attributes’ dCache.org CapabilityDISKTAPE DISK-TAPE Attributes Access Latency (Media) Probability of dataloss 0 s 1 hour Should we ? … 0 s

QoS The Big Picture INDIGO-DataCloud RIA IaaS D&M GUI REST API Canonical Storage Property Information System Platform as a Service D&M GUI REST API

WP4.3 Session Wednesday May 4, 16:30 Aula A1, Building 36 Zdenek Sustr RIA

Summary Session WP 4.3 (cont)  Main success of this meeting: Putting different tasks together.  Status of OCCI networking code contribution to OOI (OpenStack) wrt. Release 1, plus plans thereafter  Progress on SDN evaluation and directions it needs to investigate in more depth  solutions for sites who already have on-site functionality and do not want to change it, just add solutions for inter-cloud  Options to set up testing/pilot environment to test networking features  what can partners do, what can be made available from WP3…  Status of related activities in other projects (Beacon, OpenNebula, EGI Federated Cloud) INDIGO-DataCloud RIA alkdsfjalsdfk

Good luck for the review in October and have a good trip home. INDIGO-DataCloud RIA alkdsfjalsdfk