1. INDIGO-DataCloud RIA-653549 Catania Summit
May 12, 2017, Catania
Resource Virtualization Summary
WP4.2
Storage
Virtualisation
WP4.3
Network
Virtualisation
WP4.1
Computing Virtualisation
Marcus Hardt, KIT
Patrick Fuhrmann, DESY
Zdenek Sustr, CESNET
Patrick Fuhrmann, DESY
Dr. Alvaro Lopez Garcia
Spanish National Research Council – CSIC
INDIGO-DataCloud RIA Catania Summit

2. INDIGO-DataCloud RIA-653549 Catania Summit
Reminder
WP4 provides interoperability for infrastructures in the data , compute and networking area, consequently:
We use standards wherever possible (till it hurts)
We make sure that code is accepted upstream or provide plug-ins to allow customers to use unmodified main stream systems.
Significantly reduce the dependendy on INDIGO
TOSCA
WaTTS
Improved Schedulers
Containers synchronization
Containers support in Cloud Management
OCCI
CDMI
Virtual Router
Quality of Service in Storage
OCCI in Networking
INDIGO-DataCloud RIA Catania Summit

3. INDIGO-DataCloud RIA-653549 Catania Summit
Milestones Month 23
Milestone
Title
MS16
All WP4 components provide the agreed AAI mechanisms
MS20
Support of Batch Systems for containers
MS27
Full Reference Implementation of Quality of Service in storage
MS30
Cross Site access to storage systems
MS31
Prototype implementation for local virtual networks
Please find all INDIGO WP4 Milestones here :
INDIGO-DataCloud RIA Catania Summit

4. INDIGO-DataCloud RIA-653549 Catania Summit
Our Demos
Zdenek
INDIGO-DataCloud RIA Catania Summit

5. INDIGO-DataCloud RIA-653549 Catania Summit
WP4 Demos
Inter-site Networking with Virtual Router
CDMI Based Storage QoS Management
Fair Share Scheduler Usage
bDocker and uDocker
Pre-emptible Instances
WaTTS, the Token Translation Service
INDIGO-DataCloud RIA Catania Summit

6. Contributions merged upstream
Essential in the context of sustainability and acceptance.
OpenStack
NovaDocker
Heat Translator (On the client side) and Tosca Parser
OOI (OCCI for OpenStack)
Cloud info provider has been merged into EGI upstream code.
Keystone library
OPIE: Pre-emptible Instance (in discussion in the open stack developers)
OpenNebula
OneDock
rOCCI
OpenID Connect
OpenID Connect via KeyStone (in the client side, via browser)
Apache: Lib Cloud authentication.
Erlang library
Java implementation for dCache.
INDIGO-DataCloud RIA Catania Summit

7. WP4.1 Update on Computing Virtualisation
RIA
Alvaro Lopez Garcia
Spanish National Research Council - CSIC

8. IaaS Design in Computing
TOSCA Parser
INDIGO-IAM
OpenID
Connect
HEAT Translator
Java-reposync
Local
Repository
DockerHub
INDIGO
Cloud-info-provider
Information system
SSM
(Accounting)
GLUE
JSON
Cloud UR
OCCI (1.2)
TOSCA
Scheduling
FaSS, Synergy
OpenStack
Heat / IM
(Orchestration)
OpenNebula
Docker Container
Virtual Machines
INDIGO-DataCloud RIA Catania Summit

9. INDIGO-DataCloud RIA-653549 Catania Summit
Container Support
Container support in Cloud Management Frameworks
OpenStack (nova-docker)
OpenNebula (ONEDocker).
Container repository (DockerHub) synchronization
Batch system container support
uDocker
MPI Support, run in parallel
bDocker
Almost there
nxt
INDIGO-DataCloud RIA Catania Summit

10. INDIGO-DataCloud RIA-653549 Catania Summit
Improved Scheduling
Synergy
General purpose manager for OpenStack, implementing FairShare scheduling, based on user, group and ‘past usage’ information.
SLURM “Multifactor Algorithm”
External product interacting with OpenStack Compute.
OpenNebula FairShare Scheduler (FaSS)
Under development.
Framework is finished (communication between FaSS and OpenNebula)
The actual algorithm will be provided with an update to Release 2
Will be accepted as an OpenNebula plug-in.
Partition Director
Switches physical compute resources between a Batch System and a Cloud Infrastructure.
Reassigning shares and quotas.
Next step is to provided the PD for other batch systems, e.g. HTCondor.
INDIGO-DataCloud RIA Catania Summit

11. Improved Scheduling (cont.)
OPIE: OpenStack pre-emptible Instances Extension
Pre-emptible instance model: interruptible by higher priority VMs.
Technical prerequisite to support spot instance markets.
Working on upstream integration. (Still in discussion)
Advantages:
Providers can use preemptible instances to get their resource utilization increased
Users can leverage preemptible instances for fault tolerant tasks at a fraction of their normal price
INDIGO-DataCloud RIA Catania Summit

12. INDIGO-DataCloud RIA-653549 Catania Summit
Heat Translator aaS
See yesterdays plenary from IBM
HEAT
HOT
HEAT
Translator
TOSCA
Template
HEAT
Translator
HOT
TOSCA
Client
INDIGO-DataCloud RIA Catania Summit

13. INDIGO-DataCloud RIA-653549 alkdsfjalsdfk
New features in Computing
Container persistency has been added to ONEDock
The Heat Translator, the TOSCA parser and the Infrastructure Manager now support all INDIGO use cases
There is a new OpenStack OCCI Interface (OOI) version, supporting OCCI 1.2
plus the networking part of the specification.
The keystoneauth library for authentication (including support for OpenID Connect) in OpenStack is no longer released as a fork, as all the needed changes have been contributed upstream, e.g. the support for OpenID Connect.
The OpenStack and Nova clients are not released anymore within the INDIGO release. INDIGO is releasing the corresponding pluggable modules for the clients instead.
INDIGO-DataCloud RIA alkdsfjalsdfk

14. Outlook
Improving OpenStack Keystone OpenID Connect support in collaboration with upstream developers.
Better compliance with OpenID Connect specification.
Simpler OpenStack CLI usage.
Release Heat Translator as a Service.
Finalizing direct submission of TOSCA templates to Heat.

15. WP4.2 Storage Virtualisation
RIA
Benjamin, Marcus, Bas
From KIT
Vladimir, Enrico, Andrea
From INFN
Marina, Paul, Patrick, Jürgen, Marina
From DESY
Staszek
From PNSC

16. INDIGO-DataCloud RIA-653549 Catania Summit
Status
RDA (QoS and DLC working group)
QoS WG meetings at all RDA Plenaries
Group still not created, but requested
SNIA
INDIGO-DataCloud recognized as “Commercial Vendor”
Suggested QoS Storage Attribute Extensions approved
Implementation
SNIA Reference Framework significantly improved and production ready
Plugins for StoRM, GPFS, CEPH(S3) and dCache available
INDIGO-DataCloud RIA Catania Summit

17. INDIGO-DataCloud RIA-653549 Catania Summit
From the SNIA web page
Shipping Commercial CDMI Servers
Arsys CloudStorage (Powered by Scality)
Indigo Project - Storage Quality of Service and Data Lifecycle
Coho Data
Compuverde Object Store
Critical Path Messaging Platform (Powered by Scality)
DDN WOS
Mezeo MezeoCloud (Zimbra)
NetApp StorageGRID 9
NetApp StorageGRID Webscale
ProphetStor
Scality Ring
SGI OmniStor (Powered by Scality)
Tarmin GridBank
XOR Systems - Cloud Aqua
INDIGO-DataCloud RIA Catania Summit

18. Evaluation Deployment
Implementation of CDMI extensions
Including workflows, Staging
Webservice at KIT continuously verifies 5 evaluation endpoints:
Location
Type
Quality
KIT
GPFS, HPSS
Disk, Tape
BARI
GPFS
Disk
CNAF
GPFS, StoRM
POZNAN
CEPH, S3
DESY
dCache
INDIGO-DataCloud RIA Catania Summit

19. INDIGO-DataCloud RIA-653549 Catania Summit
To the federated layer
Collected Capabilities
Access Latency
Number of Copies
Storage Lifetime
Location
Available Transition
INDIGO-DataCloud RIA Catania Summit

20. INDIGO-DataCloud RIA-653549 Catania Summit
Outlook
Finally create the RDA working group, document our current findings and let the group agree on what we already did.
Check if CDMI, in it’s current form, is appropriate for further development in QoS and DLC.
Documenting the plug-in structure for others to add more components.
Find a killer customer.
INDIGO-DataCloud RIA Catania Summit

21. INDIGO-DataCloud RIA-653549 Catania Summit
WaTTS
WaTTS creates credentials for services that do not natively support OpenID Connect. Such services are for example ssh, S3 storage, OpenNebula or certificate authorities, CAs.
OIDCC is now OpenID Connect certified client library
Plenty of security enhancements
Rcauth.eu support (Research and Collaboration Authentication CA Service for Europe)
CLI interface now available
INDIGO-DataCloud RIA Catania Summit

22. WP4.3 Update on Network Virtualisation
RIA
Zdenek Sustr
CZNET

23. OCCI Layout for the network
Clients
Servers
INDIGO-DataCloud RIA Catania Summit

24. Network Orchestration
Network Orchestrator Wrapper
Standalone component to orchestrate VXLAN-based networks in OpenNebula.
Included in INDIGO Release 2.
rOCCI-server backend for NOW to enable network management over OCCI.
ROCCI-server + NOW translate OCCI requests, perform consistency checks with the knowledge of the site’s network infrastructure, and pass valid calls to OpenNebula. 24

25. More new features in OCCI
IP Reservations – the ability to have a fixed IP address reserved and assign it to different VMs
Security Groups – firewalls, i.e., setting allowed address and port range combinations per VM
INDIGO has developed OCCI specifications for both features.
An IP reservation is treated as a sub-class of OCCI Network with just one IP.
A security group is a new type of OCCI Resource, that can be linked with any VM.
Implementation in OOI (LIP) and rOCCI+NOW (CESNET) is now ongoing.
INDIGO-DataCloud RIA Catania Summit

26. INDIGO-DataCloud RIA-653549
Inter-site Networking
INDIGO Virtual Router – installed in a private virtual network, provides networking services to VMs in that network independently of the underlying cloud manager. Can set up OpenVPN connections to other INDIGO Virtual Routers, spanning a complex network across hybrid sites.
The INDIGO Virtual router is now available in a preliminary release and has been demonstrated at this conference.
INDIGO-DataCloud RIA

27. INDIGO-DataCloud RIA-653549 Catania Summit
Outlook
Finish the implementation of IP Reservations and Security Groups in OOI and rOCCI+NOW
Implement best practices in the deployment of the INDIGO Virtual Router
Rely more deeply on the features of cloud-init and Ansible
Define a suitable release vector for the INDIGO Virtual Router
Neither RPM/DEB nor Docker are appropriate here
INDIGO-DataCloud RIA Catania Summit