EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Information Services et al. Standup Chair: Morris Riedel With thanks to Sergio Andreozzi..... and the audience 11/21/2016 Amsterdam,

EGI-InSPIRE RI Summary

EGI-InSPIRE RI Overview and Focus 12/05/2011EGI User Virtualization Workshop3

EGI-InSPIRE RI Scope Information about services living within a VM instance Community-specific  out of scope Perspective: Information about VM instances itself Infrastructure  in scope Information for ‘Scheduling’ (overlaps with Monitoring) Perspective: Discovery Infrastructure  in scope How do we find systems where a dedicated VM image can run? Avoid overlaps (we came back to this again and again…) Monitoring break-out (e.g. health-states, who is running an appliance) Very closely related, also to accounting (tracking resource usage?)

EGI-InSPIRE RI Common Understanding Discover Information about Hypervisor type/virtualization technology Need careful investigation to understand overlaps with monitoring Assuming there will be a ‘submission of VMs to sites possible’… ‘Scheduling related parameters useful to perform decisions’ Different capabilities often related to compute ones… CPU/GPU, # cores, cpu hardware architectures supported (x86, etc.), Machine speed (i.e. GhZ), bandwidth, network type, memory, disk space, IP address, firewall rules, running time) (Also discussed in VM Mgt break-out) End-user information? More scientific end-user input required… Bottom line: basically all that has been useful in Grids !

EGI-InSPIRE RI Other Related Work and Topics Available VM appliances  VM Image Repository Also information about hypervisor types/ virtualization technologies Answers: ‘what images work with what VM management environments OVF will be a key for this – but will all support it? Related but not directly clear and discussed Pricing (Understanding costs of running a VM image on different hosts) Security info: Am I allowed to instantiate a VM via a certain endpoint? Information protection: who is allowed to see what information? Consumers of the information End-users and Brokers/meta-schedulers (e.g. WMS, others) Also the ‘services’ itself work with this information ‘locally’ or to be able to contact ‘related services’… setup working interconnections?

EGI-InSPIRE RI Standards (1) Are there standards we should be using? Information Model Context Many Specifications have covered the same things (mostly compute-based parameters) OCCI defines key/value pairs for VM OVF covers specification of VMs (also network/storage) Work within DMTF (Virtual system profile, Virtual system virtualisation profile, etc.) Leverage GLUE 2.0 with additions? Endpoint to discover ‘VM Management endpoints’ ExecutionEnvironment to discover types of instances where to deploy VM images

EGI-InSPIRE RI Standards (2) GLUE2 Execution Environments…

EGI-InSPIRE RI Standards (4) Are there standards we should be using? Information Discovery Interface context OCCI has a generic discovery interface Including resource description based on key/value pairs Some mandatory: e.g. related to # cores, cpu speed CDMI has a discovery service for data Can express capabilities of the storage system (e.g., type of FS)

EGI-InSPIRE RI Standards (4) OCCI attributes & states (  Monitoring?!)…

EGI-InSPIRE RI Best practices Are there best practices we should adopt? From ‘Early Cloud Work’ StratusLab will address during 2 nd year (federation of cloud systems) Based on OpenNebula (related to monitoring) Info about the physical system (what is available/used) (Ganglia for status  Monitoring?!) From Grid For information model, evaluate type of info captured in GLUE 2.0 Difficult to upgrade information models Due to dependencies on usage/sensors; re-use experience and try to get it right for the ‘emerging VM work’ Quality of data more than completeness Not all information was always really used (  Investigate which ones)

EGI-InSPIRE RI Software Maturity & Availability What is the maturity of the software? What is the availability of the software? How to deliver the information? Traditional models like in Grids now (hierarchy of BDII’s) Alternative and complementary with messaging: ActiveMQ How to aggregate/store/query info BDII/OpenLDAP Evaluate approach of repository as DB vs. in-memory Understand scaling from P2P vs. centralized vs. hierarchical  important when EGI users towards

EGI-InSPIRE RI Priorities What are the priorities? 1.Overall about Information Model: What capabilities need to be represented 1.Compute The following are basically considered as part of compute: VM Appliances information Type of VMs you can use 2.Storage 3.Network 2.Then work on the ‘transport of information’ via useful systems Take into account lessons learned from Grids with scalability, deployment time, etc. 3.Understand overlaps with monitoring and set boundary

EGI-InSPIRE RI Gaps, Issues & Concerns Where are the gaps, issues & concerns? Issue is how to we come to a consensus of which exact capabilities are needed (one agreed profile?) (De-facto) standards define the same things, is this something we can steer or is it rather something ‘out of our control zone’  SDOs? Understanding moves of IEEE Cloud standardization ‘lighthouse’ ‘information isolation’ for users about usage of the infrastructure when performed in monitoring Amount of resources available to a particular user Basically part of the information model, but we need to understand where to set the boundary or granularity level ()

EGI-InSPIRE RI Work items and Next Steps What work is needed to remove these? Filter for GLUEx-based information based on lessons learned Evaluate what is really used in production Understand what information is just overhead Evaluate how to simplify the information needed at finer grain level They have been modeled in Grid but somewhat difficult to capture in a meaningful way Explore “Freshness” of information; should we have a policy in the profile like “info should be not less than 5 minutes’”? Or do we have to think about different models? Current ‘database-based models’ vs. in-memory models

EGI-InSPIRE RI Work needed to remove the gaps From the technical perspective, moving towards virtualization does not show ‘massive big problems’ Issue with revising the info model, the process can take time, but manageable Analyse overlaps and differences between Grid and Cloud standards related to info model/discovery Hopefully leverage common meetings of SDOs (e.g. next week DMTF/OGF/SNIA will meet to discuss integration of OCCI/CDMI/OVF) Start from there for the further work Minimum profile based on outcome or investigate common profile within the community Then filter which capabilities have been really used in Grids and add necessary new ones

EGI-InSPIRE RI Notes from the session

EGI-InSPIRE RI Scope Services, VM images, ? We want to discover info about: Available VM appliances Scheduling related parameters useful to perform decisions Hypervisor type/Virtualization tech. CPU/GPU, # cores… (note from VM Mgt: Bandwidth, network type, memory, disk space, IP address, # cores, CPU, firewall rules, running time) AuthZ info (can I instantiate a VM through a certain endpoint?) Pricing To investigate overlapping of scheduling related info with monitoring requirements (Monitoring: who is starting/running VMs)

EGI-InSPIRE RI Consumers of the Information End-users Brokers/meta-schedulers (e.g., WMS) Other services

EGI-InSPIRE RI Standards Information Model Leverage GLUE 2.0 (?) Endpoint to discover VM Management endpoints ExecutionEnvironment to discover types of instances where to deploy VM images OCCI defines key/value pairs for VM OVF covers specification of VMs/network/storage DMTF Virtual system profile Virtual system virtualisation profile Information Discovery Interface OCCI has a generic discovery interface + resource description based on key/value pairs Some mandatory: e.g. related to # cores, cpu speed CDMI has a discovery service for data Can express capabilities of the storage system (e.g., type of FS)

EGI-InSPIRE RI Best Practices From Cloud StratusLab will address during 2 nd year (federation of cloud systems) Based on OpenNebula (related to monit.) info about the physical system (what is available/used) Ganglia for status From Grid For info model, evaluate type of info captured in GLUE 2.0 Difficult to upgrade information models due to dependencies on usage/sensors; re-use experience and try to get it right Importance for info service, quality of data more than completeness

EGI-InSPIRE RI Guidelines Evaluate how to simplify the information needed at finer grain level They have been modeled in Grid but somewhat difficult to capture in a meaningful way “Freshness” of information; should we have a policy in the profile like “info should be old less than 5’”?

EGI-InSPIRE RI Software How to deliver the information Messaging: ActiveMQ How to aggregate/store/query info BDII/OpenLDAP … Evaluate approach of repository as DB vs. in-memory P2P vs. centralized vs. hierarchical

EGI-InSPIRE RI Priorities Information Model What capabilities need to be represented Compute VM Appliances or as part of VM registries Type of VMs you can use Storage Network

EGI-InSPIRE RI Gaps Isolation between users about usage of the infrastructure Amount of resources available to a particular user

EGI-InSPIRE RI Work needed to remove the gaps About isolation of info, not high-priority From the technical viewpoint, moving towards virtualisation does not show big problems Issue with revising the info model, the process can take time Analyse overlaps and differences between Grid and Cloud standards related to info model/discovery maybe leverage common meetings of SDOs (next week DMTF/OGF/SNIA will meet to discuss integration of OCCI/CDMI/OVF) Minimum profile