StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INSFO-RI-261552 Integrating Cloud Monitoring.

Slides:



Advertisements
Similar presentations
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Management Overview StratusLab.
Advertisements

Canada-EU Future Internet Workshop Waterloo, Canada March 24th, 2011 Ignacio M. Llorente DSA-Research.org Distributed Systems Architecture Research Group.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab: Enhancing grid.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Work Package 4 Software Integration.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INSFO-RI Ioannis Konstantinou Greek.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Sharing Virtual Appliances.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI VM Lifecycle ACGRID-III (Hanoi)
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Introduction to Cloud Technology.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab: Cloud Federation.
1/8 Enhancing Grid Infrastructures with Virtualization and Cloud Technologies Ignacio M. Llorente Business Workshop EGEE’09 September 21st, 2009 Distributed.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab Cloud Distribution.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Cloud Computing and Challenges.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Marketplace & Image Metadata.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks C. Loomis (CNRS/LAL) M.-E. Bégin (SixSq.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Storage Services ACGRID-III.
StratusLab Roadmap StratusLab Tutorial (Orsay, France) 28 November 2012.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Virtual Machine Isolation.
WNoDeS – Worker Nodes on Demand Service on EMI2 WNoDeS – Worker Nodes on Demand Service on EMI2 Local batch jobs can be run on both real and virtual execution.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Client Installation ACGRID-III.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Registration ACGRID-III (Hanoi)
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI The StratusLab Cloud Distribution.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab: Tutorial Recap.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab: Storage Services.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INSFO-RI StratusLab project Update.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab Collaborations.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab: Storage Services.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab: Marketplace EGI.
3 nd Helix Nebula Workshop on Interoperability among e-Infrastructures and Commercial Clouds Sergio Andreozzi Strategy and Policy Manager, EGI.eu EGI Technical.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INSFO-RI Operating grid services on.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab: Enhancing Grid.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Management Overview StratusLab.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Work Package 3 Dissemination.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Demonstration StratusLab First.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Work Package 2 Interactions.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI VM Lifecycle StratusLab Tutorial.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Client Installation StratusLab.
WP5 – Infrastructure Operations Test and Production Infrastructures StratusLab kick-off meeting June 2010, Orsay, France GRNET.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Technical Overview StratusLab.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab: Enhancing Grid.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab Marketplace C.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Work Package 2 Interactions.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Cloud Computing and Challenges.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Services for Distributed e-Infrastructure Access Tiziana Ferrari on behalf.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Work Package 5 Infrastructure.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Virtual Machine Isolation.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Work Package 5 Infrastructure.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Introduction to Cloud Technology.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Exploitation and Sustainability.
The StratusLab Distribution and Its Evolution 4ème Journée Cloud (Bordeaux, France) 30 November 2012.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab: Overview & Collaboration.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI A pan-European Research Infrastructure supporting the digital European Research.
Current Status of the Project C. Loomis CNRS/LAL The StratusLab project is co-funded by the European Community’s Seventh Framework Programme (Capacities)
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab Sustainability.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Work Package 4 Software Integration.
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI Features, Use Cases, and Sustainability.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Role and Challenges of the Resource Centre in the EGI Ecosystem Tiziana Ferrari,
StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INFSO-RI StratusLab: VM Lifecycle EGI.
New Paradigms: Clouds, Virtualization and Co.
C Loomis (CNRS/LAL) and V. Floros (GRNET)
StratusLab First Periodic Review
Cloud Challenges C. Loomis (CNRS/LAL) EGI-TF (Amsterdam)
StratusLab Roadmap C. Loomis (CNRS/LAL) EGI TCB (Amsterdam)
StratusLab Final Periodic Review
StratusLab Final Periodic Review
Grid Computing.
EGI-Engage Engaging the EGI Community towards an Open Science Commons
StratusLab Sustainability
Management of Virtual Execution Environments 3 June 2008
Public vs Private Cloud Usage Costs:
Presentation transcript:

StratusLab is co-funded by the European Community’s Seventh Framework Programme (Capacities) Grant Agreement INSFO-RI Integrating Cloud Monitoring and Accounting with Grid Operational Tools Vangelis Floros, GRNET Instrumenting the Infrastructure, EGI Technical Forum, Amsterdam, 14 Sept. 2010

2 Overview The SratusLab project  Goals  Infrastructure  Work done so far Monitoring and Accounting  Basics  Requirements from the combination of cloud and grid technologies  Impact on operational tools Conclusions and Future work

3 The project StratusLab (StratusLab.eu)  Enhancing Grid Infra. with Virtualization and Cloud Technologies  1 June 2010—31 May 2012 (24 months)  6 partners from 5 countries (CNRS, UCM, GRNET, TCD, SixSq, TID)  Budget: 3.1 M€ (2.3 M€ EC) Activities  WP1 (NA): Project Coordination  WP2 (NA): Interaction with Targeted Communities  WP3 (NA): Dissemination  WP4 (SA): Software Integration and Distribution  WP5 (SA): Infrastructure Operation  WP6 (JRA): Innovative Cloud-like Mgt. of Grid Svcs. and Resources

4 The goals Bridge cloud and grid technologies Understand technology gaps and implications from combining these two approaches in the same environment Provide production level grid services on top of cloud technologies Develop cloud-like approaches for grid service provision

5 Grid Resource Center Grid Resource Center with StratusLab Distribution StratusLab Distribution Private Cloud Cloud API Grid Services Public Clouds users

6 GRNET and CNRS provide the computing infrastructure - 31 nodes, 248 cores ~TB of storage - Connectivity through GÉANT TCD provides the appliance repository - Apache server - Access through WEBDAV - 200~GB and growing StratusLab Infrastructure

7 Work done so far Test installations of OpenNebula 1.4 and 2.0 beta  Official supported platforms RedHat and Debian (actually using CentOS 5.5 and Ubuntu 10.04) Deployed two minimal grid sites  GR-42-STRATUSLAB (CE, SE, 2xWNs)  GR-43-STRATUSLAB (CE, SE, 5xWNs)  SAM tests running in Greek NGI  MPI support enabled Quattor integration and WN tests Started working on aspects of monitoring and accounting

8 Layers of monitoring Grid Services (WMS, SE, BDII, MPI etc) Grid Services (WMS, SE, BDII, MPI etc) Cloud Services (VMs, V storage, V network, hosted software) Cloud Services (VMs, V storage, V network, hosted software) Physical Infrastructure (hosts, network, storage) Physical Infrastructure (hosts, network, storage)

9 Solving the monitoring puzzle Too many things to monitor Individual tools exist to monitor the different layers of such infrastructure  Nagios, Ganglia: Grid and Physical layers  Cloud specific based on virtual machine manager –Cloud interfaces (OCCI, AWS EC2, VMWare vCloud APIs) provide basic monitoring of virtualized resources –Internal/Administration APIs (OpenNebula XMP-RPC) provides extended monitoring of virtualized and physical resources Combined Grid Cloud and Cloud Phys. Infr. needed for a holistic view of the resources

10 Accounting WHO is doing WHAT, WHERE, for HOW LONG Enforce site policies, user quota “Charge” users based on usage Guarantee fair share of resources among users

11 Accounting in the grid and the cloud Grid  A large volume of information is kept enabling the profiling of the resource utilization by grid users  Resources consumed during job submission (CPU, Memory)  Volume of data and number of replicas in SE/LFC  Used primarily to better understand resource utilization and take scheduling decisions Cloud  Pay per usage model  Proper, accurate and detailed accounting is fundamental  Important for charging users based on: –Virtual Appliances used –Network bandwidth consumed –Storage occupied

12 Grid on the Cloud Grid sites on the cloud  Dynamic and Elastic  Nodes can be added and removed on the fly  Storage can be allocated and released on demand to satisfy temporal peak demands  VMs can be migrated to enable optimum performance (e.g. collocate them to run OpenMP and MPI code) Challenge: Apply the pay-per usage model for grid sites  One resource provider might host multiple grid sites form different domains  Fair share among this sites or “credit”-based approach will be required  Implications from hosting grid services in hybrid cloud environments Impact of grid workloads on cloud utilization  Profiling of cloud resource usage  Co-relate cloud utilization with grid workloads

13 Grid/Cloud interaction examples Use the right VM for the right job:  Multicore WNs to run OpenMP code  Collocated WNs (same node or same rack) for optimal inter-process communication suitable for MPI parallel applications Cloud-aware LRMS and/or WMS  Scheduling decision on site and on VO level Cloud-aware BDII  Able to cope with dynamic behavior of virtualized grid sites

14 Conclusions Deploying grid sites over cloud services introduces new challenges Impact on both layers (grid and cloud)  Accounting per VO, site, users  What information should be available to whom Proper monitoring and accounting is crucial  Avoid double accounting  Check consistency between the two layers Gets even more complication when applying cloud bursting approaches using commercial cloud services

15 Future work Work has just started in StratusLab  basic experiments  development of proof-of-concepts OpenNebula evolves  Takes into account requirements stemming from grid-over-cloud  Advanced user management, monitoring and accounting capabilities Will need close interaction with other players  EMI to convey middleware requirements  EGI to adapt grid operations and tools to the dynamic nature of cloud- based grid sites

Copyright © 2010, Members of the StratusLab collaboration: Centre National de la Recherche Scientifique, Universidad Complutense de Madrid, Greek Research and Technology Network S.A., SixSq Sàrl, Telefónica Investigación y Desarrollo SA, and The Provost Fellows and Scholars of the College of the Holy and Undivided Trinity of Queen Elizabeth Near Dublin. This work is licensed under the Creative Commons Attribution 3.0 Unported License