EGI-InSPIRE EGI-InSPIRE RI The European Grid Infrastructure Steven Newhouse Director, EGI.eu Project Director, EGI-InSPIRE 29/06/2016CoreGrid Workshop - August

EGI-InSPIRE RI The European Grid Infrastructure –Current Status and organisational structure An Evolving Infrastructure –Our challenges and possible directions Engagement with Computer Scientists –Opportunities and collaborations Wrap up Summary 29/06/2016CoreGrid Workshop - August

EGI-InSPIRE RI Infrastructure is the basic physical and organisational structures needed for the operation of a society or enterprise, or the services and facilities necessary for an economy to function 29/06/2016CoreGrid Workshop - August Infrastructure (Wikipedia) The Enterprise is the European Research Area

EGI-InSPIRE RI A grid consists of distributed resources controlled by separate organisations that be systematically used securely by users external to that organisation Resources can include: –Commodity or HPC clusters –Disk or tape storage –Instruments –Data Archives or Digital Libraries What is a Grid? 29/06/2016CoreGrid Workshop - August

EGI-InSPIRE RI European Data Grid (EDG) –Explore concepts in a testbed Enabling Grid for E-sciencE (EGEE) –Moving from prototype to production European Grid Infrastructure (EGI) –Routine usage of a sustainable e-infrastructure 29/06/2016CoreGrid Workshop - August European Grid Infrastructure

EGI-InSPIRE RI European Grid Infrastructure (Apr ‘10) users: +5% LCPUs (cores): +75% 40PB disk: +60% 61PB tape: +56% 15 million jobs/month: +10% 317 sites: +18% 52 countries: +8% 175 VOs: +8% 29 active VOs: +32% 6

EGI-InSPIRE RI EGI Collaboration NGINGI NGINGI NGINGI NGINGI Research Community Research Community Research Community Research Community EIROEIRO EIROEIRO EGI.eu Research Community 29/06/2016CoreGrid Workshop - August

EGI-InSPIRE RI Coordination for European Grid resources –Established February 8th 2010 –Central policy & services needed to run a grid –Sustainable small coordinating organisation Governance & ownership by its participants –EGI Council votes linked to fees –Resources from within its participants Located in Amsterdam with approx. 40 staff –Coordinating core (~20 people) in Amsterdam 29/06/2016CoreGrid Workshop - August EGI.eu EGI and EGI.eu supported by EGI-InSPIRE project

EGI-InSPIRE RI The EGI-InSPIRE Project 29/06/2016CoreGrid Workshop - August Integrated Sustainable Pan-European Infrastructure for Researchers in Europe A 4 year project with €25M EC contribution –Project cost €72M –Total Effort ~€330M –Effort: 9261PMs Project Partners (50) EGI.eu, 39 NGIs, 2 EIROs Asia Pacific (8 partners) Funded Un-Funded

EGI-InSPIRE RI Deploy Technology Innovation –Distributed Computing continues to evolve To include: Grids, Desktops, Virtualisation, Clouds, … Enable Software Innovation –Provide reliable persistent technology platform Tools built on gLite/UNICORE/ARC/Globus Support Research Innovation –Infrastructure for data driven research Support for international research (e.g. ESFRI) 29/06/2016CoreGrid Workshop - August EGI means Innovation

EGI-InSPIRE RI Will come from outside EGI –Moving research technologies into production Partnership with technology projects –EMI (European Middleware Initiative) –IGE (Initiative for Globus in Europe) –EDGI (European Desktop Grid Initiative) –StratusLab –VenusC 29/06/2016CoreGrid Workshop - August Technology Innovation

EGI-InSPIRE RI The DCI Projects ProjectEDGIEGI-InSPIREEMIIGEStratusLabVENUS-C Start Date 01/06/201001/05/ /10/201001/06/2010 Duration (months) Total Budget 2,436,000 €72,000,000 €24,000,000 €3,693,000 €3,137,221 €8,803,046 € Funding from the EC 2,150,000 €25,000,000 €12,000,000 €2,350,000 €2,300,000 €4,500,000 € Total effort in person- month /06/2016CoreGrid Workshop - August 2010

EGI-InSPIRE RI Venus-C EGI-InSPIRE Operational Infrastructure EDGI StratusLab DCI Projects - Taxonomy EMIIGE Resources Technology Users 13 29/06/2016CoreGrid Workshop - August 2010

EGI-InSPIRE RI Advice to the EGI Community –The software base to build middleware e.g. OS, external software –The platform on which to build applications e.g. job submission interface, resource descriptions Components from within the EGI Community –Contained in the Unified Middleware Distribution –Defined integration and support through EGI –Describe what will be released and when. EGI Technology Roadmap 29/06/2016CoreGrid Workshop - August

EGI-InSPIRE RI EGI Technology Roadmap 29/06/2016 CoreGrid Workshop - August UMD Roadmap Components (Developed within EGI) EGI Technology Components Domain Specific Components Time  From outside EGI  Maintained by end-users

EGI-InSPIRE RI Growing the use of the infrastructure –Diversify & expand the user communities –Provide expanded supported software services Deliver the resources more effectively –Add value in providing resources to researchers –Learn & extend lessons from other communities Empowering the end-users –Give more control to our ‘customers’ –Support for data intensive science Critical Influences on EGI 29/06/2016CoreGrid Workshop - August

EGI-InSPIRE RI Storage VMM Clusters VMM Clusters Staff Coordination by EGI.eu Technology assessment, Integrated Operations & User Support GEANT gLite ARC VMM: Virtual Machine Managers Staff Federated Virtualised Infrastructure 29/06/2016CoreGrid Workshop - August 2010

EGI-InSPIRE RI Running Virtual Machine Images remotely –Site policy on trusting the image source –Sandboxing running VMs on a site –Provisioning infrastructure across a grid Resource Allocation and Provisioning –Mapping the VMs to the resources –Co-allocation of VMs, data and networking –Setting Service Level Agreements and monitoring Managing the provisioning infrastructure –Scalable WAN messaging system to integrate it –Autonomic monitoring of the resources Federated Virtualised Infrastructure 29/06/2016CoreGrid Workshop - August

EGI-InSPIRE RI Consider IP network providers –Supports traffic from different communities –Customised solutions within a generic framework –Standards drive integrated deployment And for sustainable e-Infrastructures? –Any application, any domain, any technology –A platform for domain specific innovation & use –Integration of any compliant resource 29/06/2016CoreGrid Workshop - August Be a Neutral Infrastructure

EGI-InSPIRE RI Improve the efficiency of the infrastructure –The jobs, users & data will continue to increase –Effectiveness of the resources needs to match User input into upper middleware layers –VOs decide what services are deployed where –VOs manage their own deployed infrastructure –Empower the VOs to meet their own needs Flexibility and responsibility 29/06/2016CoreGrid Workshop - August In the Future…

EGI-InSPIRE RI Grids have benefited from commoditisation –Hardware: HTC & HPC affordable to all –Networking: GBs can be moved over WAN –Software: Open source software comes of age Impacts of commodity virtualisation… –For transactional models  The ‘Cloud’: A model based on compute not data –For large distributed data-oriented models  The emergence of true ‘function shipping’? 29/06/2016CoreGrid Workshop - August Can we learn from others?

EGI-InSPIRE RI EGEE: –Demonstrated a production e-infrastructure EGI: –Provide a sustainable production e-infrastructure EGI.eu established in Amsterdam –Supported transition through EGI-InSPIRE Contact: 29/06/2016CoreGrid Workshop - August Summary