EGEE – EGI Transition Jürgen Knobloch, CERN EGI_DS Technical Director WLCG OB CERN 13 November 2009

EGI_DS – European Grid Initiative – Design Study EGI Design Study started Sept 2007 to establish a sustainable pan-European grid infrastructure after the end of EGEE-3 in 2010 The main foundations of EGI are 37 National Grid Initiatives (NGIs) Project is funded by the European Commission's 7 th Framework Program J. Knobloch/CERN: European Grid Initiative – EGI Other EGI_DS members: DFN-Verein (DFN), Germany LMU, Germany Science & Technology Facilities Council (STFC), UK 22-Oct-09 2

The Road to the EGI Blueprint 22-Oct-09 3 J. Knobloch/CERN: European Grid Initiative – EGI WP3 consolidation WS Jan-08 WP3 consolidation WS Jan-08 MW: Coordination – no development Provision of a large-scale production Grid infrastructure MW: Coordination – no development Provision of a large-scale production Grid infrastructure Rome workshop March-08 Rome workshop March-08 D5.1: Draft Definition of EGI Org D5.2: Draft Convention D5.1: Draft Definition of EGI Org D5.2: Draft Convention Geneva workshop 30-June-08 Geneva workshop 30-June-08 Istanbul workshop 22-Sep-08 Istanbul workshop 22-Sep-08 D4.4: Early Draft of Blueprint Second Draft of Blueprint FeedbackFeedback Requirements from NGIs, EGEE Final Blueprint Prague Policy Board 20-Jan-09 Prague Policy Board 20-Jan-09 FeedbackFeedback FeedbackFeedback Feedback from: NGIs, EGEE, other Grids PRACE,WLCG,etc. Feedback from: NGIs, EGEE, other Grids PRACE,WLCG,etc. Blueprint approved!

The EGI Actors J. Knobloch/CERN: European Grid Initiative – EGI NGI 2 NGI 1 NGI n … EGI National Grid Initiatives (NGIs) Resource Centres Research Teams Research Institutes EGI.eu 22-Oct-09 4

EGI Management/Governance J. Knobloch/CERN: European Grid Initiative – EGI EGI.eu EGI Director Non-voting Representatives extra-EU NGIs, Chair of UFSC, … Associate Members e.g. EIROforum member, … Members NGI 1, NGI 2, NGI 3, … NGI n EGI Council EGI Advisory Committees e.g. Middleware Coordination Board (MCB) User Forum Steering Committee (UFSC) User Forum (UF) Middleware Unit Administration & PR Unit Operations Unit User Community Services CTO Middleware Maintenance CAO Admin & PR UCO User Coordination COO Operations 22-Oct-09 5

EGI Financial Reference Model J. Knobloch/CERN: European Grid Initiative – EGI NGI International tasks Middleware maintenance EGI.euTotal Operations User services Costs EGI Effort (FTEs) EGI costs (M€/year) Income NGI membership fees (M€/year) 1,01 EC Contribution (M€/year) NGI, Middleware consortia and EIRO/ESFRI co- funding Contribution (M€/year) Oct-09 6

EGI Operations Model NGI’s are the main actors Aims –Autonomy for NGIs  using a common infrastructure for local and international work for greater efficiency –Sustainability –Subsidiarity: do things at as local a level as possible (EGI.eu pulls things together) –Increased reliability through pushing responsibility down to sites –Preserving current scalability in presence of more middleware stacks to be supported, more non-EGEE Grids integrated,... 7 J. Knobloch/CERN: European Grid Initiative – EGI22-Oct-09

J. Knobloch/CERN: European Grid Initiative – EGI 8 EGI Operations Tasks (1/2) Performed by the NGI’s with the support of EGI.eu Operation of tools and services –Grid configuration repositories –Grid accounting repositories –Grid repositories for SLA compliance and performance monitoring –Grid operations portal –Monitoring of status of services operated by sites User support –Central ticket handling system –Gathering requirements for user support tools and process 22-Oct-09

J. Knobloch/CERN: European Grid Initiative – EGI 9 EGI Operations Tasks (2/2) Security –Security policy development and maintenance –Coordination of security and incident response –Expert team for security vulnerabilities in MW Other international tasks –Middleware deployment/roll-out and support –Brokering availability of NGI’s resources for VO’s –Interoperations between NGI’s and with other grids –Network support –Definition of best practises, procedures, requirements –Catch-all production grid core services 22-Oct-09

Characteristics of NGIs Each NGI … should be a recognized national body with a single point-of-contact … should mobilise national funding and resources … should ensure an operational national grid infrastructure … should support user communities (application independent, and open to new user communities and resource providers) … should contribute and adhere to international standards and policies Responsibilities between NGIs and EGI are split to be federated and complementary J. Knobloch/CERN: European Grid Initiative – EGI22-Oct-09 10

The NGI Tasks NGI International Tasks –Necessary for the functioning of any NGI EGI Global Tasks –Central tasks part of EGI.eu not necessarily centrally located –Bidding by NGIs – selection procedure – review by Council Regional Operations Centres (ROC)  NGIs –ROCs organized by geographical region –Migration will proceed in steps –Decentralise key tasks –Devolution to NGIs will continue after EGEE III 22-Oct J. Knobloch/CERN: European Grid Initiative – EGI

2. EGI_DS Review 12 NGIs in Europe

Grid Middleware 22-Oct J. Knobloch/CERN: European Grid Initiative – EGI Authentication of users Authorization of VOs to run jobs Distribution and scheduling of jobs Monitoring of submitted jobs Accounting of users and VOs Management of data

J. Knobloch/CERN: European Grid Initiative – EGI 14 Role of EGI.eu in Middleware Central technical coordination by EGI.eu middleware unit (led by CTO) Objectives for EGI in middleware field –Maintenance –Support –Development for standard compliance and stack convergence –Developments/Procurement as required by Applications and Operations –Certification –Establishment of common requirements for interoperability and new developments Only last 2 points by EGI.eu personnel (8 FTE’s) The other ones by Consortia seeking EU co-funding in agreement with EGI, but outside its budget –A first estimation for support, maintenance and devel of the components in the list in the next slide is about 100 FTE’s 22-Oct-09

EGI.eu Functionality Overview Management, Outreach & Dissemination - Representation of EU Grid Efforts Operations & Resource Provisioning & Security Application Support & Training Middleware (Build&Test, Component Selection/Validation/Deployment) Standardisation & Policies Industry take-up J. Knobloch/CERN: European Grid Initiative – EGI22-Oct-09 15

J. Knobloch/CERN: European Grid Initiative – EGI 16 Application Support and Training The goal of EGI is to offer the European Research Teams the means for an easier and more efficient use of IT resources, for their Applications The primary actors of grid Application Support and Training are the NGI’s International Applications will require coordinated NGI’s effort –EU co-funding expected for this critical NGI international task –Coordination implemented via Specialised Support Centres, manned and controlled by the NGI’s For EGI.eu only some specific “gluing” tasks 22-Oct-09

Benefits provided by EGI The current pan-European grid infrastructure established by EGEE and related projects is heavily used – the (growing) user communities rely on continuity and sustainability. EGI can ensure a common infrastructure avoiding divergence between countries by establishing a National Grid Initiative in each member state and a relatively small central EGI.eu International and interdisciplinary research projects and communities needing to share data and compute resources can count on a pan-European infrastructure, thus avoiding duplication of such infrastructures Resource providers can experience an efficient resource sharing when international research teams use a unified middleware. Operational efforts and costs can be partially shared. Funding agencies: –National level: better global return for the money invested by the funding bodies for the procurement of ICT resources, pushing for the creation of global pools based on a well defined certified set of services –International level: better return from past EC investments J. Knobloch/CERN: European Grid Initiative – EGI22-Oct-09 17

Transition in progress The transition is in progress – already achieved these steps: –Acceptance of the strategy defined in the Blueprint by the EGI_DS Policy Board; –Choice of the location of the central EGI location, EGI.eu (previously called EGI.org) in Amsterdam; –Acceptance of the EGI Memorandum of Understanding (MoU) and subsequent establishment of the EGI Council; –Appointment of the interim EGI Director; –Publication of calls for proposals in the 7 th Framework of the European Commission targeted to EGI and related e-infrastructures; –Establishment of editorial structure for the preparation of proposals for EGI-InSPIRE, SSCs, Middleware, and other topics related to EGI; –gLite Consortium – Collaboration Agreement signature progressing; –Drafts of proposals being finalized  11 days to go! –S. Newhouse elected director of InSPIRE 22-Oct J. Knobloch/CERN: European Grid Initiative – EGI

Memorandum of Understanding Goals –Identify parties ready to commit resources –Establish EGI (interim) collaboration First MoU draft at Policy Board March 2 nd –Followed by work of legal experts Second MoU draft May 7 th –Accompanied by Letter of Intent (LoI) to sign MoU distributed for signature June 13 th 24 signatures  Council meeting July 9 th 22-Oct J. Knobloch/CERN: European Grid Initiative – EGI

EGI Council 22-Oct J. Knobloch/CERN: European Grid Initiative – EGI 22 Oct 2009: 33 NGIs 2 EIRO (CERN, EMBL) + Observers (Armenia, Belarus, Georgia, Kazakhstan, Moldova, Russia, Ukraine)

EGI Questionnaire to NGIs March 6 th submitted questionnaire to Policy Board members –24 countries, institutions filled questionnaire: Austria, Belgium, Belarus, Bulgaria, CERN, Croatia, Czech Rep, Finland, France, Germany, Greece, Ireland, Italy, Lithuania, Moldova, Netherlands, Poland, Portugal, Slovakia, Spain, Sweden, Switzerland, Turkey, UK –10 countries provided only contact names: Albania, Belarus, Denmark, NDGF, Georgia, Latvia, Norway, Russia, Slovenia, Ukraine Outcome: –Established 7 mailing lists on specific topics –The required functions are well covered by NGIs –The person power matches approximately the blueprint needs 22-Oct J. Knobloch/CERN: European Grid Initiative – EGI

Responses 22-Oct J. Knobloch/CERN: European Grid Initiative – EGI

NGI vs Task 22-Oct J. Knobloch/CERN: European Grid Initiative – EGI SLAs GOCDB Accounting Dashboard GOCDB Accounting Dashboard Operations Tools Dvt. Application Support Team WLCG Relationship

EGEE Communities 22-Oct >270 VOs from several scientific domains: Astronomy & Astrophysics Civil Protection Computational Chemistry Comp. Fluid Dynamics Computer Science/Tools Condensed Matter Physics Earth Sciences Fusion High Energy Physics Life Sciences >270 VOs from several scientific domains: Astronomy & Astrophysics Civil Protection Computational Chemistry Comp. Fluid Dynamics Computer Science/Tools Condensed Matter Physics Earth Sciences Fusion High Energy Physics Life Sciences J. Knobloch/CERN: European Grid Initiative – EGI CPU usage increased *10

The second year of EGEE III Close collaboration EGEE III ↔ EGI_DS EGEE III second year: –Management – strong links with project proposal preparation –Dissemination and training – prepare EGEE community for EGI and move activities to NGIs –Application community – move towards structures defined in EGI blueprint; prepare the transition from scientific discipline clusters (SDCs) to Specific Support Centres (SSCs) –Operations – prepare move from central to national operations –Software certification – forming integrated product teams 22-Oct J. Knobloch/CERN: European Grid Initiative – EGI

EGI proposals being developed EGI-InSPIRE (Integrated Sustainable Pan-European Infrastructure for REsearchers) –Transition to a sustainable federated European e-infrastructure built from national resources –Deliver production quality services to all its user communities and users –Provide a scalable user support model for international research communities within Europe and their worldwide collaborators –Interoperation and interoperability with other e-infrastructures ROSCOE (Robust Scientific Communities for EGI) –Provide SSC service for heavy grid users (HEP, Life Science, Computational Chemistry & Materials Science Technology, Grid Observatory, Photon Science, Complexity Science, and Humanities) SAFE (SSCs for Astronomy & Astrophysics, Fusion & Earth Science) –Astronomy & Astrophysics, Fusion & Earth Science EMI (European Middleware Initiative) –Expected to be the major middleware source for the Unified Middleware Distribution (UMD) serving EGI (and also PRACE) 22-Oct J. Knobloch/CERN: European Grid Initiative – EGI

Other related proposals Extra-European Collaboration –CHAIN –CHAIN (Co-ordination and Harmonisation of Advanced e-Infrastructures) to foster collaboration with the HPC community as well as with extra-European Grids e.g. in Africa, Asia and Latin America. Training, Dissemination, Business Outreach –CUE (Creating Users of E-infrastructures) enable users and research to access and use e-infrastructures Dissemination– GridTalk-II –bringing the success stories of Europe’s e-infrastructure to policy makers in government and business, to the broader scientific community and to the general public.... and there are more –Additional SSC proposals, OGF-Europe II, etc. 22-Oct J. Knobloch/CERN: European Grid Initiative – EGI

Relationships between projects 22-Oct J. Knobloch/CERN: European Grid Initiative – EGI Projects need to define their relationships by e.g. Service Level Agreement Service Level Description Memorandum of Understanding

Conclusion EGI_DS has defined the vision of a sustainable grid infrastructure in Europe, it developed an implementation plan for EGI, and it secured approval of the vision and plan by the European NGIs. The plan is now being implemented 22-Oct J. Knobloch/CERN: European Grid Initiative – EGI