European Grid Initiative Design Study Ludek Matyska CESNET, Czech Republic Member of the EGI_DS team

Terena, May 20, 2008www.eu-egi.org2 Current state Scientific communities started to use Grids as a base for new ways of collaboration Some are using Grids on a daily basis Also, industry is starting to become interested This creates a dependability on Grid infrastructure(s) However, all this means that a Long Term Perspective for Grids is needed

Terena, May 20, 2008www.eu-egi.org3 European Commission “…for Grids we would like to see the move towards long-term sustainable initiatives less dependent upon EU-funded project cycles” Viviane Reding, EU commissary, at the EGEE’06 conference, September 25 th, 2006

Terena, May 20, 2008www.eu-egi.org4 EGEE Infrastructure > 200 sites in 40 countries > CPUs > 13 PB storage > 200k jobs/day > 250 Virtual Organizations Countries participating in EGEE TERAGRI D OSG EELA Baltic Grid See-Grid DEISA EUMedGrid EUChinaGrid EUIndiaGrid NAREGI

Terena, May 20, 2008www.eu-egi.org5 Added value of Sharing Applications have very different requirements and may be broadly classified as –Provisioned Large scale, long term “Grand Challenge” –Scheduled Require large resources for short periods –Opportunistic No real-time nor mission critical All can coexist on the same infrastructure 5

Terena, May 20, 2008www.eu-egi.org6 European Grid Initiative Define and find ways to create sustainable European e-Infrastructure Coordinate integration and interaction of National Grid Initiatives (NGI) Define conditions and organizational basis for the European (trans-national) level of production Grid infrastructure suitable for and shared by very large set of scientific disciplines (connecting National Grid Infrastructures)

Terena, May 20, 2008www.eu-egi.org7 Evolution Testbeds Utility Service Routine Usage National Global European e-Infrastructure

Terena, May 20, 2008www.eu-egi.org8 38 European NGIs + Asia, US, Latin America + PRACE + OGF-Europe + …

Terena, May 20, 2008www.eu-egi.org9 EGI Design Study Project proposal: submitted to FP7-INFRASTRUCTURES , Design Studies Goal: Conceptual setup and operation of a new organizational model of a sustainable pan-European grid infrastructure Consortium: 9 Partners  EGI Preparation Team NGI (National) Representatives  EGI Advisory Board Person months: ~300 Duration: 1 Sept 2007 – 30 Nov 2009 (27 Months)

Terena, May 20, 2008www.eu-egi.org10 EGI Preparation Team Members: Johannes Kepler Universität Linz (GUP) Greek Research and Technology Network S.A. (GRNET) Istituto Nazionale di Fisica Nucleare (INFN) CSC – Scientific Computing Ltd. (CSC) CESNET, z.s.p.o. (CESNET) European Organization for Nuclear Research (CERN) Verein zur Förderung eines Deutschen Forschungsnetzes – DFN-Verein (DFN) Science & Technology Facilities Council (STFC) Centre National de la Recherche Scientifique(CNRS)

Terena, May 20, 2008www.eu-egi.org11 EGI the Acronym EGI “the organization” is one of the planned results of EGI_DS EGI is expected to take over the EU Grid activities (like EGEE, DEISA etc.), coordinate the national Grid activities and operate the Sustainable European Grid Infrastructure We will use term EGI to represent both the EGI organization and NGIs together

Terena, May 20, 2008www.eu-egi.org12 Characteristics of NGIs Each NGI … should be a recognized national body with a single point-of-contact … should mobilise national funding and resources … should operate the national e-Infrastructure … should support user communities (application independent, and open to new user communities and resource providers) … should contribute and adhere to international standards and policies Responsibilities between NGIs and EGI are split to be federated and complementary

Terena, May 20, 2008www.eu-egi.org13 EGI_DS Work Distribution WP2: EGI Requirements Consolidation (Fotis Karayannis, GRNET) WP3: EGI functionality definition (Laura Perini, INFN) WP4: Study of EGI legal and organisational options (Beatrice Merlin, CNRS) WP5: Establishment of EGI (Jürgen Knobloch, CERN) WP6: EGI Promotion and Links with Other Initiatives (Per Öster, CSC)

Terena, May 20, 2008www.eu-egi.org14 EGI DS Chronology February 26-27, 2007: EGI Workshop Munich May 2, 2007: Proposal submitted to the EC within FP7-INFRA , Design Studies Sept. 1, 2007: Project start Oct. 2, 2007:  EGI Workshop, Budapest, Hungary March 13/14, 2008:  2 nd EGI Workshop, Rome, Italy

Terena, May 20, 2008www.eu-egi.org15 EGI Workshop Budapest Presentation of the EGI_DS project to all NGI representatives Requirements Analysis and Uses Cases  Summary Budapest EGI Workshop First information on Functional Definition Convention and Legal Aspects

Terena, May 20, 2008www.eu-egi.org16 EGI Workshop Budapest

Terena, May 20, 2008www.eu-egi.org17 EGI Workshop Rome Practically all NGIs represented –Plus up to 2 experts per NGI Presentation of different EGI aspects: –Grid Operations –User Oriented Functions (Application support) –Middleware –Management of the EGI organization –Legal Structures All were drafts for discussion

Terena, May 20, 2008www.eu-egi.org18 EGI Workshop Rome

Terena, May 20, 2008www.eu-egi.org19 Grid Operations Key assumptions Continuity requirement: –As some large communities are using Grids already in a production way, the transition to EGI must be non-disruptive Functionality requirement: –The key functionality must not change because of the transitions

Terena, May 20, 2008www.eu-egi.org20 What is EGI Operations? To answer this question, we need a much better idea of what “the EGI Grid” will be… Is it: A large-scale, production Grid infrastructure – build on National Grids that interoperate seamlessly at many levels, offering reliable and predictable services to a wide range of applications, ranging from “mission critical” to prototyping and research? A loosely coupled federation of NGIs with little or no cross-grid activity, heterogeneous and sometimes incompatible middleware stacks, no cross-grid accounting, no need for coordinated operations or management 20

Terena, May 20, 2008www.eu-egi.org21 What is EGI Operations? The NGI representatives agreed on the following: Focus on:  A large-scale, production Grid infrastructure – build on National Grids that interoperate seamlessly at many levels, offering reliable and predictable services to a wide range of applications, ranging from “mission critical” to prototyping and research A loosely coupled federation of NGIs with little or no cross-grid activity, heterogeneous and sometimes incompatible middleware stacks, no cross-grid accounting, no need for coordinated operations or management 21

Terena, May 20, 2008www.eu-egi.org22 What is EGI Operations? However, most behave like A loosely coupled federation of NGIs with little or no cross-grid activity, heterogeneous and sometimes incompatible middleware stacks, no cross-grid accounting, no need for coordinated operations or management The creates serious problems when trying to define the roadmap (the Blueprint in the EGI_DS understanding) Is the operation managed or coordinated? Is there a common middleware or everybody is free to use what they see fit? What does “interoperability” means precisely? 22

Terena, May 20, 2008www.eu-egi.org23 How is Operation to be achieved Multi-level Operation Model Definition of set of services that must be operated on a coherent way Federated approach, delegation of responsibilities to NGIs Support for multiple middleware systems –However, see the middleware discussion The EGI core team will be primary responsible for planning and coordination

Terena, May 20, 2008www.eu-egi.org24 User Oriented Functions Application Support –Based on the support centres and activities at national level –Coordinate to increase synergies –Reduce the users’ cost to use the Grid Training –Sharing t-Infrastructure(s), materials and experience –Cross border synergies Dissemination –PR and support for broad scientific publishing

Terena, May 20, 2008www.eu-egi.org25 Management of the EGI organization Light-weight schema Focused mainly on coordination and planning –Actual services outsourced to NGIs However, responsibility for smooth operation at the European level –Cost models and money flow –Contribution (fees), Service charges, collocated development grants

Terena, May 20, 2008www.eu-egi.org26 Management Structure of the EGI organisation Director + staff EGI Council CTO (Develop ments) COO (Operatio ns) Political Bodies such as: eIRG, EU, ESFRI, Ntl. Bodies,... Advisory Committees EGI organisation Dev. Group Oper. Group Adm.+PR Group Strategy Cttee CAO (Admin.+ PR) EGI

Terena, May 20, 2008www.eu-egi.org27 EGI Management and its Environment Political environment: eIRG, ESFRI, EU... Membership: NGIs, ass. members Unit 1: EGI Operations Unit 2: EGI Developments Other Ext. Relations: Media,... External Relations Internal Relations EGI Organisation Management: Director + staff + Heads of Units (CTO, COO, CAO) EGI Council: Governing body of EGI Unit 3: EGI Administration

Terena, May 20, 2008www.eu-egi.org28 Legal Structures The basic requirements — Autonomous legal entity — Fastness of creation — Not for profit organisation but ability to provide services to third parties — Open to public and private NGI organisations.. — …residing in any European country — Limited liability Either national or international entity — European Reserach Infrastructure (ERI) To be adopted by EU Council in December 2008 Tender for location

Terena, May 20, 2008www.eu-egi.org29 Middleware No Grid without a middleware –And the middleware must be developed – Cost! The proposal: Not forcing one middleware –However, a clear road towards a convergence of functions and services necessary –And must be strongly driven by EGI Proposed support for three stacks: –gLite (EGEE), UNICORE (DEISA), ARC (NorduGRID) –Not accepted as proposed by NGIs Open questions: –Who will develop the middleware? And who will pay? –What and where to converge and what could be left independent? –Interaction between middleware development and EGI –Do we really want one common shared middleware?

Terena, May 20, 2008www.eu-egi.org30 EGI Functionality Overview Management, Outreach & Dissemination - Representation of EU Grid Efforts Operations & Resource Provisioning & Security Application Support & Training Middleware (Build&Test, Component Selection/Validation/Deployment) Standardisation & Policies Industry take-up

Terena, May 20, 2008www.eu-egi.org31 Definition of EGI Organisation Initial functions and services provided by EGI Estimation of resource requirements for executing the functions Relationships with NGIs and global communities and resource centres Description of functions and scope of NGIs Transition process to EGI model

Terena, May 20, 2008www.eu-egi.org32 Resource Estimation FTEs needed to carry out each function Distinction: core functions, middleware functions Workload distribution: EGI Organisation, NGIs First draft proposal: –Core functions: 82 FTEs –Middleware development: 155 FTEs

Terena, May 20, 2008www.eu-egi.org33 EGI_DS Schedule Duration 27 months: Develop EGI ProposalNGIs signing Proposal Start of EGEE-III Final Draft of EGI Blueprint Proposal EGI Blueprint Proposal EGEE-III transition to EGI-like structure EGI Entity in place EU Call Deadline for EGI Proposal Submission of EGEE-III Start of EGI Design Study EGEE-II (2YEARS)‏ EGEE-III (2YEARS)‏ EGI operational

Terena, May 20, 2008www.eu-egi.org34 Upcoming Event June 30-July 1, 2008:  EGI Workshop, Geneva, Switzerland “Draft Papers on the EGI Structure” To present a coherent proposal to –EGI organization management and legal structure –Dealing with middleware –Clear roadmap both for EGI constitution adoption Transition of contemporary Grid infrastructures into European Grid Infrastructure

Terena, May 20, 2008www.eu-egi.org35 EGI – European Grid Initiative The EGI Organization is proposed to become a “Glue” between various grid communities in Europe and beyond EGI_DS works on definition of required mechanisms and functionalities of the EGI Organisation EU NGIs (or NGI forming teams) expressed strong support to this idea –However, there is still discrepancy between declaration of support and real behavior  EGI_DS is trying to define ways towards a single shared sustainable environment for the scientific communities for their everyday work  But is it the future science wants?

Terena, May 20, 2008www.eu-egi.org36