1. EU DataGrid progress and EGEE plans Fabrizio Gagliardi EDG Project Leader - EGEE Proposal Coordinator Fabrizio.Gagliardi@cern.ch

2. 14 May 2003EDG6 Barcelona2 The EU DataGrid Project 9.8 M Euros EU funding over 3 years, twice as much from partners 90% for middleware and applications (HEP, Earth Obs. and Bio Med.) Three year phased developments & demos (2001-2003) 2 nd annual project review successfully passed in Feb 2003! Total of 21 partners Research and Academic institutes as well as industrial companies Related projects and activities: DataTAG (2002-2003) CrossGrid (2002-2004) GRIDSTART (2002-2004) Grace (2002-2004)

3. 14 May 2003EDG6 Barcelona3 Genomic Exploration Earth Observation High Energy Physics Applications more and more scientists begin to use the EDG middleware and testbed, relying on Grid technology to solve huge data challenges

4. 14 May 2003EDG6 Barcelona4 Project priorities refocused at EDG5 Quality Policy Statement published http://eu-datagrid.web.cern.ch/eu-datagrid/WP12/default.htm List of priorities defined at a project retreat http://documents.cern.ch/age?a021130 Followed-up at the Budapest project conference http://www.tomiexpress.hu/datagrid/ Show-stoppers found by users on the application testbed were the highest priority Incremental improvements driven by the needs of the applications (HEPCAL, Bio and EO use cases) After initial middleware development and testbed deployment, effort has been refocused on quality and stability

5. 14 May 2003EDG6 Barcelona5 Atlas & CMS Evaluations RESULTS Atlas software was used in the EDG Grid environment Several hundred simulation jobs of length 4-24 hours were executed, data was replicated using grid tools Results of simulation agreed with ‘non-Grid’ runs OBSERVATIONS Good interaction with EDG middleware providers and with WP6/8 With a substantial effort it was possible to perform the jobs Showed up bugs and performance limitations (fixed or to be fixed in EDG 2.0) We need EDG 2.0 release for use in large scale data challenges RESULTS Could distribute and run CMS s/w in EDG environment Generated ~250K events for physics with ~10,000 jobs in 3 week period OBSERVATIONS Were able to quickly add new sites to provide extra resources Fast turnaround in bug fixing and installing new software Test was labour intensive (since software was developing and the overall system was fragile) EDG 2.0 should fix the major problems providing a system suitable for full integration in distributed production

6. 14 May 2003EDG6 Barcelona6 EDG Application Testbed for the EU review Since Last Year: Improved software (EDG 1.4.3). Doubled sites. More waiting…  Australia, Taiwan, USA (U. Wisc.), UK Sites, INFN, French sites, CrossGrid, … Significantly more CPU/Storage. Hidden Infrastructure MDS Hierarchy, Resource Brokers, User Interfaces, VO Replica Catalogs, VO Membership Servers, Certificate Authorities SiteCountryCPUsStorage CC-IN2P3*FR620192 GB CERN*CH1381321 GB CNAF*IT481300 GB Ecole Poly.FR6220 GB Imperial Coll.UK92450 GB LiverpoolUK210 GB ManchesterUK915 GB NIKHEF*NL142433 GB OxfordUK130 GB PadovaIT11666 GB RAL*UK6332 GB SARANL010000+ GB TOTAL5107514969 GB *also Dev. TB; +200 TB including tape

7. 14 May 2003EDG6 Barcelona7 Connections of the different nodes of the EDG testbed are made possible by the EU- funded GEANT project  connecting more than 30 countries across Europe  speeds of up to 10 Gbit/s  high data throughput  Quality of Service EDG and GEANT: the first major production quality tests of the network EDG & GEANT  speed  reliability  monitoring capabilities

8. 14 May 2003EDG6 Barcelona8 Through links with sister projects, there is the potential for a truely global scientific applications grid Related Grid Projects

9. 14 May 2003EDG6 Barcelona9 Second project review feedback Feb ‘03 Congratulations for a good review. Good presentations and no "Murphy's law“ for the demos. An impressive job. This success reflects the interest of all the partners involved. Congratulates the project management for taking the risk of concentrating on production quality. Would like to see the promise fulfilled of no relevant loss of functionality by the end of the project.

10. 14 May 2003EDG6 Barcelona10 Second EU review successfully passed, needs to finish well Major re-orientation of the project towards production accepted Need to develop further plans in view of future project EGEE Continue collaboration with US and across Atlantic testbeds (with DataTAG) Need to accommodate other applications Plan long term support of EDG developments Major proposal for further EU funding (EGEE) Current Plans

11. 14 May 2003EDG6 Barcelona11 EGEE vision: Enabling Grids for E-science in Europe Goal Create a wide European Grid production quality infrastructure on top of present and future EU RN infrastructure Build on EU and EU member states major investments in Grid Technology International connections (US and AP) Several pioneering prototype results Larg Grid development team (>60 people) Requires major EU funding effort Approach Leverage current and planned national and regional Grid programmes (e.g. LCG) Work closely with relevant industrial Grid developers, NRENs and US-AP projects EGEE Applications Geant network

12. 14 May 2003EDG6 Barcelona12 Why EGEE? The Historical Analogy Prior to the EU Geant program,there was in Europe a multitude of exploratory projects in networking technology. Geant was truly production oriented, and brought European telecom operators actively into the picture In a similar way, EGEE can ensure preservation of current investments in European Grid R&D, extending the present infrastructure and focussing all activities towards establishing a production quality Grid

13. 14 May 2003EDG6 Barcelona13 Why EGEE? The Societal Impact Access to a production quality GRID will change the way science and much else is done in Europe A geneticist at a conference, inspired by a talk she hears, will be able to launch a complex biomolecular simulation from her mobile phone. A team of engineering students will be able to run the latest 3D rendering programs from their laptops using the Grid. An international network of scientist will be able to model a new flood of the Danube in real time, using meteorological and geological data from several centers across Europe.

14. 14 May 2003EDG6 Barcelona14 Why EGEE? The Political Context Current Grid R&D projects run out within 18 months The EGEE partners have already made major progress in aligning national and regional Grid R&D efforts, in preparation for EGEE Launching EGEE now will preserve the current strong momentum of the European Grid community, and the enthusiasm of the hundreds of young European researchers already involved in EU Grid projects (>150 in EDG only)

15. 14 May 2003EDG6 Barcelona15 EGEE Approach Most of partners built national and regional Grid federations to participate in EGEE Condition to participate in EGEE is to have already an established Grid activity or be an established Grid technology centre EGEE overall project funding needs will require staggered proposals to respond to several separate EU calls EU FP6 Communication Network Development Call opened on December 17 th, proposal submitted on May 6 th

16. 14 May 2003EDG6 Barcelona16 The EGEE consortium of Grid Federations Total of 70 full partners covering entire EU and beyond

17. 14 May 2003EDG6 Barcelona17 Integrated Infrastructure Initiative (I3) Response to FP6 call “Communication Network Development – Grids” Three lines of EU funding (with current funding breakdown): Networking Activities:  Management & coordination (7% of total funding)  Dissemination and outreach (7% of total funding)  Application support (10% of total funding) Specific Service Activities:  Provision and procurement of Grid services (52% of total funding) Joint Research Activity  Engineering development to improve the services provided by the Grid infrastructure (24% of total funding)

18. 14 May 2003EDG6 Barcelona18 Networking activities General management and coordination of the project User groups, Industry Forum Dissemination, outreach, training and pilot applications

19. 14 May 2003EDG6 Barcelona19 Networking activities User Training and Induction; User Support and Consultancy; Applications Interface. The lead partner for the Training activity is the UK (Edinburgh) Dissemination and Outreach; The lead partner for the Dissemination activity is the European Research Network organisation Terena (Amsterdam) Both lead partners will rely on a supporting network in the partner regions, partially funded by the project. The User Support and Consultancy is envisaged as a distributed effort

20. 14 May 2003EDG6 Barcelona20 Networking activities The Applications Interface Activity has three components: two Pilot Application Centres: high energy physics (LCG: www.cern.ch/lcg ) and bio/medical www.cern.ch/lcg one more generic component dealing with the longer term recruitment and support of other communities The Pilot Application Centres are based at CERN and in France, with participation to their activities by other partners envisaged The third, more generic component is distributed more evenly among the partners, in order to ensure as broad a contact with scientific communities as possible

21. 14 May 2003EDG6 Barcelona21 Specific service activities Integration of national and international Grid infrastructures

22. 14 May 2003EDG6 Barcelona22 Specific service activities The structure of the Grid services will comprise: EGEE Operations Management at CERN EGEE Core Infrastructure Centres in the UK, France, Italy, Germany and CERN (leveraging LCG at the start), responsible for managing the overall Grid infrastructure Regional Operations Centres, responsible for coordinating regional resources, regional deployment and support of services in all other countries

23. 14 May 2003EDG6 Barcelona23 Joint research activity Hardening and re-engineering of Middleware

24. 14 May 2003EDG6 Barcelona24 Joint research activity Middleware Centres for key services: Resource Access (Italy); Data Management (CERN); Information Collection and Accounting (UK); Resource Brokering (Italy); Closely connected to this middleware development is a Quality Assurance team (France) and a Grid Security team (Northern Europe Consortium). A Middleware Integration team and Middleware Testing Centre will be located at CERN. Middleware services and related support teams require critical mass. With matching funding from the host institutions, this translates to over 40 FTEs for all middleware activities (compare with about 60 FTEs in EDG).

25. 14 May 2003EDG6 Barcelona25 Summary of budget requirements for EGEE EGEE will integrate existing computer resources, no hardware requests. The bulk of the budget is for human resources Building strong teams where already existing expertise, to progress as quickly as possible. Host partners for these teams will provide matching funding and demonstrate adequate expertise for the job The partners estimated a total funding need of about 100 M € corresponding to about 33 M € EU funding. Not included in this are US and Russian contributions (4 M €), and overall project management, corresponding to 7% of the EU funding over two years

26. 14 May 2003EDG6 Barcelona26 EGEE and Industry Industrial participation encouraged both as potential end-users and IT technology and service suppliers Normally through national and regional Grid EGEE federations EGEE will maintain an Industry Forum to keep selected Industrial and Commercial interested parties in close contact Services developed in first EGEE 2 years phase (2004- 5) might be tendered to Industry in second phase (2006-7)

27. 14 May 2003EDG6 Barcelona27 Conclusions The EU DataGrid project has successfully fulfilled its role of EU Grid flagship project in collaboration with several other EU and international projects Essential to keep the momentum and the current lead in production Grids in Europe Important to build an international cooperation between European and US/AP Grid infrastructure projects The scientific user communities are already international (HEP is an excellent example) and so the computing resources and most of the experimental instruments EGEE proposes the right framework and plans to accomplish the above objectives EGEE executive summary document linked to this conference agenda (http://agenda.cern.ch/fullAgenda.php?ida=a03816)