1. The European DataGrid Project Team http://www.eu-datagrid.org/ The EU DataGrid

2. EU DataGrid: Introduction 2 Tutorial Roadmap u Introduction u The Testbed u Middleware Issues n Job Submission n Data Management n Information and Monitoring u Applications n HEP n EO n Biology

3. The EU DataGrid: Introduction The European DataGrid Project Team http://www.eu-datagrid.org/

4. EU DataGrid: Introduction 4 Contents u European DataGrid (EDG) Project scope u EDG structure u Major components of EDG Middleware u DataGrid in Numbers u Relation to Sister Projects

5. EU DataGrid: Introduction 5 The Grid Vision u Flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resource n From “The Anatomy of the Grid: Enabling Scalable Virtual Organizations” u Enable communities (“virtual organizations”) to share geographically distributed resources as they pursue common goals -- assuming the absence of… n central location, n central control, n omniscience, n existing trust relationships.

6. EU DataGrid: Introduction 6 Grids: Elements of the Problem u Resource sharing n Computers n Storage u Security n Resource sharing always conditional n issues of trust, policy, payment, … u Coordinated problem solving n Beyond client-server n Distributed data analysis u Virtual Organisations n Community overlays on classic organisational structures n Large or small, static or dynamic

7. EU DataGrid: Introduction 7 EU DataGrid u DataGrid funded by European Union whose objective to exploit and build the next generation computing infrastructure providing intensive computation and analysis of shared large-scale databases. u Enable data intensive sciences by providing world wide Grid test beds to large distributed scientific organizations ( “Virtual Organizations, VOs”) u Duration: Jan 1, 2001 - Dec 31, 2003 u Applications/End Users Communities: HEP, Earth Observation, Biology u Specific Project Objectives: n Middleware for fabric & grid management n Large scale testbed n Collaborate and coordinate with other projects n Contribute to Open Standards and international bodies ( GGF, Industry&Research forum)

8. EU DataGrid: Introduction 8 DataGrid Main Partners u CERN – International (Switzerland/France) u CNRS - France u ESA/ESRIN – International (Italy) u INFN - Italy u NIKHEF – The Netherlands u PPARC - UK

9. EU DataGrid: Introduction 9 Research and Academic Institutes CESNET (Czech Republic) Commissariat à l'énergie atomique (CEA) – France Computer and Automation Research Institute, Hungarian Academy of Sciences (MTA SZTAKI) Consiglio Nazionale delle Ricerche (Italy) Helsinki Institute of Physics – Finland Institut de Fisica d'Altes Energies (IFAE) - Spain Istituto Trentino di Cultura (IRST) – Italy Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany Royal Netherlands Meteorological Institute (KNMI) Ruprecht-Karls-Universität Heidelberg - Germany Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands Swedish Research Council - Sweden Assistant Partners Industrial Partners Datamat (Italy) IBM-UK (UK) CS-SI (France)

10. EU DataGrid: Introduction 10 Project Schedule u Project started on 1/Jan/2001 u Series of deployments to the “application testbed” n Early 2001 s Globus 1 only - no EDG middleware s To understand some deployment issues n Early 2002 s First release of EU DataGrid software to tolerant users within the project n Subsequent releases s Adding functionality s Concentrating on achieving production quality s This is where we are today… n Final release late 2003 s Not much added functionality u Project stops on 31/Dec/2003

11. EU DataGrid: Introduction 11 DataGrid Work Packages u The EDG collaboration is structured in 12 Work Packages n WP1: Work Load Management System n WP2: Data Management n WP3: Grid Information and Monitoring n WP4: Fabric Management n WP5: Storage Element / Storage Resource Manager n WP6: Testbed and demonstrators – Production quality International Infrastructure n WP7: Network Monitoring n WP8: High Energy Physics Applications n WP9: Earth Observation n WP10: Biology n WP11: Dissemination n WP12: Management

12. EU DataGrid: Introduction 12 Bodies u Project Management Board (PMB) n Look after the politics u Project Technical Board (PTB) n Meets infrequently to approve deliverables n Work Package Managers meet weekly u Architecture Task Force (ATF) n Define interfaces u Quality Assurance Group (QAG) n Define relevant standards u Integration Team (ITeam) n Glue it all together

13. EU DataGrid: Introduction 13 EDG Interfaces Computing Elements SystemManagers Scientists OperatingSystems File Systems StorageElements Mass Storage Systems HPSS, Castor User Accounts Certificate Authorities ApplicationDevelopers Batch Systems PBS, LSF u Next slides show major components of middleware developed by WP1-5 and 7

14. EU DataGrid: Introduction 14 The EDG WMS u The user interacts with Grid via a Workload Management System (WMS) u The Goal of WMS is the distributed scheduling and resource management in a Grid environment. u What does it allow Grid users to do? n To submit their jobs n To execute them on the “best resources” s The WMS tries to optimize the usage of resources n To get information about their status n To retrieve their output

15. EU DataGrid: Introduction 15 Computing Element u Is a Grid Job Queue n Publishes information about itself n Checks the job is permitted n Sends it to an an appropriate internal queue

16. EU DataGrid: Introduction 16 SRM: Storage Resource Manager u SRM subset implementation n A defacto international standard for Storage Resource Management u Web service uses Java AXIS and EDG security u Supports multiple VOs u Functions n Writing a file n Reading a file

17. EU DataGrid: Introduction 17 Replica Manager u Hides the SRM u Coordinates use of n Replica Location Service n Replica Metadata Catalog n Replica Optimization Service

18. EU DataGrid: Introduction 18 R-GMA: Information & Monitoring u Relational implementation of GMA from GGF u Makes use of GLUE schema u Interoperable with MDS u Deals with information on n The Grid itself s Resources and Services (for which the Globus MDS is a common solution) s Job status information n Grid applications s This is information published by user jobs.

19. EU DataGrid: Introduction 19 WP6: TestBed Integration u Exact definition of RPM lists (components) for the various testbed machine profiles (CE, RB, UI,, WN, IC etc.) – check dependencies u Perform preliminary centrally (CERN) managed tests on EDG m/w before green light for spread EDG testbed sites deployment u Provide, update end user documentation for installers/site managers, developers and end users u Define EDG release policies, coordinate the integration team staff with the various WorkPackage managers – keep high inter-coordination. u Set up the Authorization Working Group to manage authorization policies on the testbed

20. EU DataGrid: Introduction 20 Grid aspects covered by EDG VOMS Provides certificate with VOs, groups and roles RGMA: Information & Monitoring Provides info on resource utilization & performance User Interface Submit & monitor jobs, retrieve output Grid Fabric Management Configure, installs & maintains grid sw packages and environ. Workload Management System Manages submission of jobs to Res. Broker, obtains information and retrieves output Network performance Provides efficient network transport, bandwidth monitoring Computing Element Gatekeeper to a grid computing resource Testbed admin. Certificate auth.,user reg., usage policy etc. Storage Resource Manager Grid-aware storage area Applications HEP, EO, Biology Replica Manager Replicates and locates data

21. EU DataGrid: Introduction 21 Applications (WP8-10) High Energy Physics Biomedical Applications Earth Observation Science Applications

22. EU DataGrid: Introduction 22 Software 50 use cases 18 software releases >300K lines of code 728454 as measured by SLOCCount on 25 June People >350 registered users 12 Virtual Organisations 16 Certificate Authorities >200 people trained 278 man-years of effort 100 years funded Testbeds >15 regular sites >10’000s jobs submitted >1000 CPUs >5 TeraBytes disk 3 Mass Storage Systems Scientific applications 5 Earth Obs institutes 9 bio-informatics apps 6 HEP experiments DataGrid in Numbers (Out of date)

23. EU DataGrid: Introduction 23 Related Grid Projects Through links with sister projects, there is the potential for a truly global scientific applications grid Demonstrated at IST2002 and SC2002 in November