CERN The European DataGrid Project Technical status Bob Jones (CERN) Deputy Project Leader

B.Jones– Nov n° 2 DataGrid scientific applications Developing grid middleware to enable large-scale usage by scientific applications Earth Observation about 100 Gbytes of data per day (ERS 1/2) 500 Gbytes, for the ENVISAT mission Bio-informatics n Data mining on genomic databases (exponential growth) n Indexing of medical databases (Tb/hospital/year) Particle Physics  Simulate and reconstruct complex physics phenomena millions of times  LHC experiments will generate 6-8 PetaBytes/year

B.Jones– Nov n° 3 The Project u 9.8 M Euros EU funding over 3 years u 90% for middleware and applications (HEP, Earth Obs. and Bio Med.) u Three year phased developments & demos ( ) u Total of 21 partners n Research and Academic institutes as well as industrial companies u Extensions (time and funds) on the basis of first successful results: n DataTAG ( ) n CrossGrid ( ) n GridStart ( )

B.Jones– Nov n° 4 Research and Academic Institutes CESNET (Czech Republic) Commissariat à l'énergie atomique (CEA) – France Computer and Automation Research Institute, Hungarian Academy of Sciences (MTA SZTAKI) Consiglio Nazionale delle Ricerche (Italy) Helsinki Institute of Physics – Finland Institut de Fisica d'Altes Energies (IFAE) - Spain Istituto Trentino di Cultura (IRST) – Italy Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany Royal Netherlands Meteorological Institute (KNMI) Ruprecht-Karls-Universität Heidelberg - Germany Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands Swedish Research Council - Sweden Assistant Partners Industrial Partners Datamat (Italy) IBM-UK (UK) CS-SI (France)

B.Jones– Nov n° 5 EDG structure : work packages  The EDG collaboration is structured in 12 Work Packages: n WP1: Work Load Management System n WP2: Data Management n WP3: Grid Monitoring / Grid Information Systems n WP4: Fabric Management n WP5: Storage Element n WP6: Testbed and demonstrators n WP7: Network Monitoring n WP8: High Energy Physics Applications n WP9: Earth Observation n WP10: Biology n WP11: Dissemination n WP12: Management } } Applications

B.Jones– Nov n° 6 Project Schedule u Project started on 1/1/2001 u TestBed 0 (early 2001) n International test bed 0 infrastructure deployed s Globus 1 only - no EDG middleware u TestBed 1 ( now ) n First release of EU DataGrid software to defined users within the project: s HEP experiments, Earth Observation, Biomedical applications u Project successfully reviewed by EU on March 1 st 2002 u TestBed 2 (end 2002) n Builds on TestBed 1 to extend facilities of DataGrid u TestBed 3 (2 nd half 2003) u Project completion expected by end 2003

B.Jones– Nov n° 7 EDG middleware GRID architecture Collective Services Information & Monitoring Replica Manager Grid Scheduler Local Application Local Database Underlying Grid Services Computing Element Services Authorization Authentication and Accounting Replica Catalog Storage Element Services SQL Database Services Fabric services Configuration Management Configuration Management Node Installation & Management Node Installation & Management Monitoring and Fault Tolerance Monitoring and Fault Tolerance Resource Management Fabric Storage Management Fabric Storage Management Grid Fabric Local Computing Grid Grid Application Layer Data Management Job Management Metadata Management Service Index APPLICATIONS GLOBUS M / W

B.Jones– Nov n° 8 EDG interfaces Collective Services Information & Monitoring Replica Manager Grid Scheduler Local Application Local Database Underlying Grid Services Computing Element Services Authorization Authentication and Accounting Replica Catalog Storage Element Services SQL Database Services Fabric services Configuration Management Configuration Management Node Installation & Management Node Installation & Management Monitoring and Fault Tolerance Monitoring and Fault Tolerance Resource Management Fabric Storage Management Fabric Storage Management Grid Application Layer Data Management Job Management Metadata Management Object to File Mapping Service Index Computing Elements SystemManagers Scientists OperatingSystems File Systems StorageElements Mass Storage Systems HPSS, Castor User Accounts Certificate Authorities ApplicationDevelopers Batch Systems PBS, LSF, etc.

B.Jones– Nov n° 9 EDG overview : current project status  EDG currently provides a set of middleware services  Job & Data Management  GRID & Network monitoring  Security, Authentication & Authorization tools  Fabric Management  Runs on Linux Red Hat 6.1 platform  Site install & config tools and set of common services available  5 core EDG 1.2 sites currently belonging to the EDG-Testbed  CERN(CH), RAL(UK), NIKHEF(NL), CNAF(I), CC-Lyon(F),  also deployed on other testbed sites (~15) via CrossGrid, DataTAG and national grid projects  actively used by application groups  Intense middleware development continuously going on, concerning: n New features for job partitioning and check-pointing, billing and accounting n New tools for Data Management and Information Systems. n Integration of network monitoring information inside the brokering polices

B.Jones– Nov n° 10 Dubna Moscow RAL Lund Lisboa Santander Madrid Valencia Barcelona Paris Berlin Lyon Grenoble Marseille Brno Prague Torino Milano BO-CNAF PD-LNL Pisa Roma Catania ESRIN CERN IPSL Estec KNMI Testbed Sites

B.Jones– Nov n° 11 Tutorials DAY1 u Introduction to Grid computing and overview of the DataGrid project u Security u Testbed overview u Job Submission lunch u hands-on exercises: job submission DAY2 u Data Management u LCFG, fabric mgmt & sw distribution & installation u Applications and Use cases u Future Directions lunch u hands-on exercises: data mgmt The tutorials are aimed at users wishing to "gridify" their applications using EDG software and are organized over 2 full consecutive days December 2 & 3 – Edinburgh 5 & 6 - Turin 9 & 10 - NIKHEF To date about 120 people trained

B.Jones– Nov n° 12 Through links with sister projects, there is the potential for a truely global scientific applications grid Demonstrated at IST2002 and SC2002 in November Related Grid projects

B.Jones– Nov n° 13 Details of release 1.3 u Based on Globus 2.0beta but with binary modifications taken from Globus 2.2 n large file transfers (gridFTP) n “lost” jobs (GASS cache) n unstable information system (MDS 2.2) n new Replica Catalog schema u More reliable job submission n Res Broker returns errors if overloaded n Stability tests successfully passed n Minor extensions to JDL u Improved data management tools n GDMP v3.2 s automatic &explicit triggering of staging to MSS s support for parallel streams (configurable) n Edg-replica-manager v2.0 s uses GDMP for MSS staging s shorter alias’ for commands (e.g. edg-rm-l edg-replica-manager-listReplicas) s new file mgmt commands: getbestFile, cd, ls, cat etc. s support for parallel streams (configurable) u Better fabric mgmt s Bad RPMs no longer block installation Available on Linux RH 6. Not backward compatible with EDG 1.2 Addresses bugs found by applications in EDG being deployed in November

B.Jones– Nov n° 14 Incremental Steps for testbed 2 1. Fix “show-stoppers” for application groups – mware WPs (continuous) 2. Build EDG1.2.x with autobuild tools 3. Improved (automatic) release testing 4. Automatic installation & configuration procedure for pre-defined site 5. Start autobuild server for RH 7.2 and attempt build of release 1.2.x 6. Updated fabric mgmt tools 7. Introduce prototypes in parallel to existing modules u RLS u R-GMA 8. GLUE modified info providers/ consumers 9. Storage Element v Introduce Reptor 11. Add NetworkCost Function 12. GridFTP server access to CASTOR 13. Introduce VOMS 14. Improved Res. Broker 15. LCFGng for RH Storage Element v Integrate mapcentre and R-GMA 18. Storage Element V3.0 Expect this list to be updated regularly Prioritized list of improvements to be made to the current release as established with users from September through to end of 2002

B.Jones– Nov n° 15 Plans for the future u Further developments in 2003 n Further iterative improvements to middleware driven by users needs n More extensive testbeds providing more computing resources n Prepare EDG software for future migration to Open Grid Services Architecture u Interaction with LHC Computing grid (LCG) n LCG intends to make use of the DataGRID middleware n LCG is contributing to DataGRID s Testbed support and infrastructure n Get access to more computing resources in HEP computing centres s Testing and verification n Reinforce the testing group and maintain a certification testbed s Fabric management and middleware development u New EU project n Make plans to preserve current major asset of the project: probably the largest Grid development team in the world s EoI for FP6 ( ), possible extension of the project, etc.