1. INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org The EGEE middleware and the GILDA t-Infrastructure Roberto Barbera University of Catania and INFN EGEE Tutorial Pisa, 23.10.2005

2. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 2 JRA1: Middleware Engineering and Integration JRA2: Quality Assurance JRA3: Security JRA4: Network Services Development SA1: Grid Operations, Support and Management SA2: Network Resource Provision NA1: Management NA2: Dissemination and Outreach NA3: User Training and Education NA4: Application Identification and Support NA5: Policy and International Cooperation 24% Joint Research28% Networking 48% Services Emphasis in EGEE is on operating a production Grid and on supporting the end-users. The EGEE Activities

3. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 3 Outline Generalities Overview of the gLite middleware The GILDA t-Infrastructure –services –tools –applications Summary and conclusions

4. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 4 Generalities

5. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 5 User Interface A multi-VO Grid User Interface Grid services EGEE is establishing a production grid service to support multiple, diverse VO’s

6. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 6 A multi-VO grid Users join VO Virtual organisation contributes resources & negotiates access Grid middleware runs on each resource –Data storage –(Usually) batch jobs on pools of processors Additional services (both people and middleware) enable the grid Effect: collaboration across administrative domains INTERNET

7. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 Introduction

8. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 8 VO’s and middleware The tools, services used by the VO’s applications Application development environment, portals, semantics, workflow In EGEE-1: Mainly VO-specific Basic Grid services: AA, job submission, info, … Middleware: “collective services” Application toolkits, standards Application Emphasis of EGEE -1 middleware

9. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 9 UI JDL Logging & Book-keeping ResourceBroker Job Submission ServiceStorageElementComputeElement InformationService Job Status ReplicaCatalogue DataSets info Author. &Authen. Job Submit Event Job Query Job Status Input “sandbox” Input “sandbox” + Broker Info Globus RSL Output “sandbox” Job Status Publish grid-proxy-init Expanded JDL SE & CE info A typical job workflow

10. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 10 Parts of the Grid “ecosystem”... LCG 2004 2001 EGEE Used in USA EU NextGrid DEISA GridCC Future grids EDG GlobusMyProxyCondor... VDT DataTAG CrossGrid... OSG, … SRM …

11. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 11 Overview of EGEE Middleware The gLite Grid services follow a Service Oriented Architecture –facilitate interoperability among Grid services –allow easier compliance with upcoming standards Architecture is not bound to specific implementations –services are expected to work together –services can be deployed and used independently The gLite service decomposition has been largely influenced by the work performed in the LCG project

12. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 12 EGEE middlewares face to face LCG (the present) Security –GSI Job Management –Condor + Globus –CE, WN –Logging & Bookkeeping Data Management –LCG services Information & Monitoring –BDII (evolution of MDS) Grid Access –CLI + API gLite (the future) Security –GSI and VOMS Job Management –Condor + Globus + blahp –CE, WN –Logging & Bookkeeping –Job Provenance –Package management Data Management –LFC –gLite-I/O + FiReMan Information & Monitoring –BDII –R-GMA + Service Discovery Grid Access –CLI + API + Web Services

13. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 13 gLite components overview Grid Access Service API Access Services Job Provenance Job Management Services Computing Element Workload Management Package Manager Metadata Catalog Data Services Storage Element Data Movement File & Replica Catalog Authorization Security Services Authentication Auditing Information & Monitoring Information & Monitoring Services Job Monitoring Site Proxy CLI Dynamic Connectivity Service Monitoring Service Discovery Accounting Near Future now

14. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 14 Job Management Services

15. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 15 DM Interaction Overview File and Replica Catalog StorageIndex Fireman Database WMS Storage Element SRM Storage gLite I/OgridFTP File Transfer and Placement Service FTS FPS Transfer Agent Database VOMS MyProxy Get credential Store credential File I/O File namespace and Metadata mgmt File replication Proxy renewalReplica Location WSDL API

16. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 16 gLite status gLite v1.3 released 05/08/2005 –File Placement Service, File Placement Service clients added to UI and WNs modules –new data transfer agents including architecture refactoring to allow proper inter-VO scheduling and – documentationdocumentation gLite v1.2 released 22/07/2005 – File Transfer Service and the File Transfer Agents – improvements in all modules. – documentation page.documentation gLite v. 1.1 released 13/05/2005 –File Transfer Service and the Metadata Catalog –documentationdocumentation gLite v. 1.0 released 05/04/2005 –documentation page.documentation http://www.glite.org/

17. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 17 The GILDA t-Infrastructure

18. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 18 EGEE Virtuous Cycle Researchers Diagnosticians Designers User Induction Training Outreach Dissemination Successful Users User Registration Positive Referrals Application Developers Dissemination Developer Initial Training Application Developers At home on EGEE Positive Referrals Developer Advanced Training New EGEE Applications Pushing Limits Advanced EGEE Applications Success Stories & Experts

19. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 19 JRA1: Middleware Engineering and Integration JRA2: Quality Assurance JRA3: Security JRA4: Network Services Development SA1: Grid Operations, Support and Management SA2: Network Resource Provision NA1: Management NA2: Dissemination and Outreach NA3: User Training and Education NA4: Application Identification and Support NA5: Policy and International Cooperation 24% Joint Research28% Networking 48% Services Emphasis in EGEE is on operating a production Grid and on supporting the end-users. The EGEE Activities

20. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 20 The GILDA project (https://gilda.ct.infn.it)

21. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 21 The GILDA Test-bed (https://gilda.ct.infn.it/testbed.html) ~15 sites in 3 continents !

22. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 22 All gLite Services available and tested on GILDA gLite Services

23. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 23 The GILDA Sponsors (https://gilda.ct.infn.it/sponsors.html)

24. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 24 The GILDA Certification Authority (https://gilda.ct.infn.it/CA)

25. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 25 The GILDA Certification Authority

26. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 26 The GILDA Virtual Organization

27. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 27 The GILDA VOMS

28. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 28 The GILDA VOMS Groups and Roles

29. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 29 The GILDA Accounting System (based on DGAS)

30. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 30 The GILDA Monitoring System (based on GridICE) (http://alifarm7.ct.infn.it:50080/gridice)

31. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 31 The Grid Tutor (https://grid-tutor.ct.infn.it, https://glite-tutor.ct.infn.it)

32. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 32 The Grid Demonstrator (1/2) (https://grid-demo.ct.infn.it, https://glite-demo.ct.infn.it) Demonstrator Applications

33. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 33 GEMS example Interactive MPI jobs !

34. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 34 hadronTherapy example CATANA beam line in reality CATANA beam line simulated by hadronTherapy hadronTherapy in GENIUS

35. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 35 GATE example

36. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 36 gMOD example Grid for Entertainment !

37. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 37 The GILDA User Interface Plug&Play combined (https://gilda.ct.infn.it/UIPnPcomb/)

38. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 38 UI Tutorial layout and acronyms Students User Interfaces UI PnP UI PnP UI PnP UI PnP WN RB/BDII CE SE LFC Catalog FireMan Catalog RB/BDII Test RB RLS Catalog

39. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 39 WMS layout in GILDA UI LCG UI gLite UI PnP “combined” RB LCG RB gLite BDII LCG CE LCG CE gLite CE LCG WN LCG WN LCG … WN gLite WN LCG WN LCG … GILDA site

40. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 40 DMS layout in GILDA UI LCG UI gLite UI PnP “combined” LCG RLS Catalog LCG LFC Catalog gLite Fireman Catalog SE LCG GILDA site SE LCG GILDA site … SRM d C a c h e GILDA site I/O s e r v e r SRM d C a c h e GILDA site I/O s e r v e r

41. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 41 IS layout in GILDA UI LCG UI gLite UI PnP “combined” BDII LCG R-GMA gLite CE LCG WN LCG WN LCG … GILDA site SE LCG CE gLite WN gLite GILDA site SRM d C a c h e I/O s e r v e r

42. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 42 The GILDA Live User Interface (https://gilda.ct.infn.it/live-cd/)

43. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 43 The GILDA Tutorials/Demonstrations (1/2) (https://gilda.ct.infn.it/tutorials.html) 2004 Edinburgh, 7 April 2004, slides, pictures Tunis, 22-23 April 2004, pictures Edinburgh, 26-28 April 2004, slides, pictures CERN, 17-19 May 2004, pictures Catania, 24-25 May 2004, home page, pictures Dubna, 29 June - 2 July 2004, agenda Edinburgh, 6 July 2004, home page Catania, 14-16 July 2004, home page, pictures Vico Equense, 19 July 2004, slides, pictures Vico Equense, 6-10 September 2004, home page Catania, 4-8 October 2004, home page, agenda Vilnius, 5-6 October 2004, agenda London, 6 October 2004 Madrid, 6-7 October 2004, agenda Heidelberg, 11-14 October 2004 CERN, 16 October 2004 Prague, 26 october 2004, home page Warsaw, 4-6 November 2004, home page, agenda Lyon, 9-10 November 2004, agenda The Hague, 15-17 November 2004, pictures Merida, 15-20 November 2004, home page, agenda, slides, pictures Tunis, 20 November 2004 Rio de Janeiro, 22-23 November 2004, home page, agenda, pictures The Hague, 24 November 2004, agenda CERN, 29-30 November 2004, agenda Kosice, 30 November - 1 December 2004, agenda Tunis, 6-7 December 2004 Bochum, 7-10 December 2004, home page, agenda Edinburgh, 8 December 2004, home page Istanbul, 9-10 December 2004, agenda, slides, pictures Shanghai, 9-10 December 2004, agenda Aurillac, 13-14 December 2004 Prague, 16 December 2004, home page, pictures Tel Aviv, 22-23 December 2004, agenda, picturesslidespictures slidespictures home pagepicturesagendahome page picturesslidespictureshome page agenda home page agenda pictureshome pageagenda slidespictureshome page agendapicturesagenda home pageagendahome pageagendaslidespicturesagendahome pagepicturesagendapictures 2005 CERN, 13 January 2005, agenda Torino, 18-19 January 2005, home page, agenda CERN, 20 January 2005, agenda CERN, 2-4 February 2005, agenda Roma, 3 February 2005, home page, agenda, pictures Sydney, 3-4 February 2005, home page CERN, 9-11 February 2005, agenda Amsterdam, 14-16 February 2005, home page Trento, 23-25 February 2005, home page, agenda Amsterdam, 28 February - 1 March 2005, home page Julich, 9 March 2005, Clermont-Ferrand, 9-31 March 2005, agenda Vienna, March-August 2005 Hamburg, 23-24 March 2005, home page, agenda Ula-Merida, 31 March-1 April 2005, agenda Zilina, 4 April 2005, home page and agenda Edinburgh, 9-13 May 2005, home page and agenda St.Augustin,25 May 2005, home page and agenda Catania, 13-15 June 2005, home page, agenda, pictures Valencia, 14-16 June 2005, home page, agenda Lyon, 17 June 2005, home page and agenda Bratislava, 27-30 June 2005, agenda Karlsruhe, 8 July 2005, home page and agenda Vico Equense, 10-22 July 2005, home page, agenda Budapest, 11-16 July 2005, home page, agenda Clermont-Ferrand, 25-27 July 2005, home page, agenda Madrid, 26-27 July 2005, home page, agenda Swansea, 6 August 2005, home page and agenda Taipei, 22-23 August 2005, home page and agenda CERN, 24 August 2005, home page and agenda Tokyo, 25-26 August 2005, home page and agenda Seoul, 29-30 August 2005, home page and agenda Clermont-Ferrand, 05-07 September 2005, home page, agenda Lausanne, 5-9 September 2005, home page and agenda Saint-Malo, 12-15 September 2005, home page, agenda Karlsruhe, 26-30 September 2005, home page, agenda Raanana, 28-29 September 2005, home page, agenda London, 6 October 2005, home page and agenda Sophia Antipolis, 10-14 October 2005, home page, agendaagendahome pageagenda home pageagendapictureshome pageagendahome page agendahome pageagendahome pageagenda home page and agenda home pageagendapictureshome pageagendahome page and agendaagendahome page and agendahome pageagendahome pageagendahome pageagendahome pageagendahome page and agenda home page agendahome page and agendahome pageagendahome pageagendahome pageagendahome page and agendahome page agenda

44. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 44 The GILDA Tutorials/Demonstrations (2/2) (https://gilda.ct.infn.it/tutorials.html)

45. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 45 The GILDA Video Tutorials (https://gilda.ct.infn.it/video.html)

46. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 46 GILDA summary numbers 15 sites in 3 continents > 2300 certificates issued, >15% renewed at least once > 75 tutorials and demos performed in 15 months > 50 jobs/day on the average Job success rate above 80% > 950,000 hits (> 46,000 visits) on (of) the web site from 10’s of different countries > 0.6 TB of videos and UI’s downloaded from the web site

47. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 47 Training achievements

48. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 48 EGEE-NA4 Applications and GILDA 7 Virtual Organizations supported: –Biomedicine (Biomed) –Earth Science Academy (ESR) –Earth Science Industry (CGG) –Astroparticle Physics (MAGIC) –Computational Chemistry (GEMS) –Grid Search Engines (GRACE) –Astrophysics (PLANCK) Development of complete interfaces with GENIUS for 3 Biomed Applications: GATE, hadronTherapy, and Friction/Arlecore Development of complete interfaces with GENIUS for 4 Generic Applications: EGEODE (CGG), MAGIC, GEMS, and CODESA-3D (ESR) (successfull demos of EGEODE and GEMS at EGEE review) Development of complete interfaces with GENIUS for 16 demonstrative applications available on the GILDA Grid Demonstrator (https://grid-demo.ct.infn.it)https://grid-demo.ct.infn.it Development of complete interface with CLI for NEMO

49. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Tutorial, Pisa, 23.10.2005 49 Summary and conclusions The EGEE middleware: –Is exiting prototyping phase and entering real production phase (LHC first real data are only 2 years away from now!) –Implements a full and complete stack of grid services that can be used all together or separately at user’s discretion –Closely follow the standardization process going in GGF and other for a GILDA is a real virtual laboratory for dissemination of grid computing: –It is a “de facto” standard t-Infrastructure adopted both by EGEE and some forthcoming EU-FP6 projects (EELA, EUCHINAGRID, EUMEDGRID, ICEAGE) –It is a complete suite of grid elements (test-bed, CA, VO, monitoring system, web portal, live user interface, user interface plug&play) and applications fully dedicated to dissemination purposes and pre-porting of new applications to EGEE Infrastructure –GILDA runs latest production (stable) version of both the LCG grid middleware but it is also early adopting gLite in order to make the transition to the new middleware smoother and easier