INFSO-RI Enabling Grids for E-sciencE The EGEE middleware and the GILDA t-Infrastructure Roberto Barbera University of Catania and INFN EGEE Tutorial Pisa,

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, JRA1: Middleware Engineering and Integration JRA2: Quality Assurance JRA3: Security JRA4: Network Services Development SA1: Grid Operations, Support and Management SA2: Network Resource Provision NA1: Management NA2: Dissemination and Outreach NA3: User Training and Education NA4: Application Identification and Support NA5: Policy and International Cooperation 24% Joint Research28% Networking 48% Services Emphasis in EGEE is on operating a production Grid and on supporting the end-users. The EGEE Activities

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, Outline Generalities Overview of the gLite middleware The GILDA t-Infrastructure –services –tools –applications Summary and conclusions

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, Generalities

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, User Interface A multi-VO Grid User Interface Grid services EGEE is establishing a production grid service to support multiple, diverse VO’s

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, A multi-VO grid Users join VO Virtual organisation contributes resources & negotiates access Grid middleware runs on each resource –Data storage –(Usually) batch jobs on pools of processors Additional services (both people and middleware) enable the grid Effect: collaboration across administrative domains INTERNET

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, Introduction

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, VO’s and middleware The tools, services used by the VO’s applications Application development environment, portals, semantics, workflow In EGEE-1: Mainly VO-specific Basic Grid services: AA, job submission, info, … Middleware: “collective services” Application toolkits, standards Application Emphasis of EGEE -1 middleware

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, UI JDL Logging & Book-keeping ResourceBroker Job Submission ServiceStorageElementComputeElement InformationService Job Status ReplicaCatalogue DataSets info Author. &Authen. Job Submit Event Job Query Job Status Input “sandbox” Input “sandbox” + Broker Info Globus RSL Output “sandbox” Job Status Publish grid-proxy-init Expanded JDL SE & CE info A typical job workflow

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, Parts of the Grid “ecosystem”... LCG EGEE Used in USA EU NextGrid DEISA GridCC Future grids EDG GlobusMyProxyCondor... VDT DataTAG CrossGrid... OSG, … SRM …

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, Overview of EGEE Middleware The gLite Grid services follow a Service Oriented Architecture –facilitate interoperability among Grid services –allow easier compliance with upcoming standards Architecture is not bound to specific implementations –services are expected to work together –services can be deployed and used independently The gLite service decomposition has been largely influenced by the work performed in the LCG project

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, EGEE middlewares face to face LCG (the present) Security –GSI Job Management –Condor + Globus –CE, WN –Logging & Bookkeeping Data Management –LCG services Information & Monitoring –BDII (evolution of MDS) Grid Access –CLI + API gLite (the future) Security –GSI and VOMS Job Management –Condor + Globus + blahp –CE, WN –Logging & Bookkeeping –Job Provenance –Package management Data Management –LFC –gLite-I/O + FiReMan Information & Monitoring –BDII –R-GMA + Service Discovery Grid Access –CLI + API + Web Services

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, gLite components overview Grid Access Service API Access Services Job Provenance Job Management Services Computing Element Workload Management Package Manager Metadata Catalog Data Services Storage Element Data Movement File & Replica Catalog Authorization Security Services Authentication Auditing Information & Monitoring Information & Monitoring Services Job Monitoring Site Proxy CLI Dynamic Connectivity Service Monitoring Service Discovery Accounting Near Future now

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, Job Management Services

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, DM Interaction Overview File and Replica Catalog StorageIndex Fireman Database WMS Storage Element SRM Storage gLite I/OgridFTP File Transfer and Placement Service FTS FPS Transfer Agent Database VOMS MyProxy Get credential Store credential File I/O File namespace and Metadata mgmt File replication Proxy renewalReplica Location WSDL API

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, gLite status gLite v1.3 released 05/08/2005 –File Placement Service, File Placement Service clients added to UI and WNs modules –new data transfer agents including architecture refactoring to allow proper inter-VO scheduling and – documentationdocumentation gLite v1.2 released 22/07/2005 – File Transfer Service and the File Transfer Agents – improvements in all modules. – documentation page.documentation gLite v. 1.1 released 13/05/2005 –File Transfer Service and the Metadata Catalog –documentationdocumentation gLite v. 1.0 released 05/04/2005 –documentation page.documentation

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA t-Infrastructure

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, EGEE Virtuous Cycle Researchers Diagnosticians Designers User Induction Training Outreach Dissemination Successful Users User Registration Positive Referrals Application Developers Dissemination Developer Initial Training Application Developers At home on EGEE Positive Referrals Developer Advanced Training New EGEE Applications Pushing Limits Advanced EGEE Applications Success Stories & Experts

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, JRA1: Middleware Engineering and Integration JRA2: Quality Assurance JRA3: Security JRA4: Network Services Development SA1: Grid Operations, Support and Management SA2: Network Resource Provision NA1: Management NA2: Dissemination and Outreach NA3: User Training and Education NA4: Application Identification and Support NA5: Policy and International Cooperation 24% Joint Research28% Networking 48% Services Emphasis in EGEE is on operating a production Grid and on supporting the end-users. The EGEE Activities

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA project (

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA Test-bed ( ~15 sites in 3 continents !

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, All gLite Services available and tested on GILDA gLite Services

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA Sponsors (

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA Certification Authority (

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA Certification Authority

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA Virtual Organization

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA VOMS

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA VOMS Groups and Roles

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA Accounting System (based on DGAS)

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA Monitoring System (based on GridICE) (

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The Grid Tutor (

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The Grid Demonstrator (1/2) ( Demonstrator Applications

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, GEMS example Interactive MPI jobs !

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, hadronTherapy example CATANA beam line in reality CATANA beam line simulated by hadronTherapy hadronTherapy in GENIUS

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, GATE example

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, gMOD example Grid for Entertainment !

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA User Interface Plug&Play combined (

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, UI Tutorial layout and acronyms Students User Interfaces UI PnP UI PnP UI PnP UI PnP WN RB/BDII CE SE LFC Catalog FireMan Catalog RB/BDII Test RB RLS Catalog

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, WMS layout in GILDA UI LCG UI gLite UI PnP “combined” RB LCG RB gLite BDII LCG CE LCG CE gLite CE LCG WN LCG WN LCG … WN gLite WN LCG WN LCG … GILDA site

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, DMS layout in GILDA UI LCG UI gLite UI PnP “combined” LCG RLS Catalog LCG LFC Catalog gLite Fireman Catalog SE LCG GILDA site SE LCG GILDA site … SRM d C a c h e GILDA site I/O s e r v e r SRM d C a c h e GILDA site I/O s e r v e r

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, IS layout in GILDA UI LCG UI gLite UI PnP “combined” BDII LCG R-GMA gLite CE LCG WN LCG WN LCG … GILDA site SE LCG CE gLite WN gLite GILDA site SRM d C a c h e I/O s e r v e r

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA Live User Interface (

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA Tutorials/Demonstrations (1/2) ( Edinburgh, 7 April 2004, slides, pictures Tunis, April 2004, pictures Edinburgh, April 2004, slides, pictures CERN, May 2004, pictures Catania, May 2004, home page, pictures Dubna, 29 June - 2 July 2004, agenda Edinburgh, 6 July 2004, home page Catania, July 2004, home page, pictures Vico Equense, 19 July 2004, slides, pictures Vico Equense, 6-10 September 2004, home page Catania, 4-8 October 2004, home page, agenda Vilnius, 5-6 October 2004, agenda London, 6 October 2004 Madrid, 6-7 October 2004, agenda Heidelberg, October 2004 CERN, 16 October 2004 Prague, 26 october 2004, home page Warsaw, 4-6 November 2004, home page, agenda Lyon, 9-10 November 2004, agenda The Hague, November 2004, pictures Merida, November 2004, home page, agenda, slides, pictures Tunis, 20 November 2004 Rio de Janeiro, November 2004, home page, agenda, pictures The Hague, 24 November 2004, agenda CERN, November 2004, agenda Kosice, 30 November - 1 December 2004, agenda Tunis, 6-7 December 2004 Bochum, 7-10 December 2004, home page, agenda Edinburgh, 8 December 2004, home page Istanbul, 9-10 December 2004, agenda, slides, pictures Shanghai, 9-10 December 2004, agenda Aurillac, December 2004 Prague, 16 December 2004, home page, pictures Tel Aviv, December 2004, agenda, picturesslidespictures slidespictures home pagepicturesagendahome page picturesslidespictureshome page agenda home page agenda pictureshome pageagenda slidespictureshome page agendapicturesagenda home pageagendahome pageagendaslidespicturesagendahome pagepicturesagendapictures 2005 CERN, 13 January 2005, agenda Torino, January 2005, home page, agenda CERN, 20 January 2005, agenda CERN, 2-4 February 2005, agenda Roma, 3 February 2005, home page, agenda, pictures Sydney, 3-4 February 2005, home page CERN, 9-11 February 2005, agenda Amsterdam, February 2005, home page Trento, February 2005, home page, agenda Amsterdam, 28 February - 1 March 2005, home page Julich, 9 March 2005, Clermont-Ferrand, 9-31 March 2005, agenda Vienna, March-August 2005 Hamburg, March 2005, home page, agenda Ula-Merida, 31 March-1 April 2005, agenda Zilina, 4 April 2005, home page and agenda Edinburgh, 9-13 May 2005, home page and agenda St.Augustin,25 May 2005, home page and agenda Catania, June 2005, home page, agenda, pictures Valencia, June 2005, home page, agenda Lyon, 17 June 2005, home page and agenda Bratislava, June 2005, agenda Karlsruhe, 8 July 2005, home page and agenda Vico Equense, July 2005, home page, agenda Budapest, July 2005, home page, agenda Clermont-Ferrand, July 2005, home page, agenda Madrid, July 2005, home page, agenda Swansea, 6 August 2005, home page and agenda Taipei, August 2005, home page and agenda CERN, 24 August 2005, home page and agenda Tokyo, August 2005, home page and agenda Seoul, August 2005, home page and agenda Clermont-Ferrand, September 2005, home page, agenda Lausanne, 5-9 September 2005, home page and agenda Saint-Malo, September 2005, home page, agenda Karlsruhe, September 2005, home page, agenda Raanana, September 2005, home page, agenda London, 6 October 2005, home page and agenda Sophia Antipolis, October 2005, home page, agendaagendahome pageagenda home pageagendapictureshome pageagendahome page agendahome pageagendahome pageagenda home page and agenda home pageagendapictureshome pageagendahome page and agendaagendahome page and agendahome pageagendahome pageagendahome pageagendahome pageagendahome page and agenda home page agendahome page and agendahome pageagendahome pageagendahome pageagendahome page and agendahome page agenda

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA Tutorials/Demonstrations (2/2) (

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, The GILDA Video Tutorials (

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, GILDA summary numbers 15 sites in 3 continents > 2300 certificates issued, >15% renewed at least once > 75 tutorials and demos performed in 15 months > 50 jobs/day on the average Job success rate above 80% > 950,000 hits (> 46,000 visits) on (of) the web site from 10’s of different countries > 0.6 TB of videos and UI’s downloaded from the web site

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, Training achievements

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, EGEE-NA4 Applications and GILDA 7 Virtual Organizations supported: –Biomedicine (Biomed) –Earth Science Academy (ESR) –Earth Science Industry (CGG) –Astroparticle Physics (MAGIC) –Computational Chemistry (GEMS) –Grid Search Engines (GRACE) –Astrophysics (PLANCK) Development of complete interfaces with GENIUS for 3 Biomed Applications: GATE, hadronTherapy, and Friction/Arlecore Development of complete interfaces with GENIUS for 4 Generic Applications: EGEODE (CGG), MAGIC, GEMS, and CODESA-3D (ESR) (successfull demos of EGEODE and GEMS at EGEE review) Development of complete interfaces with GENIUS for 16 demonstrative applications available on the GILDA Grid Demonstrator ( Development of complete interface with CLI for NEMO

Enabling Grids for E-sciencE INFSO-RI EGEE Tutorial, Pisa, Summary and conclusions The EGEE middleware: –Is exiting prototyping phase and entering real production phase (LHC first real data are only 2 years away from now!) –Implements a full and complete stack of grid services that can be used all together or separately at user’s discretion –Closely follow the standardization process going in GGF and other for a GILDA is a real virtual laboratory for dissemination of grid computing: –It is a “de facto” standard t-Infrastructure adopted both by EGEE and some forthcoming EU-FP6 projects (EELA, EUCHINAGRID, EUMEDGRID, ICEAGE) –It is a complete suite of grid elements (test-bed, CA, VO, monitoring system, web portal, live user interface, user interface plug&play) and applications fully dedicated to dissemination purposes and pre-porting of new applications to EGEE Infrastructure –GILDA runs latest production (stable) version of both the LCG grid middleware but it is also early adopting gLite in order to make the transition to the new middleware smoother and easier