1. INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org Grid Infrastructure & Related Projects Eddie Aronovich Tel-Aviv University, School of CS Eddie.Aronovich@cs.tau.ac.il

2. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 2 Acknowledgements Presentation is based on slides from: –Roberto Barbera, University of Catania and INFN (EGEE Tutorial Roma, 02.11.2005) –Mike Mineter, Concepts of grid computing –Fabrizio Gagliardi, EGEE Project Director, CERN, Geneva, Switzerland (Naregi Symposium 2005 – Tokyo) –Fabrizio Gagliardi, EGEE Project Director, CERN, Geneva, Switzerland (APAC, 27 September 2005) –Guy Warner, NeSC Training Team (An Induction to EGEE for GOSC and the NGS NeSC, 8th December 2004 ) –http://www.beingrid.eu

3. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 3 What is a grid ? EGEE project Connectivity with other projects How a job is submitted A few more words

4. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 4 When the Grid can do ? Coordinates Distributed Resources Open for connectivity (using standard, open, general-purpose protocols) Deliver non-trivial qualities of service

5. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 5 A grid Grid middleware runs on each shared resource –Data storage –(Usually) batch jobs on pools of processors Users join VO’s Virtual organisation negotiates with sites to agree access to resources Distributed services (both people and middleware) enable the grid INTERNET

6. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 6 What is a grid ? EGEE project How a job is submitted A few more words

7. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 7 EGEE project in 1K words https://goc.grid-support.ac.uk/gridsite/monitoring/

8. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 8 30+ Grid Projects @ EGEE’06

9. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 9 20 + Businesses @ EGEE06 Capitalising on e-Science to make e-Business

10. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 10 Related EU projects EUGRID ISS e G

11. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 11 EGEE as partner Ongoing collaborations –with non EU partners in EGEE: US, Israel, Russia, Korea, Taiwan… –with other European infrastructure projects:  GÉANT  DEISA  SEE-GRID –with other European grid projects  DILIGENT  GRIDCC –with non-European projects:  OSG: OpenScienceGrid (USA)  NAREGI (Japan) EGEE as incubator –>10 related projects have been created

12. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 12 Collaborating e-Infrastructures Potential for linking ~80 countries TWGRID

13. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 13 e-Infrastructure implementation blocks From a talk by Mario Campolargo, Brussels, 30 May 2005

14. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 14 Support for digital repositories GÉANT. INFRASTRUCTURE GRID. INFRASTRUCTURE KNOWLEDGE. INFRASTRUCTURE 3 layered model to support access to heterogeneous information and connect resources through common shared services Grids for digital repositories: Sharing of resources Secure Access Control Data management Execution of computationally demanding applications (e.g. multi- media content)

15. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 15 What is Service Oriented Architecture (SOA)? An SOA application is a composition of services A “service” is the atomic unit of an SOA Services encapsulate a business process Service Providers Register themselves Service use involves: Find, Bind, Execute Most well-known instance is Web Services Service Registry Service Provider Service Consumer FindRegister Bind, Execute

16. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 16 What is a grid ? EGEE project Connectivity with other projects How a job is submitted A few more words

17. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 17 Jobs life (major components) ReplicaCatalogue Logging & Book-keeping ResourceBrokerStorageElementComputingElement InformationService Job Status DataSets info Author. &Authen. Job Submit Event Job Query Job Status Input “sandbox” Input “sandbox” + Broker Info Output “sandbox” Publish SE & CE info “User interface”

18. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 18 UI Network Server Job Contr. Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB/WMS node Characts. & status

19. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 19 UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status submitted Job Status UI: allows users to access the functionalities of the WMS (via command line, GUI, C++ and Java APIs)

20. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 20 UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status edg-job-submit myjob.jdl Myjob.jdl JobType = “Normal”; Executable = "$(CMS)/exe/sum.exe"; InputSandbox = {"/home/user/WP1testC","/home/file*”, "/home/user/DATA/*"}; OutputSandbox = {“sim.err”, “test.out”, “sim.log"}; Requirements = other. GlueHostOperatingSystemName == “linux" && other. GlueHostOperatingSystemRelease == "Red Hat 7.3“ && other.GlueCEPolicyMaxCPUTime > 10000; Rank = other.GlueCEStateFreeCPUs; submitted Job Status Job Description Language (JDL) to specify job characteristics and requirements

21. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 21 UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage Input Sandbox files Job waiting submitted Job Status NS: network daemon responsible for accepting incoming requests

22. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 22 UI Network Server Job Contr. - CondorG Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage waiting submitted Job Status WM: acts to satisfy the request Job Workload manager

23. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 23 UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage waiting submitted Job Status Match- Maker/ Broker Where must this job be executed ?

24. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 24 UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage waiting submitted Job Status Match- Maker/ Broker Matchmaker: responsible to find the “best” CE for a job

25. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 25 UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage waiting submitted Job Status Match- Maker/ Broker Where are (which SEs) the needed data ? What is the status of the Grid ?

26. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 26 UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage waiting submitted Job Status Match- Maker/ Broker CE choice

27. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 27 UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage waiting submitted Job Status Job Adapter Job Adapter: responsible for the final “touches” to the job before performing submission (e.g. creation of wrapper script, PFN, etc.)

28. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 28 UI Network Server Job Contr. Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage Job Status Job Controller: responsible for the actual job management operations (done via CondorG) Job submitted waiting ready

29. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 29 UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage Job Status Job submitted waiting ready scheduled

30. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 30 UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node RB storage Job Status submitted waiting ready scheduled running “Grid enabled” data transfers/ accesses Job Input Sandbox files

31. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 31 UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node RB storage Job Status Output Sandbox files submitted waiting ready scheduled running done

32. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 32 UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node RB storage Job Status submitted waiting ready scheduled running done edg-job-get-output

33. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 33 UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node RB storage Job Status Output Sandbox files submitted waiting ready scheduled running done cleared

34. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 34 UI Log Monitor Logging & Bookkeeping Network Server Job Contr. - CondorG Workload Manager Computing Element RB node LM: parses CondorG log file (where CondorG logs info about jobs) and notifies LB LB: receives and stores job events; processes corresponding job status Log of job events edg-job-status edg-job-get-logging-info Job status

35. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 35 What about security ? Certification Domain A Server XServer Y Policy Authority Policy Authority Task Domain B Sub-Domain A1 GSI Certification Authority Sub-Domain B1 Authority Federation Service Virtual Organization Domain No Cross- Domain Trust

36. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 36 What is a grid ? EGEE project Connectivity with other projects How a job is submitted A few more words

37. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 37 Used CPU time this year

38. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 38 How many “hours” there are in a year ?

39. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 39 Over the Chasm

40. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 40 Applications in EGEE Astrophysics - MAGIC, PlanckPlanck Computational Chemistry Earth Sciences - Earth Observation, Solid Earth Physics, Hydrology, Climate Financial Simulation - E-GRID Fusion Geophysics - EGEODE High Energy Physics - LHC (ALICE, ATLAS, CMS, LHCb) BaBar, CDF, DØ, ZEUSATLAS Life Sciences - Bioinformatics (Drug Discovery, GPS@, Xmipp_MLrefine, etc.) Medical imaging (GATE, CDSS, gPTM3D, SiMRI 3D, etc.) Multimedia press for more details

41. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 41 Steps to become grid production Contact IAG needed contribution: –30CPUs or cores –1TB of storage attached to the processosrs –4 computers for operational activities Plan your site Get a certificates Install the site Certify the site….

42. INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org Questions ?

43. Enabling Grids for E-sciencE INFSO-RI-508833 Eddie Aronovich – CS_IAP, Apr 2007 43 Contacts Israeli Academic Grid (IAG) http://iag.iucc.ac.il/ EGEE Website http://www.eu-egee.org Grid Demos https://glite-demo.ct.infn.it/