INFSO-RI Enabling Grids for E-sciencE Grid Infrastructure & Related Projects Eddie Aronovich Tel-Aviv University, School of CS
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr Acknowledgements Presentation is based on slides from: –Roberto Barbera, University of Catania and INFN (EGEE Tutorial Roma, ) –Mike Mineter, Concepts of grid computing –Fabrizio Gagliardi, EGEE Project Director, CERN, Geneva, Switzerland (Naregi Symposium 2005 – Tokyo) –Fabrizio Gagliardi, EGEE Project Director, CERN, Geneva, Switzerland (APAC, 27 September 2005) –Guy Warner, NeSC Training Team (An Induction to EGEE for GOSC and the NGS NeSC, 8th December 2004 ) –
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr What is a grid ? EGEE project Connectivity with other projects How a job is submitted A few more words
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr When the Grid can do ? Coordinates Distributed Resources Open for connectivity (using standard, open, general-purpose protocols) Deliver non-trivial qualities of service
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr A grid Grid middleware runs on each shared resource –Data storage –(Usually) batch jobs on pools of processors Users join VO’s Virtual organisation negotiates with sites to agree access to resources Distributed services (both people and middleware) enable the grid INTERNET
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr What is a grid ? EGEE project How a job is submitted A few more words
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr EGEE project in 1K words
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr Grid EGEE’06
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr EGEE06 Capitalising on e-Science to make e-Business
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr Related EU projects EUGRID ISS e G
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr EGEE as partner Ongoing collaborations –with non EU partners in EGEE: US, Israel, Russia, Korea, Taiwan… –with other European infrastructure projects: GÉANT DEISA SEE-GRID –with other European grid projects DILIGENT GRIDCC –with non-European projects: OSG: OpenScienceGrid (USA) NAREGI (Japan) EGEE as incubator –>10 related projects have been created
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr Collaborating e-Infrastructures Potential for linking ~80 countries TWGRID
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr e-Infrastructure implementation blocks From a talk by Mario Campolargo, Brussels, 30 May 2005
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr Support for digital repositories GÉANT. INFRASTRUCTURE GRID. INFRASTRUCTURE KNOWLEDGE. INFRASTRUCTURE 3 layered model to support access to heterogeneous information and connect resources through common shared services Grids for digital repositories: Sharing of resources Secure Access Control Data management Execution of computationally demanding applications (e.g. multi- media content)
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr What is Service Oriented Architecture (SOA)? An SOA application is a composition of services A “service” is the atomic unit of an SOA Services encapsulate a business process Service Providers Register themselves Service use involves: Find, Bind, Execute Most well-known instance is Web Services Service Registry Service Provider Service Consumer FindRegister Bind, Execute
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr What is a grid ? EGEE project Connectivity with other projects How a job is submitted A few more words
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr Jobs life (major components) ReplicaCatalogue Logging & Book-keeping ResourceBrokerStorageElementComputingElement InformationService Job Status DataSets info Author. &Authen. Job Submit Event Job Query Job Status Input “sandbox” Input “sandbox” + Broker Info Output “sandbox” Publish SE & CE info “User interface”
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB/WMS node Characts. & status
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status submitted Job Status UI: allows users to access the functionalities of the WMS (via command line, GUI, C++ and Java APIs)
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status edg-job-submit myjob.jdl Myjob.jdl JobType = “Normal”; Executable = "$(CMS)/exe/sum.exe"; InputSandbox = {"/home/user/WP1testC","/home/file*”, "/home/user/DATA/*"}; OutputSandbox = {“sim.err”, “test.out”, “sim.log"}; Requirements = other. GlueHostOperatingSystemName == “linux" && other. GlueHostOperatingSystemRelease == "Red Hat 7.3“ && other.GlueCEPolicyMaxCPUTime > 10000; Rank = other.GlueCEStateFreeCPUs; submitted Job Status Job Description Language (JDL) to specify job characteristics and requirements
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage Input Sandbox files Job waiting submitted Job Status NS: network daemon responsible for accepting incoming requests
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage waiting submitted Job Status WM: acts to satisfy the request Job Workload manager
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage waiting submitted Job Status Match- Maker/ Broker Where must this job be executed ?
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage waiting submitted Job Status Match- Maker/ Broker Matchmaker: responsible to find the “best” CE for a job
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage waiting submitted Job Status Match- Maker/ Broker Where are (which SEs) the needed data ? What is the status of the Grid ?
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage waiting submitted Job Status Match- Maker/ Broker CE choice
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage waiting submitted Job Status Job Adapter Job Adapter: responsible for the final “touches” to the job before performing submission (e.g. creation of wrapper script, PFN, etc.)
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage Job Status Job Controller: responsible for the actual job management operations (done via CondorG) Job submitted waiting ready
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node CE characts & status SE characts & status RB storage Job Status Job submitted waiting ready scheduled
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node RB storage Job Status submitted waiting ready scheduled running “Grid enabled” data transfers/ accesses Job Input Sandbox files
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node RB storage Job Status Output Sandbox files submitted waiting ready scheduled running done
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node RB storage Job Status submitted waiting ready scheduled running done edg-job-get-output
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Network Server Job Contr. - CondorG Workload Manager Replica Location Server Inform. Service Computing Element Storage Element RB node RB storage Job Status Output Sandbox files submitted waiting ready scheduled running done cleared
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr UI Log Monitor Logging & Bookkeeping Network Server Job Contr. - CondorG Workload Manager Computing Element RB node LM: parses CondorG log file (where CondorG logs info about jobs) and notifies LB LB: receives and stores job events; processes corresponding job status Log of job events edg-job-status edg-job-get-logging-info Job status
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr What about security ? Certification Domain A Server XServer Y Policy Authority Policy Authority Task Domain B Sub-Domain A1 GSI Certification Authority Sub-Domain B1 Authority Federation Service Virtual Organization Domain No Cross- Domain Trust
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr What is a grid ? EGEE project Connectivity with other projects How a job is submitted A few more words
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr Used CPU time this year
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr How many “hours” there are in a year ?
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr Over the Chasm
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr Applications in EGEE Astrophysics - MAGIC, PlanckPlanck Computational Chemistry Earth Sciences - Earth Observation, Solid Earth Physics, Hydrology, Climate Financial Simulation - E-GRID Fusion Geophysics - EGEODE High Energy Physics - LHC (ALICE, ATLAS, CMS, LHCb) BaBar, CDF, DØ, ZEUSATLAS Life Sciences - Bioinformatics (Drug Discovery, Xmipp_MLrefine, etc.) Medical imaging (GATE, CDSS, gPTM3D, SiMRI 3D, etc.) Multimedia press for more details
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr Steps to become grid production Contact IAG needed contribution: –30CPUs or cores –1TB of storage attached to the processosrs –4 computers for operational activities Plan your site Get a certificates Install the site Certify the site….
INFSO-RI Enabling Grids for E-sciencE Questions ?
Enabling Grids for E-sciencE INFSO-RI Eddie Aronovich – CS_IAP, Apr Contacts Israeli Academic Grid (IAG) EGEE Website Grid Demos