Grid-Ireland John Morrison, University College Cork (UCC) Brian Coghlan, Trinity College Dublin (TCD) Andy Shearer, NUI Galway (NUIG) Ron Perrott, Queens University Belfast (QUB) Grid-IrelandArmagh 26-NOV-2002

Grid-IrelandArmagh 26-NOV-2002 Grid-Ireland Grid-Ireland Enterprise Ireland funded from Oct Enterprise Ireland funded from Oct TCD, UCC, NUIG, QUB TCD, UCC, NUIG, QUB CosmoGrid: first virtual organization HEA Funded from Oct-2002 HEA Funded from Oct-2002 Dublin Institute for Advanced Studies with UCD, NUI, G, UCC, HEAnet, DCU, TCD, Met Office & Armagh Observatory Dublin Institute for Advanced Studies with UCD, NUI, G, UCC, HEAnet, DCU, TCD, Met Office & Armagh Observatory VO supported by Grid-Ireland & HEAnet VO supported by Grid-Ireland & HEAnet

Grid-IrelandArmagh 26-NOV-2002 CosmoGrid

Grid-IrelandArmagh 26-NOV-2002 Grids What are they? Beowulf clusters and Grid computing technology will likely merge in the next decade … and together they form the Grid Gordon Bell and Jim Gray (Feb-2002) coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations Ian Foster and Carl Kesselman (1998)

Grid-IrelandArmagh 26-NOV-2002 Grids: what are they ? an architecture an architecture IF a system acts as a unified resource IF a system acts as a unified resource THEN we say it presents a single system image (SSI) THEN we say it presents a single system image (SSI) conceptual model = computer architecture conceptual model = computer architecture NOT client/server model NOT client/server model gives each user the illusion of a single machine gives each user the illusion of a single machine SSI components: SSI components: hardware / OS / services hardware / OS / services SSI services: SSI services: login / user interface / admin / time / filesystem / … login / user interface / admin / time / filesystem / …

Grid-IrelandArmagh 26-NOV-2002 The Holy Grid Grail Submit a problem at a high level to the Grid The grid will ?  Analyse the problem: what code & data are available, where are they ?  Dynamically allocate tasks to available computational tasks  Determine the distribution and degree of parallelisation given the current grid resources  Security is automatic  Monitor remotely Concentrate upon the problem NOT how to compute it.

Grid-IrelandArmagh 26-NOV-2002 Current Grid Architecture Space

Grid-IrelandArmagh 26-NOV-2002 Grid Musicians Computing Elements SystemManagers Scientists OperatingSystems File Systems StorageElements Mass Storage Systems HPSS, Castor User Accounts Certificate Authorities ApplicationDevelopers Batch Systems PBS, LSF

Grid-IrelandArmagh 26-NOV-2002 Grid Symphony UI JDL Information Service Logging & Bookkeeping Job Submission Service Input Dataset Job Submit Event Job Status Computing ElementStorage Element Replica Catalog DB BLAST Output Dataset Resource Broker DB

Grid-IrelandArmagh 26-NOV-2002 Current Grid Architectures European CrossGrid [X#] 11 countries / 21 instutions 11 countries / 21 instutions European DataGrid [EDG] 11 countries / 21 instutions 11 countries / 21 instutionsGlobus Argonne Labs + ISI Argonne Labs + ISI CrossGrid DataGrid GLOBUS SITES Let’s look at these, starting with Globus

Grid-IrelandArmagh 26-NOV-2002 Globus Execution Flows: single entry point Certification Authorities Grid Information System Network Monitoring Cluster Nodes Disk Storage / MSSNFS Server Gatekeeper Users Application

Grid-IrelandArmagh 26-NOV-2002 Globus Execution Flows: multiple entry points Grid Information System Network Monitoring Gatekeepers (Cluster Nodes) Disk Storage / MSSNFS Server Users Application Certification Authorities Grid-Ireland gridkit 1.x works like this

Grid-IrelandArmagh 26-NOV-2002 Grid-Ireland Architecture, 2001/2002 GLOBUS SITES Applications Development Support Existing & Planned Grid Services Globus Replica Manager MPICH-G GRAM GSI Replica Catalog GASS MDS GridFTP Globus-IO CPUs Secondary Storage Physical Resources Local Filesystem Local Filesystem Notice how few services there are

Grid-IrelandArmagh 26-NOV-2002 DataGrid Execution Flows Certification Authorities VO Membership Grid Information System Resource Broker VO Replica Catalog NetworkMonitoring Worker Nodes Disk Storage / MSSStorageElement Computing Element (Gatekeeper) Users Application

Grid-IrelandArmagh 26-NOV-2002 Grid-Ireland Architecture, 2002/2003 DataGrid GLOBUS SITES Applications Development Support Existing & Planned Grid Services Globus Replica Manager VO Management VOMS VO Management VOMS Grid Info System R-GMA Grid Info System R-GMA MPICH-G VO Authorization LCAS VO Authorization LCAS DataGrid Replica Manager DataGrid Replica Manager Datagrid Broker RB Datagrid Broker RB GRAM GSI Replica Catalog GASS MDS GridFTP Globus-IO Resource Manager Resource Manager CPUs Resource Manager Resource Manager Secondary Storage Physical Resources Grid Databases Spitfire Grid Databases Spitfire Logging Bookkeeping L & B Logging Bookkeeping L & B SlashGrid Filesystem SlashGrid Filesystem Local Filesystem Local Filesystem

Grid-IrelandArmagh 26-NOV-2002 CrossGrid Execution Flows Users Application Certification Authorities VO Membership Grid Visualisation Kernel Grid Data Mining Grid Data Mining Interactive Distributed Data Access Roaming Access Distributed Data Collection Grid Monitoring Portals Performance Analysis MPI Verification Metrics and Benchmarks MPICH-G

Grid-IrelandArmagh 26-NOV-2002 Grid-Ireland Architecture, 2003/2004 CrossGrid DataGrid GLOBUS SITES Applications Development Support Existing & Planned Grid Services Grid Visualisation Kernel Grid Data Mining Grid Data Mining Interactive Distributed Data Access Globus Replica Manager Roaming Access VO Management VOMS VO Management VOMS Grid Info System R-GMA Grid Info System R-GMA MPICH-G Distributed Data Collection VO Authorization LCAS VO Authorization LCAS DataGrid Replica Manager DataGrid Replica Manager Datagrid Broker RB Datagrid Broker RB GRAM GSI Replica Catalog GASS MDS GridFTP Globus-IO Resource Manager Resource Manager CPUs Resource Manager Resource Manager Secondary Storage Resource Manager Resource Manager e.g. Instruments e.g. Instruments Resource Manager Resource Manager Optimization of Data Access Tertiary Storage Physical Resources Portals Performance Analysis MPI Verification Metrics and Benchmarks Resource Manager Resource Manager e.g. Visualization e.g. Visualization Grid Monitoring Grid Databases Spitfire Grid Databases Spitfire Logging Bookkeeping L & B Logging Bookkeeping L & B SlashGrid Filesystem SlashGrid Filesystem Local Filesystem Local Filesystem

Grid-IrelandArmagh 26-NOV-2002 WebCom-G Project Core Contribution: Condensed Graphs Grid Middleware A NEW execution model for the Grid Stress Testing Stress Testing Existing Grid Middleware Existing Grid Middleware Condensed Graphs Engine Condensed Graphs Engine Internet + Hardware

Grid-IrelandArmagh 26-NOV-2002 Execution Models Execution can be described by an acyclic graph Execution models may be classified by their FIRING RULE: control driven  thread of control control driven  thread of control data driven  opportunism data driven  opportunism demand driven  laziness demand driven  laziness Concepts: eager and lazy evaluation Condensed graphs are a chameleon: can execute a dynamic mix of all 3 models can execute a dynamic mix of all 3 models can dynamically mix eager & lazy evaluation can dynamically mix eager & lazy evaluation

Grid-IrelandArmagh 26-NOV-2002 Condensed Graphs Firing Rule Operands arrive at the operand ports: if in incorrect form, if in incorrect form, are grafted into correct form wait to be consumed wait to be consumed Firing rule requires operator & dest. too Operator: represents an appropriate process represents an appropriate process consumes the operands consumes the operands sends the result to destinations sends the result to destinations FIRE !

Grid-IrelandArmagh 26-NOV-2002 WebCom-G: DataGrid Use-case Execution Flows Certification Authorities VO Membership Grid Information System Resource Broker VO Replica Catalog NetworkMonitoring Worker Nodes Disk Storage / MSSStorageElement Computing Element (Gatekeeper) WebCom-GWebCom-G Logging & Bookkeeping

Grid-IrelandArmagh 26-NOV-2002 Grid-Ireland WebCom-G Architecture, 2003/2004 CrossGrid DataGrid GLOBUS SITES Applications Development Support Existing & Planned Grid Services Grid Visualisation Kernel Grid Data Mining Grid Data Mining Interactive Distributed Data Access Globus Replica Manager Roaming Access VO Management VOMS VO Management VOMS Grid Info System R-GMA Grid Info System R-GMA MPICH-G Distributed Data Collection VO Authorization LCAS VO Authorization LCAS DataGrid Replica Manager DataGrid Replica Manager Datagrid Broker RB Datagrid Broker RB GRAM GSI Replica Catalog GASS MDS GridFTP Globus-IO Resource Manager Resource Manager Secondary Storage Resource Manager Resource Manager e.g. Instruments e.g. Instruments Resource Manager Resource Manager Optimization of Data Access Tertiary Storage Physical Resources Portals Performance Analysis MPI Verification Resource Manager Resource Manager e.g. Visualization e.g. Visualization HDSM-G Grid Monitoring Grid Databases Spitfire Grid Databases Spitfire Logging Bookkeeping L & B Logging Bookkeeping L & B Loosely Coupled Loosely Coupled Tightly Coupled Tightly Coupled Data Intensive Data Intensive Complex Compositions Complex Compositions Stress Testing WebCom-G Condensed Graphs Grid Middleware Resource Manager Resource Manager CPUs Metrics and Benchmarks SlashGrid Filesystem SlashGrid Filesystem Local Filesystem Local Filesystem

Grid-IrelandArmagh 26-NOV-2002 WebCom-G: Pure CG Use-case Execution Flows Certification Authorities VO Membership Worker Nodes Disk Storage / MSSStorageElement WebCom-GWebCom-G VO Replica Catalog Grid Information System NetworkMonitoring Logging & Bookkeeping

Grid-IrelandArmagh 26-NOV-2002 CrossGrid DataGrid GLOBUS SITES Stress Testing WebCom-G Applications Development Support Existing & Planned Grid Services Physical Resources Grid Visualisation Kernel Interactive Distributed Data Access Globus Replica Manager VO Management VOMS VO Management VOMS Grid Info System R-GMA Grid Info System R-GMA MPICH-G VO Authorization LCAS VO Authorization LCAS DataGrid Replica Manager DataGrid Replica Manager Datagrid Broker RB Datagrid Broker RB GRAM GSI Replica Catalog GASS MDS GridFTP Globus-IO Resource Manager Resource Manager e.g. Instruments e.g. Instruments Resource Manager Resource Manager Optimization of Data Access Tertiary Storage Portals MPI Verification Resource Manager Resource Manager e.g. Visualization e.g. Visualization HDSM-G Grid Monitoring Grid Databases Spitfire Grid Databases Spitfire Logging Bookkeeping L & B Logging Bookkeeping L & B Loosely Coupled Loosely Coupled Tightly Coupled Tightly Coupled Data Intensive Data Intensive Complex Compositions Complex Compositions Condensed Graphs Grid Middleware Resource Manager Resource Manager CPUs Grid Data Mining Grid Data Mining Roaming Access Distributed Data Collection Performance Analysis Metrics and Benchmarks Resource Manager Resource Manager Secondary Storage SlashGrid Filesystem SlashGrid Filesystem Local Filesystem Local Filesystem Grid-Ireland WebCom-G Architecture, 2005/2006

Grid-IrelandArmagh 26-NOV-2002 WebCom-G: Stress Testing Existing Grid Middleware Existing Grid Middleware Internet + Hardware Loosely Coupled Loosely Coupled Tightly Coupled Tightly Coupled Data Intensive Data Intensive Complex Compositions Complex Compositions Stress Testing WebCom-G Condensed Graphs Grid Middleware ‘in-the-loop’ Imaging for SurgeryCosmoGridApplicationsMobile-Phone Electric Field EstimationLIGO/iVDGLDataAnalysis

Grid-IrelandArmagh 26-NOV-2002 Grid-Ireland WebCom-G Grid Architecture Space Core Contribution: Condensed Graphs Grid Middleware

Grid-IrelandArmagh 26-NOV-2002 Grids: 10 outstanding questions 1. Why don’t grids have more functionality yet ? 2. Why aren’t there more application developers and users ? 3. Where are the grid software development tools ? 4. Where are the grid system management tools ? 5. How do we make grids more secure ? 6. How do we define standard interfaces and definitions ? 7. How do we manage variance on the grid ? 8. How can the grid R & D be funded ? 9. What cost models are needed by the grid ? 10. What are the benefits to motivate sharing on the grid ? THESE ARE JUST THE FIRST 10 QUESTIONS THESE ARE JUST THE FIRST 10 QUESTIONS

Grid-IrelandArmagh 26-NOV-2002 Grid-Ireland THE END