1. The Grid as Infrastructure and Application Enabler Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science The University of Chicago http://www.mcs.anl.gov/~foster

2. 2 foster@mcs.anl.gov ARGONNE  CHICAGO The Grid “ Resource sharing & coordinated problem solving in dynamic, multi- institutional virtual organizations”

3. 3 foster@mcs.anl.gov ARGONNE  CHICAGO l Resources –Computing, storage, data l Communities –Operational procedures, … Grid Infrastructure A A A l Services –Authentication, discovery, … l Connectivity –Reduce tyranny of distance l Technologies –Build services & applications

4. 4 foster@mcs.anl.gov ARGONNE  CHICAGO The Grid World: Current Status l Dozens of major Grid projects in scientific & technical computing/research & education –Compute-intensive, data-intensive, remote instrumentation, collaboration, … l Open source Globus Toolkit™ a de facto standard for major protocols & services –Supporting many tools & applns in data- intensive and collaboration-intensive science l Major investments in physical infrastructure l Global Grid Forum: community & standards

5. 5 foster@mcs.anl.gov ARGONNE  CHICAGO Examples of Emerging Grid Infrastructure l iVDGL: Data-intensive infrastructure –Building a (international) community l Data Grid middleware –Chimera virtual data system l Open Grid Services Architecture –Future service & technology infrastructure

6. 6 foster@mcs.anl.gov ARGONNE  CHICAGO iVDGL: A Global Grid Laboratory l International Virtual-Data Grid Laboratory –A global Grid laboratory (US, Europe, Asia, South America, …) –A place to conduct Data Grid tests “at scale” –A mechanism to create common Grid infrastructure –A laboratory for other disciplines to perform Data Grid tests –A focus of outreach efforts to small institutions l U.S. part funded by NSF (2001-2006) –$13.7M (NSF) + $2M (matching) “We propose to create, operate and evaluate, over a sustained period of time, an international research laboratory for data-intensive science.” From NSF proposal, 2001

7. 7 foster@mcs.anl.gov ARGONNE  CHICAGO Initial US-iVDGL Data Grid Tier1 (FNAL) Proto-Tier2 Tier3 university UCSD Florida Wisconsin Fermilab BNL Indiana BU Other sites to be added in 2002 SKC Brownsville Hampton PSU JHU Caltech

8. 8 foster@mcs.anl.gov ARGONNE  CHICAGO U.S. PIs: Avery, Foster, Gardner, Newman, Szalay www.ivdgl.org iVDGL: International Virtual Data Grid Laboratory Tier0/1 facility Tier2 facility 10 Gbps link 2.5 Gbps link 622 Mbps link Other link Tier3 facility

9. 9 foster@mcs.anl.gov ARGONNE  CHICAGO Grid Evolution: Open Grid Services Architecture l Refactor Globus protocol suite to enable common base and expose key capabilities –Secure, reliable invocation; service info; notification; soft state lifetime mgmt; … l Service orientation to virtualize resources and unify resources/services/information –Standard IDL for encapsulation l Embrace key Web services technologies: WSDL as IDL, leverage commercial efforts –And WS Security, WS Routing, etc.

10. 10 foster@mcs.anl.gov ARGONNE  CHICAGO OGSA Structure l A standard substrate: the Grid service –Standard interfaces and behaviors that address key distributed system issues –The “Grid Service Specification” l … supports standard service specifications –Resource management, databases, workflow, security, diagnostics, etc., etc. –Target of current & planned GGF efforts l … and arbitrary application-specific services based on these & other definitions

11. 11 foster@mcs.anl.gov ARGONNE  CHICAGO OGSA Status l Grid service spec near completion in GGF –Globus Toolkit implementation available –IBM & Fujitsu implementations underway –Other companies committed to support it l Various higher-level services underway –OGSI-based Globus Toolkit v3 (GT3), will support GT2 interfaces by end of 2002 –Database services (UK eScience program) –Resource information & management (CIM) –Etc., etc.

12. 12 foster@mcs.anl.gov ARGONNE  CHICAGO Programs as Community Resources: Data Derivation and Provenance l Most [scientific] data are not simple “measurements”; essentially all are: –Computationally corrected/reconstructed –And/or produced by numerical simulation l And thus, as data and computers become ever larger and more expensive: –Programs are significant community resources –So are the executions of those programs l A virtual data system provides a unified view of data, programs, and executions

13. 13 foster@mcs.anl.gov ARGONNE  CHICAGO TransformationDerivation Data created-by execution-of consumed-by/ generated-by “I’ve detected a calibration error in an instrument and want to know which derived data to recompute.” “I’ve come across some interesting data, but I need to understand the nature of the corrections applied when it was constructed before I can trust it for my purposes.” “I want to search an astronomical database for galaxies with certain characteristics. If a program that performs this analysis exists, I won’t have to write one from scratch.” “I want to apply an astronomical analysis program to millions of objects. If the results already exist, I’ll save weeks of computation.” Virtual Data

14. 14 foster@mcs.anl.gov ARGONNE  CHICAGO l Virtual data catalog –Transformations, derivations, data l Virtual data language –Data definition + query l Applications include browsers and data analysis applications Chimera Virtual Data System (www.griphyn.org/chimera) Joint work with Jens Vöckler, Mike Wilde, Yong Zhao GriPhyN VDT: Replica catalog DAGMan Globus Toolkit Etc.

15. 15 foster@mcs.anl.gov ARGONNE  CHICAGO Joint work with Jim Annis, Steve Kent, FNAL Size distribution of galaxy clusters? Galaxy cluster size distribution Chimera Virtual Data System + GriPhyN Virtual Data Toolkit + iVDGL Data Grid (many CPUs) Chimera Application: Sloan Digital Sky Survey Analysis

16. 16 foster@mcs.anl.gov ARGONNE  CHICAGO Summary l “Resource sharing, coordinated problem solving in dynamic, multi-institutional virtual orgs” –Adoption in eScience, transitioning to industry l Emerging physical infrastructure –TeraGrid, iVDGL, DOE Science Grid, …, … l Open Grid Services Architecture –Integrated treatment of major Grid issues –Uniform treatment of resources, data, services l Chimera virtual data system –New abstractions for application development

17. 17 foster@mcs.anl.gov ARGONNE  CHICAGO l The Globus Project™ –www.globus.org l Technical articles –www.mcs.anl.gov/~foster l TeraGrid, iVDGL –www.teragrid.org, www.ivdgl.org l Open Grid Services Arch. –www.globus.org/ogsa l Chimera –www.griphyn.org/chimera l Global Grid Forum –www.gridforum.org For More Information