1. Ian Foster Argonne National Laboratory University of Chicago http://www.mcs.anl.gov/~foster Open Grid Services Architecture Plenary Talk at CHEP 2003, San Diego, March 26, 2003

2. 2 foster@mcs.anl.gov ARGONNE  CHICAGO Building an Open Grid Open Standards Open Source Open Infrastructure Open Grid

3. 3 foster@mcs.anl.gov ARGONNE  CHICAGO Grids and Open Standards Increased functionality, standardization Time Custom solutions Open Grid Services Arch GGF: OGSI, … (+ OASIS, W3C) Multiple implementations, including Globus Toolkit Web services Globus Toolkit Defacto standards GGF: GridFTP, GSI X.509, LDAP, FTP, … App-specific Services

4. 4 foster@mcs.anl.gov ARGONNE  CHICAGO Layered Grid Architecture Application Fabric “Controlling things locally”: Access to, & control of, resources Connectivity “Talking to things”: communication (Internet protocols) & security Resource “Sharing single resources”: negotiating access, controlling use Collective “Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services Internet Transport Application Link Internet Protocol Architecture “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”, Foster, Kesselman, Tuecke, Intl J. High Performance Computing Applications, 15(3), 2001.

5. 5 foster@mcs.anl.gov ARGONNE  CHICAGO Globus Toolkit v2 l Four key protocols and APIs –Grid Security Infrastructure (GSI) –Grid Resource Allocation & Mgmt (GRAM) –Grid Resource Information Protocol (GRIP) and Index Information Protocol (GIIP) –Grid File Transfer Protocol (GridFTP) l Implementations on many platforms –Resources, security systems, data models,… l Various collective layer protocols & tools –Info services, replica management, etc. l A basis for many Grid-enabled tools & apps –FTP, SSH, Condor, SRB, MPI, EDG, GridPort, …

6. 6 foster@mcs.anl.gov ARGONNE  CHICAGO From Resources to Services: Managing Virtual Services l Trying to manage total system properties –E.g. Dependability, end-to-end QoS l “Resource” tends to connote a tangible entity to be consumed: CPU, storage, bandwidth, … l But many interesting services may be decoupled from any particular resource –E.g. virtual data service, data analysis service –A service consumes resources, but how that happens is irrelevant to the client l “Service” forms a better base abstraction –Can apply to physical or virtual

7. 7 foster@mcs.anl.gov ARGONNE  CHICAGO Open Grid Services Architecture l Service-oriented architecture –Key to virtualization, discovery, composition, local-remote transparency l Leverage industry standards –Internet, Web services l Distributed service management –A “component model for Web services” (or: a “service model for the Grid”) l A framework for the definition of composable, interoperable services “The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 2002

8. 8 foster@mcs.anl.gov ARGONNE  CHICAGO Web Services l A simple but powerful distributed system paradigm, that allows one to: –Describe a service (WSDL) –Invoke a service (SOAP) –Discover a service (various) l Web services appears to offer a fighting chance at ubiquity (unlike CORBA) –Sophisticated tools emerging from industry l But Web services does not go far enough to serve a common base for the Grid …

9. 9 foster@mcs.anl.gov ARGONNE  CHICAGO Transient Service Instances l “Web services” address discovery & invocation of persistent services –Interface to persistent state of entire enterprise l In Grids, must also support transient service instances, created/destroyed dynamically –Interfaces to the states of distributed activities –E.g. workflow, video conf., dist. data analysis l Significant implications for how services are managed, named, discovered, and used –In fact, much of Grid is concerned with the management of service instances

10. 10 foster@mcs.anl.gov ARGONNE  CHICAGO OGSA Structure l A standard substrate: the Grid service –Standard interfaces and behaviors that address key distributed system issues –A refactoring and extension of the Globus Toolkit protocol suite l … supports standard service specifications –Resource management, databases, workflow, security, diagnostics, etc., etc. –Target of current & planned GGF efforts l … and arbitrary application-specific services based on these & other definitions

11. 11 foster@mcs.anl.gov ARGONNE  CHICAGO Open Grid Services Infrastructure Implementation Hosting environment/runtime (“C”, J2EE,.NET, …) Data access

12. 12 foster@mcs.anl.gov ARGONNE  CHICAGO Open Grid Services Infrastructure Implementation Hosting environment/runtime (“C”, J2EE,.NET, …) Data access Grid Service Handle Grid Service Reference handle resolution

13. 13 foster@mcs.anl.gov ARGONNE  CHICAGO Open Grid Services Infrastructure Implementation Hosting environment/runtime (“C”, J2EE,.NET, …) Service data element Service data element GridService (required) Data access Grid Service Handle Grid Service Reference handle resolution

14. 14 foster@mcs.anl.gov ARGONNE  CHICAGO Open Grid Services Infrastructure Implementation Hosting environment/runtime (“C”, J2EE,.NET, …) Service data element Service data element GridService (required) Data access Lifetime management Explicit destruction Soft-state lifetime Introspection: What port types? What policy? What state? Client Grid Service Handle Grid Service Reference handle resolution

15. 15 foster@mcs.anl.gov ARGONNE  CHICAGO Open Grid Services Infrastructure Implementation Service data element Other standard interfaces: factory, notification, collections Hosting environment/runtime (“C”, J2EE,.NET, …) Service data element Service data element GridService (required) Data access Lifetime management Explicit destruction Soft-state lifetime Introspection: What port types? What policy? What state? Client Grid Service Handle Grid Service Reference handle resolution

16. 16 foster@mcs.anl.gov ARGONNE  CHICAGO Open Grid Services Infrastructure GWD-R (draft-ggf-ogsi- gridservice-23) Editors: Open Grid Services Infrastructure (OGSI) S. Tuecke, ANL http://www.ggf.org/ogsi-wg K. Czajkowski, USC/ISI I. Foster, ANL J. Frey, IBM S. Graham, IBM C. Kesselman, USC/ISI D. Snelling, Fujitsu Labs P. Vanderbilt, NASA February 17, 2003 Open Grid Services Infrastructure (OGSI)

17. 17 foster@mcs.anl.gov ARGONNE  CHICAGO Realizing a Service-Oriented Architecture: How Do I l Create, name, manage, discover services? l Render resources, data, sensors as services? l Negotiate service level agreements? l Express & negotiate policy? l Organize & manage service collections? l Establish identity, negotiate authentication? l Manage VO membership & communication? l Compose services efficiently? l Achieve interoperability?

18. 18 foster@mcs.anl.gov ARGONNE  CHICAGO The OGSA Platform OGSI Transport Protocol Hosting Environment Host. Env. & Protocol Bindings OGSA Platform services: registry, authorization, monitoring, data access, etc., etc. More specialized & domain-specific services Models for resources & other entities Other models Environment- specific profiles Domain- specific profiles OGSA Platform GWD-R (draft-ggf-ogsa-platform-3) Editors: Open Grid Services Architecture Platform I. Foster, Argonne & U.Chicago http://www.ggf.org/ogsa-wg D. Gannon, Indiana U.

19. 19 foster@mcs.anl.gov ARGONNE  CHICAGO OGSA Definition Activities (Underway or Pending) l Data Access and Integration l Data Replication l Security l SLA Negotiation l Common Management Model l And others…

20. 20 foster@mcs.anl.gov ARGONNE  CHICAGO Open Grid Service Architecture: Next Steps Technical specifications –Open Grid Services Infrastructure is complete –Security, data access, Java binding, common management models, etc., in the pipeline Implementations and compliant products –OGSA-based Globus Toolkit v3, pyGlobus, … –IBM, Avaki, Platform, Sun, NEC, Oracle, …  Rich set of service defns & implementations –Time to start on OGSI-compliant services!

21. 21 foster@mcs.anl.gov ARGONNE  CHICAGO Example: Reliable File Transfer Service Performance Policy Faults service data elements Pending File Transfer Internal State Grid Service Notf’n Source Policy interfaces Query &/or subscribe to service data Fault Monitor Perf. Monitor Client Request and manage file transfer operations Data transfer operations

22. 22 foster@mcs.anl.gov ARGONNE  CHICAGO Globus Toolkit v3 (GT3) Open Source OGSA Technology l Implement core OGSI interfaces l Support primary GT2 interfaces –High degree of backward compatibility l Multiple platforms & hosting environments –J2EE, Java, C,.NET, Python l New services –SLA negotiation (GRAM-2), registry, replica location, community authorization, data, … l Growing external contributions & adoption

23. 23 foster@mcs.anl.gov ARGONNE  CHICAGO GT2 Evolution To GT3 l What happened to the GT2 key protocols? –Security: Adapting X.509 proxy certs to integrate with emerging WS standards –GRIP/LDAP/MDS: Abstractions integrated into OGSI as serviceData –GRAM: ManagedJobFactory and related service definitions –GridFTP: Unchanged in 3.0, but will evolve into OGSI-compliant service in 2003 l Also rendering collective services in terms of OGSI: RFT, RLS, CAS, etc.

24. 24 foster@mcs.anl.gov ARGONNE  CHICAGO GT Timeline l GT 1.0:1998 –GRAM, MDS l GT 2.0:2001 –GridFTP, packaging, reliability l GT3 Technology Preview:Apr-Dec 2002 –Tracking OGSI definition l GT3.0 Alpha:Jan 2003 –OGSI Base, GT2 functionality l GT3.0 Production:June 2003 –Tested, documented, etc.

25. 25 foster@mcs.anl.gov ARGONNE  CHICAGO Globus Toolkit Contributors: GT2 l Grid Packaging Technology (GPT) NCSA l Persistent GRAM Jobmanager Condor l GSI/Kerberos interchangeability Sandia l Documentation NASA, NCSA l Ports IBM, HP, Sun, SDSC, … l MDS stress testing EU DataGrid l Support IBM, Platform, UK eScience l Testing and patches Many! l Interoperable tools Many! l $$ DARPA, DOE, NSF, NASA, Microsoft, EU

26. 26 foster@mcs.anl.gov ARGONNE  CHICAGO Globus Toolkit Contributors: GT3 l Replica location service EU DataGrid l Python hosting environment LBNL l Data access & integration UK eScience l Data mediation services SDSC l Tooling, Xindice, JMS IBM l...

27. 27 foster@mcs.anl.gov ARGONNE  CHICAGO The Grid Technology Repository l Community repository l Clearing house for service definitions, code, documentation l Encourage collaboration & avoid redundant work http://gtr.globus.org International advisory committee: Ian Foster (Chair), Malcolm Atkinson, John Brooke, Fabrizio Gagliardi, Dennis Gannon, Wolfgang Gentzsch, Andrew Grimshaw, Keith Jackson, Gregor von Laszewski, Satoshi Matsuoka, Jarek Nabrzyski, Bill St. Arnaud, Jay Unger

28. 28 foster@mcs.anl.gov ARGONNE  CHICAGO GT Processes Are Evolving Rapidly (Thanks to Partners & the GT Team!) l Before 2000 –Email-based problem tracking, aka “req” l 2000 –Detailed documentation, release notes (Q1) –Legal framework for external contributions (Q1) l 2001 –Packaging; module & binary releases (Q4) –Substantial regression tests (Q4) l 2002 –Bugzilla problem reporting & tracking (Q2) –Processes for external contrib (Q2) –Distributed testing infrastructure (Q3) l 2003 (in progress) –Distributed support infrastructure: GT “support centers” –Standardized Grid testing framework(s) –GT “contrib” components –Grid Technology Repository

29. 29 foster@mcs.anl.gov ARGONNE  CHICAGO OGSA Misconceptions l OGSA means you have to code in Java –No: C client bindings now, C server side eventually (but not needed for current apps) l OGSA means all programs must be services –No: You can write services if you want, but GT2-style GRAM behavior is still supported (GRAM is just a server) l OGSA is a silver bullet for Grid computing –No, it makes some things easier, but it’s only interfaces and behaviors, after all!

30. 30 foster@mcs.anl.gov ARGONNE  CHICAGO Summary l OGSA: standards-based Grid technology –From Web services: standard IDL, discovery, binding independence, other desirable features –From Grid: naming, state, lifetime management, etc., etc. l Rapid progress on definition & implementation –OGSI is defined, GT3 implements it (and other things), multiple groups coding to it –Much more happening, much more to be done! l No silver bullet, but a good incremental step forward to our ultimate Grid software goals