1. Open Grid Services as an Enabler of Future Networked Applications Ian Foster Argonne National Laboratory University of Chicago http://www.mcs.anl.gov/~foster TERENA Networking Conference, May 20, 2003

2. 2 foster@mcs.anl.gov ARGONNE  CHICAGO Abstract Grid technologies and infrastructure are designed to support the integration of services and resources within and among enterprises, and thus to allow active collaborations across distributed, multi- organizational collaborations. Recent progress on the Open Grid Services Architecture (OGSA), which integrates Grid technologies with emerging Web services standards, is enabling broad adoption and deployment. I describe the current state and likely evolution of OGSA, and discuss implications for infrastructure and applications.

3. 3 foster@mcs.anl.gov ARGONNE  CHICAGO Partial Acknowledgements l Open Grid Services Architecture design –Carl Kesselman, Karl Czajkowski @ USC/ISI –Steve Tuecke @ANL –Jeff Nick, Steve Graham, Jeff Frey @ IBM l Grid services collaborators at ANL –Kate Keahey, Gregor von Laszewski –Thomas Sandholm, Jarek Gawor, John Bresnahan l Globus Toolkit R&D also involves many fine scientists & engineers at ANL, USC/ISI, and elsewhere (see www.globus.org) l Strong links with many EU, UK, US Grid projects l Support from DOE, NASA, NSF, IBM, Microsoft

4. 4 foster@mcs.anl.gov ARGONNE  CHICAGO Overview l Problem solving in the 21 st century l Open Grid Services Architecture l Globus Toolkit v3 l Summary

5. 5 foster@mcs.anl.gov ARGONNE  CHICAGO Problem Solving in the 21 st Century l Teams organized around common goals –Communities: “Virtual organizations” l With diverse membership & capabilities –Heterogeneity is a strength not a weakness l And geographic and political distribution –No location/organization possesses all required skills and resources l Must adapt as a function of the situation –Adjust membership, reallocate responsibilities, renegotiate resources

6. 6 foster@mcs.anl.gov ARGONNE  CHICAGO For Example: Global Knowledge Communities

7. 7 foster@mcs.anl.gov ARGONNE  CHICAGO New Opportunities Demand New Technology “ Resource sharing & coordinated problem solving in dynamic, multi- institutional virtual organizations” “When the network is as fast as the computer's internal links, the machine disintegrates across the net into a set of special purpose appliances” (George Gilder)

8. 8 foster@mcs.anl.gov ARGONNE  CHICAGO Taking Sharing to the Next Level l Sharing of communication –Telephones, mailing lists, collaboration tools l Sharing of data and knowledge –Web, semantic web l What about the rest of the infrastructure? –Services, computers, programs, sensors, …

9. 9 foster@mcs.anl.gov ARGONNE  CHICAGO Existing Technologies are Helpful, but Not Complete Solutions l Peer-to-peer technologies –Limited scope and mechanisms l Enterprise-level distributed computing –Limited cross-organizational support l Databases –Vertically integrated solutions l Web services –Not dynamic l Semantic web –Limited focus

10. 10 foster@mcs.anl.gov ARGONNE  CHICAGO What’s Missing is Support for … l Sharing & integration of resources, via –Discovery –Provisioning –Access (computation, data, …) –Security –Policy –Fault tolerance –Management l In dynamic, scalable, multi-organizational settings

11. 11 foster@mcs.anl.gov ARGONNE  CHICAGO Enter the Grid l Infrastructure (“middleware”) for establishing, managing, and evolving multi-organizational federations –Dynamic, autonomous, domain independent –On-demand, ubiquitous access to computing, data, and services l Mechanisms for creating and managing workflow within such federations –New capabilities constructed dynamically and transparently from distributed services –Service-oriented, virtualization

12. 12 foster@mcs.anl.gov ARGONNE  CHICAGO Increased functionality, standardization Custom solutions 1990199520002005 Open Grid Services Arch Real standards Multiple implementations Web services, etc. Managed shared virtual systems Computer science research Globus Toolkit Defacto standard Single implementation Internet standards The Emergence of Open Grid Standards 2010

13. 13 foster@mcs.anl.gov ARGONNE  CHICAGO Overview l Problem solving in the 21 st century l Open Grid Services Architecture l Globus Toolkit v3 l Summary

14. 14 foster@mcs.anl.gov ARGONNE  CHICAGO Open Grid Services Architecture l Service-oriented architecture –Key to virtualization, discovery, composition, local-remote transparency l Leverage industry standards –Internet, Web services l Distributed service management –A “component model for Web services” l A framework for the definition of composable, interoperable services “The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 2002

15. 15 foster@mcs.anl.gov ARGONNE  CHICAGO Web Services l XML-based distributed computing technology l Web service = a server process that exposes typed ports to the network l Described by the Web Services Description Language, an XML document that contains –Type of message(s) the service understands & types of responses & exceptions it returns –“Methods” bound together as “port types” –Port types bound to protocols as “ports” l A WSDL document completely defines a service and how to access it

16. 16 foster@mcs.anl.gov ARGONNE  CHICAGO OGSA Structure l A standard substrate: the Grid service –Standard interfaces and behaviors that address key distributed system issues –A refactoring and extension of the Globus Toolkit protocol suite l … supports standard service specifications –Resource management, databases, workflow, security, diagnostics, etc., etc. –Target of current & planned GGF efforts l … and arbitrary application-specific services based on these & other definitions

17. 17 foster@mcs.anl.gov ARGONNE  CHICAGO Open Grid Services Infrastructure Implementation Service data element Other standard interfaces: factory, notification, collections Hosting environment/runtime (“C”, J2EE,.NET, …) Service data element Service data element GridService (required) Data access Lifetime management Explicit destruction Soft-state lifetime Introspection: What port types? What policy? What state? Client Grid Service Handle Grid Service Reference handle resolution

18. 18 foster@mcs.anl.gov ARGONNE  CHICAGO Open Grid Services Infrastructure GWD-R (draft-ggf-ogsi- gridservice-23) Editors: Open Grid Services Infrastructure (OGSI) S. Tuecke, ANL http://www.ggf.org/ogsi-wg K. Czajkowski, USC/ISI I. Foster, ANL J. Frey, IBM S. Graham, IBM C. Kesselman, USC/ISI D. Snelling, Fujitsu Labs P. Vanderbilt, NASA February 17, 2003 Open Grid Services Infrastructure (OGSI)

19. 19 foster@mcs.anl.gov ARGONNE  CHICAGO Example: Reliable File Transfer Service Performance Policy Faults service data elements Pending File Transfer Internal State Grid Service Notf’n Source Policy interfaces Query &/or subscribe to service data Fault Monitor Perf. Monitor Client Request and manage file transfer operations Data transfer operations

20. 20 foster@mcs.anl.gov ARGONNE  CHICAGO Open Grid Services Architecture Open Grid Services Infrastructure OGSA services: registry, authorization, monitoring, data access, management, etc., etc. Transport Protocol Hosting Environment Host. Env. & Protocol Bindings OGSA schemas More specialized & domain-specific services Other schemas Web Services GWD-R (draft-ggf-ogsa-platform-3) Editors: Open Grid Services Architecture Platform I. Foster, Argonne & U.Chicago http://www.ggf.org/ogsa-wg D. Gannon, Indiana U. l Data access and integration l Security l SLA negotiation l Manageability l …

21. 21 foster@mcs.anl.gov ARGONNE  CHICAGO Grid Security Services Requestor Application VO Domain Credential Validation Service Authorization Service Requestor's Domain Service Provider's Domain Audit/ Secure-Logging Service Attribute Service Trust Service Provider Application Bridge/ Translation Service Privacy Service Credential Validation Service Authorization Service Audit/ Secure-Logging Service Attribute Service Trust Service Privacy Service Credential Validation Service Authorization Service Attribute Service Trust Service Credential Validation Service Authorization Service Attribute Service Trust Service WS-Stub Secure Conversation

22. 22 foster@mcs.anl.gov ARGONNE  CHICAGO OGSA Manageability l Management is a high-priority OGSA effort –GGF Common Management Model (CMM) WG –Ellen Stokes (IBM) co-chair l Goal: –Define standard schema and interfaces for a manageable resource (modeled as a service) l And also: –Allow existing models to be used & exposed –Leverage CIM schema when applicable –Define how CMM interfaces can integrate with higher-level interfaces (e.g., provisioning)

23. 23 foster@mcs.anl.gov ARGONNE  CHICAGO Common Management Model l A manageable resource is a Grid service, thus –Global resource names: Grid service handles –State data modeling + access: SDEs –Lifetime management –Service Group for grouping resources –Interface definition language: WSDL l Plus additional schema & operations –Standard manageable resource SDE schema –Interfaces for extensible lifecycle and relationship management  BaseManageableResource interface

24. 24 foster@mcs.anl.gov ARGONNE  CHICAGO Base Manageable Port Types GridService LocateRelationship LifecycleModel BaseManageableResource ServiceGroupHandleResolver OGSA port types CRM port types

25. 25 foster@mcs.anl.gov ARGONNE  CHICAGO CMM Schema l WSDL (open content model) & XSD describe resource’s manageable attributes (as SDEs) l Models are CIM-based where applicable –Re-factored for service efficiency; appropriate for higher-level management applications –Additional XML attributes: change control, measuring, lifecycle –Additional XML data types l CIM as basis for schemas: but may modify –E.g. use constructs from XML/XSD where similar ones exist in CIM

26. 26 foster@mcs.anl.gov ARGONNE  CHICAGO “Use CIM Models Where Applicable” l Class is port type, properties of class are port type service data, methods of class are port type operations –Some refactoring of classes -> portTypes l Express in WSDL/GSDL as grid service –Managed resource port type from which other resource port types are derived –Mix in the base GridService port type –Mix in other CMM port types as needed (Identity, relationship, lifecycleState)

27. 27 foster@mcs.anl.gov ARGONNE  CHICAGO OGSA Misconceptions l OGSA means you have to code in Java –No: C client bindings now, C server side eventually (but not needed for current apps) l OGSA means all programs must be services –No: You can write services if you want, but other behaviors are supported: e.g., GT3 supports GT2 GRAM, GridFTP, …, … l OGSA is a silver bullet for distributed and collaborative computing –No, it makes some things easier, but it’s only interfaces and behaviors, after all!

28. 28 foster@mcs.anl.gov ARGONNE  CHICAGO Overview l Problem solving in the 21 st century l Open Grid Services Architecture l Globus Toolkit v3 l Summary

29. 29 foster@mcs.anl.gov ARGONNE  CHICAGO Globus Toolkit v3 (GT3) Open Source OGSA Technology l Implements OGSI interfaces l Supports primary GT2 interfaces –GRAM, GridFTP, GSI –High degree of backward compatibility l Multiple platforms & hosting environments –J2EE, Java, C,.NET, Python l New services –SLA negotiation, service registry, community authorization, data management, … l Broad & growing adoption and contributions

30. 30 foster@mcs.anl.gov ARGONNE  CHICAGO Globus Toolkit Contributors: GT2 l Grid Packaging Technology (GPT) NCSA l Persistent GRAM Jobmanager Condor l GSI/Kerberos interchangeability Sandia l Documentation NASA, NCSA l Ports IBM, HP, Sun, SDSC, … l MDS stress testing EU DataGrid l Support IBM, Platform, UK eScience l Testing and patches Many! l Interoperable tools Many! l $$ DARPA, DOE, NSF, NASA, Microsoft, EU

31. 31 foster@mcs.anl.gov ARGONNE  CHICAGO Globus Toolkit Contributors: GT3 l Replica location service EU DataGrid l Python hosting environment LBNL l Data access & integration UK eScience l Data mediation services SDSC l Tooling, Xindice, JMS IBM l...

32. 32 foster@mcs.anl.gov ARGONNE  CHICAGO GT2 Evolution To GT3 l What happened to the GT2 key protocols? –Security: Adapting X.509 proxy certs to integrate with emerging WS standards –GRIP/LDAP/MDS: Abstractions integrated into OGSI as serviceData –GRAM: ManagedJobFactory and related service definitions –GridFTP: Unchanged in 3.0, but will evolve into OGSI-compliant service in 2003 l Also rendering collective services in terms of OGSI: RFT, RLS, CAS, etc.

33. 33 foster@mcs.anl.gov ARGONNE  CHICAGO Example: Reliable File Transfer Service Performance Policy Faults service data elements Pending File Transfer Internal State Grid Service Notf’n Source Policy interfaces Query &/or subscribe to service data Fault Monitor Perf. Monitor Client Request and manage file transfer operations Data transfer operations

34. 34 foster@mcs.anl.gov ARGONNE  CHICAGO The Grid Technology Repository l Community repository l Clearing house for service definitions, code, documentation l Encourage collaboration & avoid redundant work http://gtr.globus.org International advisory committee: Ian Foster (Chair), Malcolm Atkinson, John Brooke, Fabrizio Gagliardi, Dennis Gannon, Wolfgang Gentzsch, Andrew Grimshaw, Keith Jackson, Gregor von Laszewski, Satoshi Matsuoka, Jarek Nabrzyski, Bill St. Arnaud, Jay Unger

35. 35 foster@mcs.anl.gov ARGONNE  CHICAGO Overview l Problem solving in the 21 st century l Open Grid Services Architecture l Globus Toolkit v3 l Summary

36. 36 foster@mcs.anl.gov ARGONNE  CHICAGO The Need for Open Infrastructure l Broadly deployed services in support of fundamental collaborative activities –Formation & operation of virtual organizations –Authentication, authorization, discovery, … l Services, software, and policies enabling on- demand access to critical resources –Computers, databases, networks, storage, software services,… l Operational support for 24x7 availability l Integration with campus and commercial infrastructures

37. 37 foster@mcs.anl.gov ARGONNE  CHICAGO Tier0/1 facility Tier2 facility 10 Gbps link 2.5 Gbps link 622 Mbps link Other link Tier3 facility Open Infrastructure Cambridge Newcastle Edinburgh Oxford Glasgow Manchester Cardiff Soton London Belfast DL RAL Hinxton

38. 38 foster@mcs.anl.gov ARGONNE  CHICAGO Summary l OGSA: standards-based dist. sys. middleware –From Web services: standard IDL, discovery, binding independence, other desirable features –From Grid/Globus Toolkit: naming, state, lifetime management, etc., etc. l Rapid progress on definition & implementation –OGSI is defined, GT3 implements it (and other things), multiple groups coding to it –Much more happening, much more to be done! l No silver bullet, but a good incremental step towards meeting user requirements

39. 39 foster@mcs.anl.gov ARGONNE  CHICAGO l Open Grid Services Arch. –www.ggf.org/ogsa-wg –www.globus.org/ogsa l Global Grid Forum –www.ggf.org l The Globus Project™ –www.globus.org l Technical articles –www.mcs.anl.gov/~foster For More Information 2 nd Edition to appear November 2003