1. Grid Computing Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science The University of Chicago http://www.mcs.anl.gov/~foster

2. 2 foster@mcs.anl.gov ARGONNE  CHICAGO Partial Acknowledgements l Open Grid Services Architecture design u Carl Kesselman, Karl Czajkowski @ USC/ISI u Steve Tuecke @ANL u Jeff Nick, Steve Graham, Jeff Frey @ IBM l Grid services collaborators at ANL u Kate Keahey, Gregor von Laszewski u Thomas Sandholm, Jarek Gawor, John Bresnahan l Globus Toolkit R&D also involves many fine scientists & engineers at ANL, USC/ISI, and elsewhere (see www.globus.org) l Strong links with many EU, UK, US Grid projects l Support from DOE, NASA, NSF, IBM, Microsoft

3. 3 foster@mcs.anl.gov ARGONNE  CHICAGO Goals l Communicate the purpose, significance, state, adoption, & future of Grid technology l Persuade you that Grid technology represents an opportunity u Grids aren’t (particularly) about science or servers—themes of virtualization, service discovery, service management, and QoS delivery are universal u Rapid uptake in industry & science represents an exceptional opportunity for impact

4. 4 foster@mcs.anl.gov ARGONNE  CHICAGO Overview l Origins: Resource sharing within scientific collaborations u Science drivers & science Grid projects u Globus Toolkit l Evolution: Resource virtualization u Commercial drivers u OGSA: Grid meets Web services

5. 5 foster@mcs.anl.gov ARGONNE  CHICAGO Overview l Origins: Resource sharing within scientific collaborations u Science drivers & science Grid projects u Globus Toolkit l Evolution: Resource virtualization u Commercial drivers u OGSA: Grid meets Web services

6. 6 foster@mcs.anl.gov ARGONNE  CHICAGO E-Science: The Original Grid Driver l Pre-electronic science u Theorize &/or experiment, in small teams l Post-electronic science u Construct and mine very large databases u Develop computer simulations & analyses u Access specialized devices remotely u Exchange information within distributed multidisciplinary teams  Need to manage dynamic, distributed infrastructures, services, and applications

7. 7 foster@mcs.anl.gov ARGONNE  CHICAGO www.griphyn.org/chimera Size distribution of galaxy clusters? Galaxy cluster size distribution Chimera Virtual Data System + GriPhyN Virtual Data Toolkit + iVDGL Data Grid (many CPUs) eScience Application: Sloan Digital Sky Survey Analysis

8. 8 foster@mcs.anl.gov ARGONNE  CHICAGO Lift Capabilities Drag Capabilities Responsiveness Deflection capabilities Responsiveness Thrust performance Reverse Thrust performance Responsiveness Fuel Consumption Braking performance Steering capabilities Traction Dampening capabilities Crew Capabilities - accuracy - perception - stamina - re-action times - SOPs Engine Models Airframe Models Wing Models Landing Gear Models Stabilizer Models Human Models NASA’s Information Power Grid: Aviation Safety

9. 9 foster@mcs.anl.gov ARGONNE  CHICAGO NETWORK IMAGING INSTRUMENTS COMPUTATIONAL RESOURCES LARGE DATABASES DATA ACQUISITION PROCESSING, ANALYSIS ADVANCED VISUALIZATION Life Sciences: Telemicroscopy

10. 10 foster@mcs.anl.gov ARGONNE  CHICAGO And Thus: The Grid “ Resource sharing & coordinated problem solving in dynamic, multi- institutional virtual organizations”

11. 11 foster@mcs.anl.gov ARGONNE  CHICAGO Underlying Technical Requirements l Dynamic formation and management of virtual organizations l Online negotiation of access to services: who, what, why, when, how l Configuration of applications and systems able to deliver multiple qualities of service l Autonomic management of distributed infrastructures, services, and applications

12. 12 foster@mcs.anl.gov ARGONNE  CHICAGO The Grid World: Current Status l Dozens of major Grid projects in scientific & technical computing/research & education l Open source Globus Toolkit™ a de facto standard for major protocols & services u Simple protocols & APIs for authentication, discovery, access, etc.: infrastructure u Information-centric design u Large user and developer base u Multiple commercial support providers u Enabler of numerous tools and applications l Global Grid Forum: community & standards

13. 13 foster@mcs.anl.gov ARGONNE  CHICAGO Overview l Origins: Resource sharing within scientific collaborations u Science drivers & science Grid projects u Globus Toolkit l Evolution: Resource virtualization u Commercial drivers u OGSA: Grid meets Web services

14. 14 foster@mcs.anl.gov ARGONNE  CHICAGO Resource Sharing within “VOs” is Not Unique to Science! l Fragmentation of enterprise infrastructure u Driven by cheap servers, fast nets, ubiquitous Internet, eBusiness workloads u Need to configure distributed collections of services to deliver specified QoS l Virtualization u Emerging service infrastructure, utility computing models, economies of scale u Services dynamically instantiated across device spectrum l B2B, B2C, C2C interactions

15. 15 foster@mcs.anl.gov ARGONNE  CHICAGO Virtualization and Distributed Service Management Less capable, integrated Less connected User service locus Larger, more integrated More connected Dynamically provisioned Device Continuum Resource & service aggregation Delivery of virtualized services with QoS guarantees Dynamic, secure service discovery & composition Distributed service management

16. 16 foster@mcs.anl.gov ARGONNE  CHICAGO Realizing the Promise Requires Significant Innovation l Automation of infrastructure operation to achieve economies of scale l Management and component models for service discovery, composition, provisioning l New applications and tools powered by distributed services and resources l Business and service models to support specialization of function

17. 17 foster@mcs.anl.gov ARGONNE  CHICAGO Grid Evolution: Open Grid Services Architecture l Refactor Globus protocol suite to enable common base and expose key capabilities l Service orientation to virtualize resources and unify resources/services/information l Embrace key Web services technologies: WSDL as IDL, leverage commercial efforts l Result: standard interfaces & behaviors for distributed system management

18. 18 foster@mcs.anl.gov ARGONNE  CHICAGO OGSA System Structure l A standard substrate: the Grid service u Standard interfaces and behaviors that address key distributed system issues u The “Grid Service Specification” l … supports standard service specifications u Resource management, databases, workflow, security, diagnostics, etc., etc. u Target of current & planned GGF efforts l … and arbitrary application-specific services based on these & other definitions

19. 19 foster@mcs.anl.gov ARGONNE  CHICAGO Transient Service Instances l “Web services” address discovery & invocation of persistent services u Interface to persistent state of entire enterprise l In Grids, must also support transient service instances, created/destroyed dynamically u Interfaces to the states of distributed activities u E.g. workflow, video conferencing, distributed data analysis, workload management l Significant implications for how services are named, discovered, managed, and used

20. 20 foster@mcs.anl.gov ARGONNE  CHICAGO OGSI, OGSA, and Web Services l OGSI (I = Infrastructure) u Small extensions to WSDL l Nested serviceType & serviceDataDescription u Conventions for naming service instances l Handles and references u portTypes for common behavior l Instance creation, lifetime management, introspection and monitoring, registration, notification, … l OGSA (A = Architecture) built on OGSI u A collection of Grid service interfaces l Resource description & provisioning l Higher-level services: messaging services, logging, etc.

21. 21 foster@mcs.anl.gov ARGONNE  CHICAGO The Grid Service = Interfaces/Behaviors + Service Data Service data element Service data element Service data element Implementation GridService (required) Service data access Explicit destruction Soft-state lifetime … other interfaces … (optional) Standard: - Notification - Authorization - Service creation - Service registry - Manageability - Concurrency + application- specific interfaces Binding properties: - Reliable invocation - Authentication Hosting environment/runtime (“C”, J2EE,.NET, …)

22. 22 foster@mcs.anl.gov ARGONNE  CHICAGO Service Data l A Grid service instance maintains a set of service data elements u Described in WSDL extension u XML element encapsulated in standard container: name, type, lifetime, etc. u Includes basic introspection information, interface-specific data, and application state l Pull and push models for information query u GridService::FindServiceData operation l Pull: queries this information via extensible query language u NotificationSource::SubscribeServiceData l Push: Subscribe to notification of changes to information

23. 23 foster@mcs.anl.gov ARGONNE  CHICAGO Notification Interfaces l NotificationSource for client subscription u Subscription expression describes which service data element changes are of interest u Creates a subscription manager service l Manages the lifetime and properties of subscription l NotificationSink for asynchronous delivery of notification messages l Simple, flexible base with wide variety of uses u Dynamic discovery/registry services, monitoring, application error notification, etc. u Intermediaries: filter, aggregate, archive, et.c u Can integrate commercial messaging services

24. 24 foster@mcs.anl.gov ARGONNE  CHICAGO Grid Service Example: Database Service l A DBaccess Grid service will support at least two portTypes u GridService u DBaccess l Each has service data u GridService: basic introspection information, lifetime, … u DBaccess: database type, query languages supported, current load, …, … l Maybe other portTypes as well u E.g., NotificationSource Grid Service DBaccess DB info Name, lifetime, etc.

25. 25 foster@mcs.anl.gov ARGONNE  CHICAGO Lifetime Management l GS instances created by factory or manually; destroyed explicitly or via soft state u Negotiation of initial lifetime with a factory (=service supporting Factory interface) l GridService interface supports u Destroy operation for explicit destruction u SetTerminationTime operation for keepalive l Soft state lifetime management avoids u Explicit client teardown of complex state u Resource “leaks” in hosting environments

26. 26 foster@mcs.anl.gov ARGONNE  CHICAGO Factory l Factory interface’s CreateService operation creates a new Grid service instance u Reliable creation (once-and-only-once) l CreateService operation can be extended to accept service-specific creation parameters l Returns a Grid Service Handle (GSH) u A globally unique URL, resolves to GSR u Uniquely identifies the instance for all time u Based on name of a handle resolver l Or Grid Service Reference (GSR)

27. 27 foster@mcs.anl.gov ARGONNE  CHICAGO Registration l The Registration interface may be used to register Grid service instances with a registry u A set of Grid services can periodically register their GSHs into a registry service, to allow for discovery of services in that set u Just registration of existence, not of information l Registrations maintained in the service data of the registry service u Standard discovery mechanisms can then be used to discover registered services u Service data schema may be registry specific

28. 28 foster@mcs.anl.gov ARGONNE  CHICAGO Example: Transient Database Services Grid Service DBaccess Factory Factory info Instance name, etc. Grid Service Registry Registry info Instance name, etc. Grid Service DBaccess DB info Name, lifetime, etc. Grid Service DBaccess DB info Name, lifetime, etc. “What services can you create?” “What database services exist?” “Create a database service”

29. 29 foster@mcs.anl.gov ARGONNE  CHICAGO OGSA Design & Implementation l OGSI (I=Infrastructure) WG in GGF defining core Grid service specification u (At least) three implementation efforts l Globus Toolkit => GT3 (alpha end 2002) u GT3 Core: Grid service specification u GT3 Base: Globus Toolkit behaviors u CIM resource model, GRAM-2 SLA negotiation, database services, … l Other GGF WGs address OGSA security, OGSA-compliant database services, etc.

30. 30 foster@mcs.anl.gov ARGONNE  CHICAGO Recap: Goals l Communicate the purpose, significance, state, adoption, & future of Grid technology l Persuade you that Grid technology represents a significant opportunity u Grids aren’t only (or particularly) about science and servers—themes of virtualization, service discovery, service management, and QoS delivery are universal u Rapid uptake in industry & science represents an exceptional opportunity for impact

31. 31 foster@mcs.anl.gov ARGONNE  CHICAGO For More Information l The Globus Project™ u www.globus.org l Context & research articles u www.mcs.anl.gov/~foster l Open Grid Services Architecture u www.globus.org/ogsa l Global Grid Forum u www.gridforum.org u Edinburgh, July 22-24 u Chicago, Oct 15-17

32. 32 foster@mcs.anl.gov ARGONNE  CHICAGO OGSA Implementation Hosting Environment Resource virtualization and QoS support Standard OGSI container Web services various OGSI/OGSA Interfaces service description, service provisioning, … 2) to enable virtualization via u Service description u Service provisioning OGSI/OGSA Interfaces service description, service provisioning, … 3) Standard container avoids implementing OGSI features in every service instance Standard OGSI container Hosting Environment Resource virtualization and QoS support Web services various 1) OGSA builds on infrastructure u Plumbing: WSDL, WS-Security, WS-Routing/Referral, reliable messaging, transactions, etc. u Hosting environments

33. 33 foster@mcs.anl.gov ARGONNE  CHICAGO Building an OGSI Container l Service data mgmt, query, subscription u Container should provide simple interface for interacting with an instance’s implementation to get and manage dynamic service data l Service instance = CLR object u Container should handle query processing l.NET support for XPath & Xquery allows for rich functionality u Container manages notification subscriptions, and drives asynchronous notification messages l Soft-state lifetime management l Soft-state registration