1. Ian Foster Argonne National Laboratory University of Chicago Univa Corporation Grid Dynamics

2. 2 Acknowledgements l Carl Kesselman, with whom I developed many ideas (& slides) l Bill Allcock, Charlie Catlett, Kate Keahey, Jennifer Schopf, Frank Siebenlist, Mike Wilde @ ANL/UC l Ann Chervenak, Ewa Deelman, Laura Pearlman @ USC/ISI l Karl Czajkowski, Steve Tuecke @ Univa l Numerous other fine colleagues in NESC, EGEE, OSG, TeraGrid, etc. l NSF & DOE for research support

3. 3 ? What is the Grid? “ Resource sharing & coordinated problem solving in dynamic, multi- institutional virtual organizations” “When the network is as fast as the computer's internal links, the machine disintegrates across the net into a set of special purpose appliances” (George Gilder) “The Anatomy of the Grid”, Foster, Kesselman, Tuecke, 2001

4. Facilities Computers Storage Networks Services Software People Implementation System-Level Problem Grid technology Decomposition U. Colorado Experimental Model NCSA Computational Model COORD. UIUC Experimental Model

5. 5 Dispatcher Grid backend Grid-enabled Business Intelligence Application Provision New Worker Process BI Server 2 BI server applications started and decommissioned by a Grid-enabled dispatcher Managed Pool of Shared Resources

6. 6 Grid Dynamics: Vision vs. Reality l Vision: On-demand access to computing u New communities form easily u On-demand resources from providers u Adapt easily to new missions, requirements l Reality: Much manual configuration, e.g.: u Manually deployed services on dedicated hardware u Manually maintained access control lists u Sysadmin-maintained allocation policies u Human-mediated resource reservation

7. 7 Grid Dynamics: A Two-Dimensional Problem l Decompose across network Clients integrate dynamically u Select & compose services u Select “best of breed” providers u Publish result as new services l Decouple resource & service providers Function Resource Data Archives Analysis tools Discovery tools Users Fig: S. G. Djorgovski

8. 8 Provisioning Service-Oriented Systems: The Role of Grid Infrastructure l Service-oriented Grid infrastructure u Provision physical resources to support application workloads Appln Service Users Workflows Composition Invocation l Service-oriented applications u Wrap applications as services u Compose applications into workflows “The Many Faces of IT as Service”, ACM Queue, Foster, Tuecke, 2005

9. 9 Grid Dynamics: Forming & Operating Communities l Define membership & roles; enforce laws & community standards u I.e., policy for service-oriented architecture u Addressing dynamic membership & policy l Build, buy, operate, & share infrastructure u Decouple consumer & provider u For data, programs, services, computing, storage, instruments u Address dynamics of community demand

10. 10 Grid Dynamics: Forming & Operating Communities l Define membership & roles; enforce laws & community standards u I.e., policy for service-oriented architecture u Addressing dynamic membership & policy l Build, buy, operate, & share infrastructure u Decouple consumer & provider u For data, programs, services, computing, storage, instruments u Address dynamics of community demand

11. 11 Defining Community: Membership and Laws l Identify VO participants and roles u For people and services l Specify and control actions of members u Empower members  delegation u Enforce restrictions  federate policy A 12 B 12 A B 1 10 1 1 16 Access granted by community to user Site admission- control policies Effective Access Policy of site to community

12. 12 Policy Challenges in VOs l Restrict VO operations based on requestor characteristics u VO dynamics create challenges l Intra-VO u VO-specific roles u Mechanisms to specify/enforce VO-level policy l Inter-VO u Different VOs define different entities/roles l Different sorts of policy need to be enforced u Access, usage, accounting, audit, …

13. 13 Evolution of Grid Security & Policy 1) Grid security infrastructure u Public key authentication & delegation u Access control lists (“gridmap” files)  Limited set of policies can be expressed 2) Utilities to simplify operational use, e.g. u MyProxy: online credential repository u VOMS, ACL/gridmap management  Broader set of policies, but still ad-hoc 3) General, standards-based framework for authorization & attribute management

14. 14 Core Security Mechanisms l Attribute Assertions u C asserts that S has attribute A with value V l Authentication and digital signature u Allows signer to assert attributes l Delegation u C asserts that S can perform O on behalf of C l Attribute mapping u {A1, A2… An}vo1  {A’1, A’2… A’m}vo2 l Policy u Entity with attributes A asserted by C may perform operation O on resource R

15. 15 Security Services for VO Policy l Attribute Authority (ATA) u Issue signed attribute assertions (incl. identity, delegation & mapping) l Authorization Authority (AZA) u Decisions based on assertions & policy VO A Service VO ATA VO AZA Mapping ATA VO B Service VO User A Delegation Assertion User B can use Service A VO-A Attr  VO-B Attr VO User B Resource Admin Attribute VO Member Attribute VO Member Attribute

16. 16 Trust in VOs l Do I “believe” an attribute assertion? u Used to evaluate cost vs. benefit of performing an operation u E.g., perform untrusted operation with extra auditing l Look at attributes of assertion signer l Rooting trust u Externally recognized source, e.g., CA u Dynamically via VO structure  delegation u Dynamically via alternative sources, e.g., reputation

17. 17 Closing the Loop: GT4 Security Toolkit VO Rights Users Rights’ Compute Center Access Services (running on user’s behalf) Rights Local policy on VO identity or attribute authority CAS or VOMS issuing SAML or X.509 ACs SSL/WS-Security with Proxy Certificates Authz Callout: SAML, XACML KCA MyProxy Shib

18. 18 Grid Dynamics: Forming & Operating Communities l Define membership & roles; enforce laws & community standards u I.e., policy for service-oriented architecture u Addressing dynamics of membership & policy l Build, buy, operate, & share infrastructure u Decouple consumer & provider u For data, programs, services, computing, storage, instruments u Address dynamics of community demand

19. 19 Community Services Provider Content Services Capacity Bootstrapping a VO by Assembling Services 1) Integrate services from other sources u Virtualize external services as VO services 2) Coordinate & compose u Create new services from existing ones Capacity Provider “Service-Oriented Science”, Science, Foster, 2005

20. 20 Providing VO Services: (1) Integration from Other Sources l Negotiate service level agreements l Delegate and deploy capabilities/services l Provision to deliver defined capability l Configure environment l Host layered functions Community A Community Z …

21. 21 Virtualizing Existing Services into a VO l Establish service agreement with service u E.g., WS-Agreement l Delegate use to VO user User A VO Admin User B VO User Existing Services

22. 22 Deploying New Services Policy Client Environment Activity Allocate/provision Configure Initiate activity Monitor activity Control activity Interface Resource provider WSRF (or WS-Transfer/WS-Man, etc.), Globus GRAM, Virtual Workspaces

23. 23 Activities Can Be Nested Policy Client Environment Interface Resource provider Client

24. 24 VO User Embedded Resource Management: E.g., EGEE & OSG Cluster Resource Manager GRAM Cluster Resource Manager GRAM VO admin delegates credentials to be used by downstream VO services. VO admin starts the required services. VO jobs comes in directly from the upstream VO Users VO job gets forwarded to the appropriate resource using the VO credentials Computational job started for VO Client-side VO Scheduler Other Services VO Admin... Monitoring and control Headnode Resource Manager GRAM Deleg VO User VO Job

25. 25 Virtual Workspaces (Kate Keahey et al.) l GT4 service for the creation, monitoring, & management of virtual workspaces l High-level workspace description l WSRF mechanisms to monitor & manage l Multiple implementations u Dynamic accounts u Xen virtual machines u (VMware virtual machines) u … l Virtual clusters as a higher-level construct

26. 26 deploy, suspend How do Grids and VMs Play Together? Client request VM EPR inspect & manage use existing VM image Create VM image VM Factory VM Repository VM Manager create new VM image Resource VM start program

27. 27 Virtual OSG Clusters OSG cluster Xen hypervisors TeraGrid cluster OSG “Virtual Clusters for Grid Communities,” Zhang et al., CCGrid 2006

28. 28 Providing VO Services: (2) Coordination & Composition l Take a set of provisioned services … … & compose to synthesize new behaviors l This is traditional service composition u But must also be concerned with emergent behaviors, autonomous interactions u See the work of the agent & PlanetLab communities “Brain vs. Brawn: Why Grids and Agents Need Each Other," Foster, Kesselman, Jennings, 2004.

29. 29 Birmingham The Globus-Based LIGO Data Grid Replicating >1 Terabyte/day to 8 sites >40 million replicas so far MTBF = 1 month LIGO Gravitational Wave Observatory www.globus.org/solutions  Cardiff AEI/Golm

30. 30 l Pull “missing” files to a storage system List of required Files GridFTP Local Replica Catalog Replica Location Index Data Replication Service Reliable File Transfer Service Local Replica Catalog GridFTP Data Replication Service “Design and Implementation of a Data Replication Service Based on the Lightweight Data Replicator System,” Chervenak et al., 2005 Replica Location Index Data Movement Data Location Data Replication

31. 31 Hypervisor/OS Deploy hypervisor/OS Composing Resources … Composing Services Physical machine Procure hardware VM Deploy virtual machine State exposed & access uniformly at all levels Provisioning, management, and monitoring at all levels JVM Deploy container DRS Deploy service GridFTP LRC VO Services GridFTP

32. 32 Dynamic Service Deployment (Argonne + China Grid) l Interface u Upload-push u Upload-pull u Deploy u Undeploy u Reload “HAND: Highly Available Dynamic Deployment Infrastructure for GT4,” Li Qi et al., 2006

33. 33 Decomposition Enables Separation of Concerns & Roles User Service Provider “Provide access to data D at S1, S2, S3 with performance P” Resource Provider “Provide storage with performance P1, network with P2, …” D S1 S2 S3 D S1 S2 S3 Replica catalog, User-level multicast, … D S1 S2 S3

34. 34 Community Commons l What capabilities are available to VO? u Membership changes, state changes l Require mechanisms to aggregate and update VO information VO-specific indexes S S SS Information A A A FRESH MORE The age of information

35. GT4 Container GT4 Monitoring and Discovery Services (Uniform Treatment of State is Wonderful!) MDS- Index GT4 Cont. RFT MDS- Index GT4 Container MDS- Index Registration & WSRF/WSN Access GridFTP adapter Custom protocols for non-WSRF entities Clients (e.g., WebMDS) GRAMUser Automated registration in container WS-ServiceGroup

36. Facilities Computers Storage Networks Services Software People Implementation System-Level Problem Grid technology Decomposition U. Colorado Experimental Model NCSA Computational Model COORD. UIUC Experimental Model

37. 37 Dispatcher Grid backend Grid-enabled Business Intelligence Application Provision New Worker Process BI Server 2 BI server applications started and decommissioned by a Grid-enabled dispatcher Managed Pool of Shared Resources

38. 38 The Integrating Role of Grid Infrastructure Coarse Grained Dev / Test Grid Infrastructure Multiple applications and workload types Multiple resource types and instances Consistent & open management interface Consistent & open enactment interface End-to-end Quality of Service Fine Grained Data Driven Workflow

39. 39 Summary: Grid Dynamics and You l Grid = dynamic behaviors & environments u Dynamic communities & activities u Decoupling of service consumption from service production u Dynamic provisioning of services l We have tools to realize dynamic scenarios u Uniform state representation & access u Flexible security & policy framework u Virtual machines, dynamic services, & other building blocks l We now need much experimentation

40. 40 For More Information l Globus Alliance u www.globus.org l Background u www.mcs.anl.gov/~foster Come to GT4 workshop, 8:30-12:00 Wednesday u Overview of features u User experiences u Future directions 2nd Edition www.mkp.com/grid2

41. 41 Available in High-Quality Open Source Software … Data Mgmt Security Common Runtime Execution Mgmt Info Services GridFTP Authentication Authorization Reliable File Transfer Data Access & Integration Grid Resource Allocation & Management Index Community Authorization Data Replication Community Scheduling Framework Delegation Replica Location Trigger Java Runtime C Runtime Python Runtime WebMDS Workspace Management Grid Telecontrol Protocol Globus Toolkit v4 www.globus.org Credential Mgmt I. Foster, Globus Toolkit Version 4: Software for Service-Oriented Systems, LNCS 3779, 2-13, 2005

42. 42 WSDL, SOAP, WS-Security Custom Services WS-A, WSRF, WS-Notification Custom WSRF Services GT4 WSRF Web Services Registry & Admin GT4 Container (e.g., Apache Axis) GT4 & Web Services: Uniform State, Security, Mgmt User Applications

43. http://dev.globus.org Guidelines (Apache) Infrastructure (CVS, email, bugzilla, Wiki) Projects Include … GlobDev