OGSA Data Architecture Dave Berry & Allen Luniewski OGF20, 7th May 2007

OGF IPR Policies Apply “I acknowledge that participation in this meeting is subject to the OGF Intellectual Property Policy.” Intellectual Property Notices Note Well: All statements related to the activities of the OGF and addressed to the OGF are subject to all provisions of Appendix B of GFD-C.1, which grants to the OGF and its participants certain licenses and rights in such statements. Such statements include verbal statements in OGF meetings, as well as written and electronic communications made at any time or place, which are addressed to: the OGF plenary session, any OGF working group or portion thereof, the OGF Board of Directors, the GFSG, or any member thereof on behalf of the OGF, the ADCOM, or any member thereof on behalf of the ADCOM, any OGF mailing list, including any group list, or any other list functioning under OGF auspices, the OGF Editor or the document authoring and review process Statements made outside of a OGF meeting, mailing list or other function, that are clearly not intended to be input to an OGF activity, group or function, are not subject to these provisions. Excerpt from Appendix B of GFD-C.1: ”Where the OGF knows of rights, or claimed rights, the OGF secretariat shall attempt to obtain from the claimant of such rights, a written assurance that upon approval by the GFSG of the relevant OGF document(s), any party will be able to obtain the right to implement, use and distribute the technology or works when implementing, using or distributing technology based upon the specific specification(s) under openly specified, reasonable, non-discriminatory terms. The working group or research group proposing the use of the technology with respect to which the proprietary rights are claimed may assist the OGF secretariat in this effort. The results of this procedure shall not affect advancement of document, except that the GFSG may defer approval where a delay may facilitate the obtaining of such assurances. The results will, however, be recorded by the OGF Secretariat, and made available. The GFSG may also direct that a summary of the results be included in any GFD published containing the specification.” OGF Intellectual Property Policies are adapted from the IETF Intellectual Property Policies that support the Internet Standards Process. IPR Notices Note Well for OGF meetings 2

Contents Current Status Architecture Document Scenarios Document 3

Two Informational Documents OGSA Data Architecture 70+ pages Describes services and their interfaces OGSA Data Scenarios 50+ pages Describes how the services can be composed to address particular scenarios 4

Progress since OGF19 Slow progress Small group We would welcome new contributors Focus on polishing the architecture document Mostly complete; review/editing in progress Updating and completing glossary We welcome feedback and comments 5

Current State Architecture Document Scenarios Document Technical content substantially complete Review of presentation in progress Scenarios Document Some scenarios still need work Integration with execution management services Aiming for submission in August 6

Architecture Document 7

Scope Files and databases (& storage) No stream management, session management, … Services and interfaces Storage, Access, Transfer Replication, Caching, Federation, Metadata catalogues Cross-cutting themes Security, Policies, … Part of the bigger OGSA picture E.g. Naming, Workflow, Transactions, Scheduling, Provisioning, … 8

Architecture Document Services Data Transfer Data Access Storage Resource Management Data Cache Data Replication Data Federation Metadata Catalogues Appendices Specifications referenced Mappings to specifications DAIS ByteIO SRM DMI … Glossary 9

Client APIs (non-OGSA) / Other services Basic Structure Client APIs (non-OGSA) / Other services Sink/ Source Sink/ Source Description Storage Access Description Access Storage Management Data Service Data Service Stored Data Resources Other Data Resources Managed Storage Service interface Resource interface 10

Transfer and Replication Client APIs (non-OGSA) / Other services Replication Transfer Replication Transfer Sink/ Source Description Sink/ Source Access Access Description Data Service Data Service Data Resources Data Resources Transfer Protocols Service interface Resource interface 11

Composite Entities Federation Cache Data Service Data Service Sink/ Source Description Sink/ Source Access Access Description Federation Cache Sink/ Source Description Sink/ Source Access Description Access Data Service Data Service Service interface Resource interface 12

Unstructured access RandomIO StreamableIO Sessionless Read() Write() Truncate() StreamableIO Stateful session resource Read() Write() Not part of data access layer Create() Destroy() 13

Structured access BasicExecute() SQLExecute() XQuery() RDF… XUpdate() 14

Storage Management (1) Directory Management Functions Synchronous & asynchronous List Files Release File Locks Remove Files Copy Files Move Files Make Directory Delete Directory 15

Storage Management (2) Space Management Functions: Sink / Source Reserve Space() Get Space() Release Space() Set Quota() Sink / Source Covered by Transfer 16

Data Transfer See the DMI WG Create EPR e.g. GridFTP Control Transfer Factory Create Source EPR User e.g. GridFTP Transfer Instance Control User wants to move file “foo” from Source Service to file “bar” to Sink Service. User contacts Source Service to get an WS-Name_foo <MyEPR> <wsa:Address> URL <ReferenceParameters> User contacts Sink Service to get an WS-Name_bar User contacts movement factory with: WS-Name_foo and WS-Name_bar (may pass in QoS parameteres + *) Movement factory talks to the service represented in WS-Name_foo and WS-Name_bar to obtain supported movement protocols Movement factory picks a protocol by some unspecified method Movement factory contacts service represented in WS-Name_foo with chosen protocol identifier and gets back a protocol specific URI_foo handle which represents the information required to transfer data from the source. Movement factory contacts WS-Name_bar with chosen protocol identifier and gets back a protocol specific URI_bar handle which represents the information required to transfer data to the sink Movement factory instantiates a movement service that initiates and monitors the transfer using URI_foo and URI_bar Sink 17

Replication Factory Replica Replica Catalogue CreateReplica() Management Process + Targets ModifyReplicatedContents() SynchroniseReplica() ValidateReplica() Destroy Replica() Replica Catalogue Maps name to list of (replica, target) pairs 18

Federation Factory Federation CreateFederation() AddResourceToFederation() RemoveResourceFromFederation() AddAccessMechanismToFederation() RemoveAccessMechanism() UpdateFederationProperties() Destroy() 19

Cache Factory Federation Create() ReConfigure() Synchronise() Destroy() 20

Gaps: For version 2 Information model for data resources For management, resource reservation, … Simple standard for describing files Name, size, creation time, etc. Registries of URIs that name languages, formats, etc. Security extensions Integration of access and transfer 3rd-party delivery Policies Replication coherency, caching coherency, catalogue consistency, etc. Sessions Streams Provisioning, etc. 21

Scenarios 22

Scenarios document Example scenarios of a generic nature Illustrates how the OGSA Data Architecture can be used to satisfy a selection of scenarios. Tests the architecture Not generating requirements 23

Scenarios Current Data replication Data federation Data pipelining Data staging Personal data service Data storage Data provenance Grid file system Possible Data caching Data warehousing Metadata catalogue service 24

Replication – Direct Access Data Storage 1 2. Provision Data Service 1 Replica Manager 1 Specify Replicas Replication Service 4. Transfer copies 4. Transfer copies Data Transfer Service 3. Register 8. Update 7. Notify Registry Service Example interfaces between the consumer/client, the services and the data storage elements with references to the relevant sections of the OGSA Data Architecture document [OGSA Data Arch]. The interfaces in the different steps of the scenario are as follows: 1. A data resource is: a. Registered with a replicating data service (details such as creation time, access control, etc. would also be included) – section 10.2 “Creating Replicas”. b. The replication service enters the data resource into a replica catalogue – section 10.3 “Discovering Replicas”. 2. The replication service uses a data transfer service to move copies of this data to different locations and tracks which data is kept where – section 6 “Data Transfer”. 3. Clients access the catalogue to find the data resource, or to return a list of resources that satisfy certain Quality of Service (QoS) requirements – section 10.3 “Discovering Replicas”. 4. Clients then access the stores either directly or indirectly – section 7 “Data Access”, i.e. any suitable data access interface such as DAIS or ByteIO [ByteIO]. 5. Changes to the data are notified to the replication service – section 3.4 “Notification of Events”. 6. Updates then occur between the data services to synchronize the replicas – section 6 “Data Transfer”. Here it has been assumed that a catalogue as described in section 12 “Metadata Catalogue & Registries” has been used. But on steps 1b) and 3, if a DAIS exposed database was employed for example, then DAIS [WS-DAI] updates or queries could also be performed. 4. Transfer copies 5. Find data Data Storage 2 6. Access data Data Service 2 User 25

Replication – Indirect Access Data Storage 1 2. Provision Data Service 1 Replica Manager 1 Specify Replicas Replication Service 4. Transfer copies 4. Transfer copies Data Transfer Service 3. Register 8. Update 7. Notify Registry Service User Example interfaces between the consumer/client, the services and the data storage elements with references to the relevant sections of the OGSA Data Architecture document [OGSA Data Arch]. The interfaces in the different steps of the scenario are as follows: 1. A data resource is: a. Registered with a replicating data service (details such as creation time, access control, etc. would also be included) – section 10.2 “Creating Replicas”. b. The replication service enters the data resource into a replica catalogue – section 10.3 “Discovering Replicas”. 2. The replication service uses a data transfer service to move copies of this data to different locations and tracks which data is kept where – section 6 “Data Transfer”. 3. Clients access the catalogue to find the data resource, or to return a list of resources that satisfy certain Quality of Service (QoS) requirements – section 10.3 “Discovering Replicas”. 4. Clients then access the stores either directly or indirectly – section 7 “Data Access”, i.e. any suitable data access interface such as DAIS or ByteIO [ByteIO]. 5. Changes to the data are notified to the replication service – section 3.4 “Notification of Events”. 6. Updates then occur between the data services to synchronize the replicas – section 6 “Data Transfer”. Here it has been assumed that a catalogue as described in section 12 “Metadata Catalogue & Registries” has been used. But on steps 1b) and 3, if a DAIS exposed database was employed for example, then DAIS [WS-DAI] updates or queries could also be performed. 4. Transfer copies 5. Request data 5. Find data Data Storage 2 6. Access data Data Service 2 Data Service 3 26

Visualisation Service Data Pipelining RenderingService 1. Submit job. 2. Store results. 3. Transfer results. Client Data Transfer Service Completed Animations Data Service 4. Return results. – 3rd Party Delivery In this scenario results may be streamed to the customers as soon as they are completed or they could be sent as a whole in one batch on final completion of the rendering job. Gives examples of 3rd Party Delivery and Data Streaming. Example interfaces between the consumer/client, the services and the data storage elements with references to the relevant sections of the OGSA Data Architecture document [OGSA Data Arch]. The interfaces in the different steps of the scenario are as follows: 1. Customer 1 (Designer) submits a rendering job to the Rendering Service – section 3.7 “Reservation and scheduling” and Execution Management Services (EMS) in OGSA. The details of the job would be defined in say JSDL [JSDL]. 2. Completed animation is stored to a common storage device – section 3.7. It is assumed that detail on how and where to store the results would be controlled by the Execution Management Services. 3. Rendering Service transfers the completed animations from the Data Service to the Visualization Service using the Data Transfer Service – section 6 “Data Transfer”. 4. The Visualization Service displays the animations to the clients (Designer & Reviewer/Customer) in an agreed format – most likely an application specific interface. 3. Transfer results. Analyst Visualisation Service 27

Data Storage – Bringing data online 1. Make files online. Data Storage Service 1. Make online. Client Nearline Storage 1. Make online. 3. Retire to nearline. 3. Retire to nearline. 2. Transfer files. Data Transfer Service The interfaces in the different steps of the scenario are as follows: 1. The files are made available online by the Data Storage Service – section 8.8 “SRM Interfaces” 2. The data are read through an appropriate interface, such as the Transfer Service – section 6 “Data Transfer”. 3. The online attribute of the files may expire and they can be retired to nearline storage – section 8.8 “SRM Interfaces”. Online Storage 2. Transfer files. Storage Devices 28

Data Staging 6. Delete output data. Input Data Output Data (copy) Data Transfer Service 2a. 4a. Data Service 1 2b. Transfer input data. 2a. 4a. 4b. Transfer output data. 4a. Stage output data. 2a. Stage input data. Client 1. Submit JSDL script. BES Container Data Service 2 BES Container: 3. Run executable & save resulting output data. Output Data Input Data (copy) 5. Delete input data (copy). 6. Delete output data. 29

Questions?

Full Copyright Notice Copyright (C) Open Grid Forum 2007. All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. The limited permissions granted above are perpetual and will not be revoked by the OGF or its successors or assignees. OGF Full Copyright Notice if necessary 31