OGSA Data Architecture WG Data Transfer Session Allen Luniewski, IBM Dave Berry, NESC

29 June 2005 GGF 14: OGSA-D WG/TM-RG Joint Meeting 2 Intellectual Property Policy I acknowledge that participation in GGF14 is subject to the GGF Intellectual Property Policy. Intellectual Property Notices Note Well: All statements related to the activities of the GGF and addressed to the GGF are subject to all provisions of Section 17 of GFD-C.1 (.pdf), which grants to the GGF and its participants certain licenses and rights in such statements. Such statements include verbal statements in GGF meetings, as well as written and electronic communications made at any time or place, which are addressed to: the GGF plenary session, any GGF working group or portion thereof, the GFSG, or any member thereof on behalf of the GFSG, the GFAC, or any member thereof on behalf of the GFAC, any GGF mailing list, including any working group or research group list, or any other list functioning under GGF auspices, the GFD Editor or the GWD process Statements made outside of a GGF meeting, mailing list or other function, that are clearly not intended to be input to an GGF activity, group or function, are not subject to these provisions. Excerpt from Section 17 of GFD-C.1 Where the GFSG knows of rights, or claimed rights, the GGF secretariat shall attempt to obtain from the claimant of such rights, a written assurance that upon approval by the GFSG of the relevant GGF document(s), any party will be able to obtain the right to implement, use and distribute the technology or works when implementing, using or distributing technology based upon the specific specification(s) under openly specified, reasonable, non-discriminatory terms. The working group or research group proposing the use of the technology with respect to which the proprietary rights are claimed may assist the GGF secretariat in this effort. The results of this procedure shall not affect advancement of document, except that the GFSG may defer approval where a delay may facilitate the obtaining of such assurances. The results will, however, be recorded by the GGF Secretariat, and made available. The GFSG may also direct that a summary of the results be included in any GFD published containing the specification. GGF Intellectual Property Policies are adapted from the IETF Intellectual Property Policies that support the Internet Standards Process.

29 June 2005 GGF 14: OGSA-D WG/TM-RG Joint Meeting 3 Session Goals Explore issues surrounding data transfer Come to consensus on how to represent data transfer in the architecture Identify key use cases that support the consensus position.

29 June 2005 GGF 14: OGSA-D WG/TM-RG Joint Meeting 4 Uses of Data Transfer Data Access (e.g., query results) Updates to the data in a data service (e.g., fwrite) Replication: replica maintenance Cache maintenance Creation of copies of data  Including creation of “remote” result sets,  … and staging of data for computation Backup/recovery of data services Reading sensor data …

29 June 2005 GGF 14: OGSA-D WG/TM-RG Joint Meeting 5 Data Transfer Mechanisms “Base” web services: e.g., in the call/return of a SOAP interchange to a port type, SwA, MTOM, DIME, … Grid FTP, “raw” FTP Info-D Base HTTP (no SOAP) Base TCP/IP (sockets) Storage level mechanisms (e.g., flash copy) …

29 June 2005 GGF 14: OGSA-D WG/TM-RG Joint Meeting 6 Issues in Data Transfer (1) Performance, performance, performance!  Potentially massive amounts of data to be transferred Transparent to the application?  Does the application care about “how” or just that its goals are met? Degree of application control of transfer mechanism and operation of the transfer? Do we transfer bytes or structured (semantically/syntactically understood?) data? Is data transfer responsible for doing transformation on the data?

29 June 2005 GGF 14: OGSA-D WG/TM-RG Joint Meeting 7 Issues in Data Transfer (2) How do services (e.g., replication) interact with data transfer services?  Are private transfer mechanisms allowed/desirable? How do we promote interoperability while allowing services to choose optimal transfer mechanisms?

29 June 2005 GGF 14: OGSA-D WG/TM-RG Joint Meeting 8 Issues in Data Transfer (3) In the source/sink model,  who connects the source to the sink?  Is there a “pipe” service between the source and sink?  Who manages it?  Is data pushed through the pipe, pulled through it or a combination?  Role of the source, destination and, possibly, a service that “is” the pipe?

29 June 2005 GGF 14: OGSA-D WG/TM-RG Joint Meeting 9 Issues in Data Transfer (4) In what ways does the source/sink model differ from Unix pipes? Can the source/sink be other than an OGSA (data) service? How do we handle sources that continuously produce data (e.g., a sensor)?  i.e., modeling streams

29 June 2005 GGF 14: OGSA-D WG/TM-RG Joint Meeting 10 Issues in Data Transfer (5) How do we provide means so that “middle men” do not have to get their hands on data just to pass it along?  Is there a distinction between how the system is architected in this regard and how services are implemented? Is the source/sink model sufficient to address these various issues?

29 June 2005 GGF 14: OGSA-D WG/TM-RG Joint Meeting 11 Next Steps ?