1. GridFTP Introduction – Page 1Grid Forum 5 GridFTP Steve Tuecke Argonne National Laboratory

2. GridFTP Introduction – Page 2Grid Forum 5 Overview u Motivation for GridFTP Working Group u Requirements u GridFTP Solution u GridFTP Working Group Documents u Role of GridFTP Working Group

3. GridFTP Introduction – Page 3Grid Forum 5 GridFTP Working Group Motivation u Data transfer solutions have been developed by the Globus Project over past ~5 years, GridFTP is 3 rd generation u Grid Forum started ~1 year ago to promote and develop Grid technologies –Critical mass of people working in this area u Grid Forum GridFTP working group formed to foster the further specification and development of GridFTP –Community effort to move GridFTP forward

4. GridFTP Introduction – Page 4Grid Forum 5 Some Important Definitions u Resource u Network protocol u Network enabled service u Application Programmer Interface (API) u Syntax u Software Development Kit (SDK)

5. GridFTP Introduction – Page 5Grid Forum 5 Resource u Entity that is to be shared –Includes computers, storage, data, software u Does not have to be physical entity –Condor pool, distributed file system, … u Defined in terms of interfaces, not devices –E.g. LSF defines compute resource –Open/close/read/write defines access to a distributed file system, e.g. NFS, AFS, DFS

6. GridFTP Introduction – Page 6Grid Forum 5 Network Protocol u A formal description of message formats and a set of rules for message exchange –Rules may define sequence of message exchanges –Protocol may define state-change in endpoint, e.g. state change u Good protocols designed to do one thing –Protocols can be layered u Examples of protocols –IP, TCP, TLS, FTP, HTTP, Kerberos

7. GridFTP Introduction – Page 7Grid Forum 5 Network Enabled Services u Implementation of a protocol that defines a set of capabilities –Protocol defines interaction with service –All services require protocols –Not all protocols are used to provide services (e.g. IP, TLS) u Examples: FTP and Web servers Web Server IP Protocol TCP Protocol TLS Protocol HTTP Protocol FTP Server IP Protocol TCP Protocol FTP Protocol Telnet Protocol

8. GridFTP Introduction – Page 8Grid Forum 5 API (Application Programming Interface) u A specification for a set of routines to facilitate application development –Refers to definition, not implementation, e.g. there are many implementations of MPI u Spec often language-specific (or IDL) –Routine name, number, order and type of arguments; mapping to language constructs –Behavior or function of routine u Examples –GSS API, MPI

9. GridFTP Introduction – Page 9Grid Forum 5 Syntax u A specification for how a defined set of information is encoded into bits –A syntax may be defined as part of a protocol or API »Protocol messages have defined syntax »A syntax may be used as API function argument –But syntax can also stand alone u Good syntax designed to do one thing –Syntaxes can be layered u Examples –XML, ASN.1, X.509, LDIF

10. GridFTP Introduction – Page 10Grid Forum 5 SDK (Software Development Kit) u A particular instantiation of an API u SDK consists of libraries and tools –Provides implementation of API specification u Can have multiple SDKs for an API u Examples of SDKs –MPICH, Motif Widgets

11. GridFTP Introduction – Page 11Grid Forum 5 Multiple APIs but a Single Protocol Example: TCP/IP u Multiple APIs: BSD sockets, Winsock, System V streams, … u Different programs use different APIs u Interoperability: programs using different APIs can exchange information TCP/IP Protocol: Reliable byte streams WinSock APIBerkeley Sockets API Application

12. GridFTP Introduction – Page 12Grid Forum 5 Single API, but Multiple Protocols E.g., GSS-API u GSS-API provides portability: any correct program compiles & runs on a platform u Does not provide interoperability: all processes must link against same SDK –E.g., GSI and Kerberos versions of GSS-API Application GSS-API GSI SDK GSI protocol Kerberos SDK Kerberos protocol TCP/IP Different message formats, exchange sequences, etc.

13. GridFTP Introduction – Page 13Grid Forum 5 I.e., Standard APIs and Protocols are Both Important: For Different Reasons u Standard APIs/SDKs are important –They enable application portability –But w/o standard protocols, interoperability is hard (every SDK speaks every protocol?) u Standard protocols are important –Enable cross-site interoperability –Enable shared infrastructure –But w/o standard APIs/SDKs, application portability is hard (different platforms access protocols in different ways)

14. GridFTP Introduction – Page 14Grid Forum 5 Grid Data Needs u Transfer of large amounts of data (petabytes or terabytes) between storage systems u Access to large amounts of data (terabytes or gigabytes) by many geographically distributed applications and users for analysis, visualization, etc.

15. GridFTP Introduction – Page 15Grid Forum 5 Requirements u Grid Security Infrastructure (GSI) and Kerberos support u Third-party control of data transfer u Parallel data transfer u Striped data transfer u Partial file transfer u Automatic negotiation of TCP buffer/window size u Support for reliable/recoverable data transfer

16. GridFTP Introduction – Page 16Grid Forum 5 Candidate Standards u FTP –Defined by a set of IETF RFCs –No partial file, parallel/striped, GSI, etc –Separate control & data channels u WebDAV –New extension to http –No third party transfer, parallel/striped, etc. –Combined control & data channel

17. GridFTP Introduction – Page 17Grid Forum 5 Separate Control & Data Channels u WebDAV combines control and data over single channel u FTP splits control and data –Supports multiple, user selectable data channel protocols u Advantage to split channels –Third party transfers handled cleanly –Can (cleanly) define new data channel protocols »E.g. parallel/striped transfer, automatic TCP buffer/window negotiation –Amenable to high-performance proxies »E.g. For firewalls, load balancing, etc.

18. GridFTP Introduction – Page 18Grid Forum 5 GridFTP Solution u Built on existing FTP standards –RFC 949: File Transfer Protocol –RFC 2228: FTP Security Extensions –RFC 2389: Feature Negotiation for the File Transfer Protocol –Draft: FTP Extensions u Extends standards with –Additions to security extensions, partial file transfer, parallel/striped transfer, TCP buffer/window size tuning,

19. GridFTP Introduction – Page 19Grid Forum 5 GridFTP Implementation Status u Modified wu-ftpd server –Most features u Modified ncftp client –Security, TCP buffer setting u Modified HPSS & Unitree ftpd server –Security u Globus Toolkit client and server SDKs, and command line tools –Most features u Striped FTP server (aka DPSS2)

20. GridFTP Introduction – Page 20Grid Forum 5 GridFTP Working Group Documents u GridFTP: A Data Transfer Protocol for the Grid –Overview of working group activities and documents –Requirements –Informational draft u GridFTP: FTP Extensions for the Grid –Protocol specification

21. GridFTP Introduction – Page 21Grid Forum 5 GridFTP Protocol Specifications u Existing standards –RFC 949: File Transfer Protocol –RFC 2228: FTP Security Extensions –RFC 2389: Feature Negotiation for the File Transfer Protocol –Draft: FTP Extensions u New drafts –GridFTP: FTP Extensions for the Grid

22. GridFTP Introduction – Page 22Grid Forum 5 GridFTP APIs u Should there be standard API(s)? –Posix I/O –SRB client –grid_storage –globus_ftp_client –MPI-IO –HDF5 –etc u Beyond scope of this working group u Common protocol beneath these APIs would allow interoperability

23. GridFTP Introduction – Page 23Grid Forum 5 Role of GridFTP Working Group u Bring together those who are interested in the future of GridFTP to help foster the… –continued specification and standardization of GridFTP –development of inter-operable GridFTP implementations –widespread adoption of GridFTP as a transfer protocol for the Grid u Develop drafts which together define GridFTP –May submit some of them to IETF u Move GridFTP forward to better address Grid data transfer requirements

24. GridFTP Introduction – Page 24Grid Forum 5 NOT Goals of GridFTP Working Group u This working group will not start from first principles –Starting point is roughly GridFTP as it now exists –FTP base is assumed u Its not design by committee –Seeking rough consensus, with broad input –Draft authors and WG chair have final say

25. GridFTP Introduction – Page 25Grid Forum 5 GF5 GridFTP Working Session u Is this appropriate for Grid Forum? u Who is interested in participating, and in what capacity? u Is the problem scoped appropriately (at least for now)? u What are the right drafts to write? u Establish rough timeline for drafts

26. GridFTP Introduction – Page 26Grid Forum 5 A Call To Arms u The Grid Forum security working group needs to do more than just gather 3 times a year to chat about data management. u But Grid Forum is only appropriate for this activity if people meaningfully participate. –I will be doing this regardless. –But it will hopefully be done better and faster with broad participation. –If there is not meaningful participation, I won’t bother with the overhead of Grid Forum.