Reliable File Transfer: Lessons Learned Bill Allcock, ANL Ravi Madduri, ANL

Overview l Quick Overview of the Service l Design Issues u Lifetime Management u Virtualization u Service Data Elements l Implementation Issues u SOAP Processing Issues u Standardization u Avoid Language specific Types u Anyhelper API u Fault Tolerance

RFT in Action Registry * The scenarios in this presentation are offered as examples and are not prescriptive 1. A Grid Service Container is started up; It contains an RFT Factory service; The RFT Factory service registers itself RFT Factory Grid Service Container

Client RFT in Action Registry * The scenarios in this presentation are offered as examples and are not prescriptive 2. From a known registry, the client discovers a factory by querying the Service data of the registry RFT Factory Grid Service Container

Client RFT in Action 3. The client calls the createService operation on the factory and passes in a TransferRequest RFT Factory Grid Service Container * The scenarios in this presentation are offered as examples and are not prescriptive

Client RFT in Action RFT Factory Grid Service Container RFT Service Instance - Start the Instance - Deserialize XML to Java - Write Request via JDBC - Persist Service State 4. The instance is started, and the factory returns a locater * The scenarios in this presentation are offered as examples and are not prescriptive

Client RFT in Action RFT Factory Grid Service Container RFT Service Instance - Start the Instance - Deserialize XML to Java - Write Request via JDBC - Persist Service State 5. Client calls Start(), subscribes to notificaitons, etc. * The scenarios in this presentation are offered as examples and are not prescriptive

RFT in Action l Service is OGSI compliant l Uses existing GridFTP (non-OGSI) protocols and tools to execute 3 rd Party Transfer for the user l Provides extensive state transition notification GridFTP Server RFT Service Instance * The scenarios in this presentation are offered as examples and are not prescriptive

Lifetime Management l Lifetime management is a key aspect of OGSI l Was not intuitively clear how to handle this for “disconnected” services l Our (perhaps not optimal) solution is to give it an indefinite lifetime l Should there be an activity monitor? u Does that really solve the problem? u Any other ideas?

Virtualization l Another Key Design Issue in services l We virtualize data movement u It works, LBL and ANL have interoperable implementation u Need to standardize… l Data Services Virtualization from DAIS u Should we pass around GSHs of file services rather than URLs? l Granularity u Single file.vs. Multi-file.vs. service composition

Service Data Elements l A huge improvement over the non-OGSI services l Information Services are (should be) baked in to the services l Defined both push (event notification) and pull (full transfer status) SDEs l Need to be cognizant of size, frequency, and performance of notifications l Lots of interesting possibilities: bandwidth, errors, network status, etc..

Implementation Issues l SOAP Deserialization u Deserialization can be a HUGE issue. Our original (very simple) XML could take up to 30 minutes to process. u A straight forward change reduced that drastically. u SOAP engine also needs to be looked at. l Standardization u Critical for success u Will be moving to the OGSI-Agreement interface

Implementation Issues (cont.) l Language Specific Data Types u Avoid them (I.e. Java vector type) u A python based service would not be able to deserialize that. l AnyHelper API u Will deserialize any general XML blob that utilizes basic types.

Implementation Issues (cont.) l Fault Tolerance u Multiple levels of Fault tolerance u GridFTP will handle remote failures u RFT provides fault tolerance of request via JDBC compliant database u Service container provides instance fault tolerance u We write only the primary key into the wsdd file to avoid slowing down container restart.