The File Mover: An Efficient Data Transfer System for Grid Applications C. Anglano, M. Canonico Dipartimento di Informatica Universita' del Piemonte Orientale, Alessandria (Italy)

Outline  Background and Motivations  The File Mover  Overlay Network  Architecture  Operation  Experimental Evaluation  Conclusions and Future Work

Background and Motivations  File transfer time is crucial for data intensive GRID applications  End-to-End approaches (e.g. FTP) achieve a throughput limited by the bandwidth available on the network path chosen by the IP routing layer  For 30% to 80% of the network path chosen by the IP routing, it was possible to find an alternative path with better performance [Savage]

The File Mover  A software system that addresses the above problem using overlay network  exploitation of network paths resulting in better throughput than the one chosen by the IP routing algorithm  aggregation of multiple paths for a single file transfer  short recovery time in case of link failure

The File Mover Overlay Network

The File Mover: architecture

Transferring a file  Virtual Path Computation  Virtual Path Creation  File Transfer

Virtual Path Computation  The best-throughput path is computed by using a variant of the Dijkstra single source shortest path algorithm

Virtual Path Computation  The best-throughput path is computed by using a variant of the Dijkstra single source shortest path algorithm

Virtual Path Creation

File Transfer  Before a virtual path can be used, it must be established, that is each virtual link must be “connected” to the next one.  pipelined file transfer

The File Mover: Fault Management  Periodically  the Transfer Manager thread of the Proxy acknowledges the data blocks  each Transfer Thread probes its successor in the virtual path  If a File Relay becomes unresponsive  the predecessor suspends the transmission  a new partial virtual path is computed  the file transfer restarts from the last block successfully received by the Proxy

Experimental Evaluation  The implementation has been deployed on a subset of the wide-area nodes of the Emulab Testbed  We performed a set of experiments, aimed at comparing the performance of the File Mover with FOBS  Two file size were used for each experiment (50Mbytes and 100Mbytes)

Experimental Results

Conclusions  The File Mover is the first file transfer system based on the overlay network paradigm  The File Mover chooses the network paths that are expected to yield the best performance in the medium-term future  Preliminary encouraging results:  good performance  the throughput achieved is reasonably close to the values measured with Iperf and predicted by the NWS

Future Work  Increase of system scalability (distributed database for the information service)  Adoption of rate-based UDP transfer protocols  Seem to provide better performance than TCP  Integration of caching policies  Relays may create local cache of the file they transfer  Improving of relay management policies  Take into consideration also relay already engaged in ongoing transfers, during virtual path creation