The Kangaroo Approach to Data Movement on the Grid Douglas Thain, Jim Basney, Se-Chang Son, and Miron Livny

“Half-Interactive” Jobs on the Grid  The “Grid” is a vision of an international, highly available, robust, shared computation service.  Grid applications generally…..are half-interactive...are brittle wrt errors...have weak consistency requirements.  Can we perform on-line data access without subjecting apps to unexpected errors? App Unreliable Network

 Make a third party responsible for executing each application’s I/O operations.  Use all available resources to hide latencies.  Never return an error to the application. (Maybe tell the user or scheduler.)  Benefits: Fault tolerance, higher throughput.  Cost: Weaker consistency. The Kangaroo Approach To Data Movement

The Vision: A Data Grid File System File System File System File System K K K K K K K Data Movement System App Disk

Kangaroo Prototype KK App K An application may contact any node in the system and perform partial-file reads and writes. Disk The node may then execute or buffer operations as conditions warrant. Buffer

User Interface  An interposition agent converts POSIX operations into the Kangaroo protocol. % vi /kangaroo/coral.cs.wisc.edu/etc/hosts % gcc -c /gsiftp/ftp.cs.wisc.edu/test.c -o /kangaroo/coral/tmp/test.o K App POSIX Kangaroo Agent get, putread, write

Macrobenchmark: Image Processing  Post-processing of satellite image data: Need to compute various enhancements and produce output for each. Read input image For I=1 to N – –Compute transformation of image – –Write output image  Example: Image size about 5 MB Compute time about 6 sec IO-cpu ratio about 0.9 MB/s

I/O Models Compared OUTPUT CPU OUTPUT Online I/O: Offline I/O: Kangaroo: INPUT OUTPUT CPU OUTPUT CPUOUTPUTINPUTOUTPUTCPU OUTPUT CPUOUTPUTINPUTOUTPUTCPU PUSH CPU ReleasedTask Done CPU Released Task Done CPU Released