The Kangaroo Approach to Data Movement on the Grid Author: D. Thain, J. Basney, S.-C. Son, and M. Livny From: HPDC 2001 Presenter: NClab, KAIST, Hyonik Lee

Motivation In many Grid applications, client usually gets data from data server, processes them and sends the results back to data server.

Problem While data are being transferred to destination server, error or latency can occur due to  Server crash  Performance variation of server  Exhausted resources -> not resilient The throughput of grid application is not good because the client should be blocked until the data are transferred in such a hostile environment.

Solution Approach Use intermediary memory or disk buffer.  Background processes move data and handling errors. Interface  Get  Put  Commit  push

Advantages One-hop Kangaroo  Insulates the client form many difficulties Network failure, destination machine crash Exhausted resources (e.g. BW)  Read operation can be satisfied from cached data. Multi-hop Kangaroo  Multiple hops help avoid the need to co-allocate network resources along all hops.  Multiple hops increase the available spooling space.

Architecture Kangaroo Transport Consistency Adaptation get, put, push, commit Application Kangaroo Transport Kangaroo Transport Consistency ack open, read, write, close, fsync File System TCP/IP get, put, commit, push get, put, push,commit Consistency

Adaptation Layer It is unreasonable to expect programmers to convert existing applications to work with Kangaroo. Adaptation Layer converts standard POSIX operations into Kangaroo operations. A number of operations are missing in the Kangaroo interface, but it is sufficient to admit many grid applications that simply must read and write data. read write fsync exit open without create open with create lseek close get put push commit FdKindHostPathoffset 0Unix/dev/null0 1KangCoral/tmp/out KangCoral/tmp/err Unix/etc/hosts785 4Kangdbhost/data/db59687 …………… Just update table POSIX Kangaroo

Consistency Layer Commit causes the caller to block until all outstanding changes have been written to some stable storage. Push causes the caller to block until all outstanding changes have been delivered to their respective destinations.

Kangaroo Transport Layer Performance  Uses all available resources (net, memory, disk) to maximize throughput Error management  Retry  Delay

Experiments

Blocking procedure call Non-blocking message