DUCKS – Distributed User-mode Chirp- Knowledgeable Server Joe Thompson Jay Doyle

Performance Usability Chirp & Condor Chirp & Condor DUCKS Motivation

DUCKS Goals Bring together functionality of Condor and CHIRP in an easy to use package. Abstract Condor and CHIRP interfaces. Intelligently distribute files over CHIRP servers. Provide simple interface for the Chirp Active Storage Program-To-Data model. Provide simple interface for the Condor Data-To- Program model.

MySQL Garbage Collector Garbage Collector Chirp Tracker Chirp Tracker Timeout Handler Timeout Handler Incoming Message Queue Incoming Message Queue Incoming Message Queue Incoming Message Queue Incoming Message Queue Incoming Message Queue Chirp Server List Chirp Server List Transaction List Transaction List Transaction Handler Transaction Handler DUCKS Client DUCKS Client

ducks_put Local MachineDUCKS Server Chirp Node Store_Request 1) Verify the pair is not already in the DB 2) Find a Chirp node with enough free space to store the file 3) Query database for path name to use on the Chirp node Store_Response 4) Parse response to get the Chirp storage location for chirp_put Store_success 5) Update DB to reflect this file storage File

ducks_get Local MachineDUCKS Server Chirp Node Get_Request 1) Query DB for Chirp location of Get_Response 2) Parse response to get the Chirp storage location for chirp_get File

ducks_delete Local MachineDUCKS Server Garbage Collector delete_Request 1) Set the delete_flag of the entry in the DB 2) Periodically query the DB for files with the delete_flag set DUCKS DB … … Result Set 3) Delete files in the list Chirp Node Delete File 4) Remove files from DB

ducks_ls Local MachineDUCKS Server ls_Request 1) Query the DB for all files owned by the user that match “% %”; 2) Start a background thread and pass it the result set containing the found files Result Set 3) Iterate through the set and send the file information to the client File_info ls Thread

Distribution via Chirp Active Storage to nodes already storing input Wrapper/ Submit Scripts input01.txt input02.txt input03.txt input04.txt exe/libs request ducks_get Chirp Nodes Client DUCKS Server Job Request Response with locations of input files

Client Condor exe/libs/input_file request Chirp Nodes DUCKS Server Distribute tasks to any available node Get input and exe/lib files from other nodes in the Chirp cluster with ducks_get

DUCKS Future Work The basic framework is implemented. Add a more robust file interface. Implement DUCKS management of job status information. (Queued, Running, Complete)

?