1. DUCKS – Distributed User-mode Chirp-Knowledgeable Server
Joe Thompson
Jay Doyle

2. DUCKS Motivation
Performing distributed jobs on many files comes with many problems.
Storage Space.
If all files are stored on one machine. Many concurrent jobs hammer the single machine with I/O.
Solution : Distribute the files to multiple machines.
Presents problem of file management (confusing).
Proposed Solution : A tracking system to abstract file location from the user.

3. DUCKS Features
Bring together functionality of Condor and CHIRP in an easy to use package.
Abstract Condor and CHIRP interfaces.
Intelligently distribute files over CHIRP servers.
Provide a semantic per-user namespace for file storage
Provide simple interface using Condor for a Program-To- Data model.
Provide simple interface using Condor for a Data-To- Program model.

4. DUCKS Components Ducks Server: Ducks Client: C Server: MySQL Database:
Event-based transaction handler process to interface with clients.
Background CHIRP monitoring process.
Background cleanup process.
MySQL Database:
Handles the metadata of the file system.
Handles user permissions and state.
Ducks Client:
Standalone command line scripts that interface with the DUCKS server, CHIRP file system, and Condor.

5. DUCKS File Input User calls a client side script.
Request to store goes to the server.
Server chooses CHIRP server and unique pathname.
Server sends this info back to the client.
Client utilizes chirp_put to store the file.
Client responds to the server.
Server updates the database.
STORE: CHIRP Path
SEND: DUCKS Name
GET: CHIRP Path
CHIRP
DUCKS
Key:
Data Transfer
Database Query

6. DUCKS File Retrieval User calls a client side script
Request for semantic name goes to the server
Server responds with CHIRP server and path
Client utilizes chirp_get to retrieve the file
REQUEST : CHIRP Path
GET: File
SEND: DUCKS Name
GET: CHIRP Path
CHIRP
DUCKS
Key:
Data Transfer
Database Query

7. DUCKS Job Execution 2 Modes : Job-to-Data & Data-to-Job Job-to-Data :
User sends file list to the server (with executable).
Server builds a Condor script for all requested files on a given machine and sends the script back to the user
The user submits the job requests
Data-to-Job :
User script builds a Condor wrapper script that requests files from DUCKS on any machine it gets to use

8. DUCKS File & Job Input Chirp 1 File File Chirp 2 File Job Chirp 3
User 1
DUCKS Client
DUCKS Database
Key:
Data Transfer
Database Query
DUCKS Server

9. DUCKS Progress Server 60% complete. Client 30% complete.
All functionality has been outlined with skeleton code.
Connect, Disconnect, & Store messages are mostly functional.
CHIRP Tracker 100% functional.
Background Retry process 100% functional.
Database table structure and interface is mostly functional.
Client 30% complete.
Connect and Disconnect messages 100%.
Store 70%.
Retrieve, Remove and LS have skeleton code.
Condor execution scripts are designed – but need porting.

10. DUCKS Demo

11. DUCKS Demo