DUCKS

1
DUCKS Distributed User-mode Chirp-Knowledgeable
Server

• Joe Thompson
• Jay Doyle

2
DUCKS Motivation

• Performing distributed jobs on many files comes
  with many problems.
• Storage Space.
• If all files are stored on one machine. Many
  concurrent jobs hammer the single machine with
  I/O.
• Solution Distribute the files to multiple
  machines.
• Presents problem of file management (confusing).
• Proposed Solution A tracking system to abstract
  file location from the user.

3
DUCKS Features

• Bring together functionality of Condor and CHIRP
  in an easy to use package.
• Abstract Condor and CHIRP interfaces.
• Intelligently distribute files over CHIRP
  servers.
• Provide a semantic per-user namespace for file
  storage
• Provide simple interface using Condor for a
  Program-To-Data model.
• Provide simple interface using Condor for a
  Data-To-Program model.

4
DUCKS Components

• Ducks Server
• C Server
• Event-based transaction handler process to
  interface with clients.
• Background CHIRP monitoring process.
• Background cleanup process.
• MySQL Database
• Handles the metadata of the file system.
• Handles user permissions and state.
• Ducks Client
• Standalone command line scripts that interface
  with the DUCKS server, CHIRP file system, and
  Condor.

5
DUCKS File Input

• User calls a client side script.
• Request to store goes to the server.
• Server chooses CHIRP server and unique pathname.
• Server sends this info back to the client.
• Client utilizes chirp_put to store the file.
• Client responds to the server.
• Server updates the database.

STORE CHIRP Path
SEND DUCKS Name GET CHIRP Path
CHIRP
DUCKS
Key
Data Transfer Database Query
6
DUCKS File Retrieval

• User calls a client side script
• Request for semantic name goes to the server
• Server responds with CHIRP server and path
• Client utilizes chirp_get to retrieve the file

REQUEST CHIRP Path GET File
SEND DUCKS Name GET CHIRP Path
CHIRP
DUCKS
Key
Data Transfer Database Query
7
DUCKS Job Execution

• 2 Modes Job-to-Data Data-to-Job
• Job-to-Data
• User sends file list to the server (with
  executable).
• Server builds a Condor script for all requested
  files on a given machine and sends the script
  back to the user
• The user submits the job requests
• Data-to-Job
• User script builds a Condor wrapper script that
  requests files from DUCKS on any machine it gets
  to use

8
DUCKS File Job Input
Chirp 1
File
File
Chirp 2
File
Job
Chirp 3
User 1
DUCKS Client
DUCKS Database
Key
DUCKS Server
Data Transfer Database Query
9
DUCKS Progress

• Server 60 complete.
• All functionality has been outlined with skeleton
  code.
• Connect, Disconnect, Store messages are mostly
  functional.
• CHIRP Tracker 100 functional.
• Background Retry process 100 functional.
• Database table structure and interface is mostly
  functional.
• Client 30 complete.
• Connect and Disconnect messages 100.
• Store 70.
• Retrieve, Remove and LS have skeleton code.
• Condor execution scripts are designed but need
  porting.

10
DUCKS Demo
11
DUCKS Demo