Bulk%20Data%20Transfer%20Activities

1
Bulk Data Transfer Activities

• We regard data transfers as first class
• citizens, just like computational jobs.
• We have transferred 3 TB of DPOSS data
• (2611 x 1.1 GB files) from SRB to UniTree
• using 3 different pipeline configurations.
• The pipelines are built using Condor and Stork
• scheduling technologies. The whole process
• is managed by DAGMan.

• What a batch system means for computational jobs,
  Stork means the same for data placement
  activities (ie. transfer, replication,
  reservations, staging) in Grid it schedules,
  runs, monitors data placement jobs and ensures
  that they complete.
• Stork can interact with heterogeneous middleware
  and end-storage systems easily and recover from
  failures successfully.
• Stork makes data placement a first class citizen
  of Grid computing.
• http//www.cs.wisc.edu/condor/stork

• Condor is a specialized workload management
  system for compute-intensive jobs.
• Condor provides a job queuing mechanism,
  scheduling policy, priority scheme, resource
  monitoring, and resource management.
• Condor chooses when and where to run jobs based
  upon a policy, carefully monitors their progress,
  and ultimately informs the user upon completion.
• http//www.cs.wisc.edu/condor

• DAGMan
• DAGman (Directed Acyclic Graph Manager) is a
  meta-scheduler for Condor. It manages
  dependencies between jobs at a higher level than
  the Condor Scheduler. DAGMan can now also
  interact with Stork.
• http//www.cs.wisc.edu/condor/dagman

• We used the native file transfer mechanisms for
  each underlying system SRB, Globus GridFTP, and
  UniTree for the transfers.
• We described each data transfer with a five stage
  pipeline, resulting in a 5x2611 node workflow
  (DAG) managed by DAGMan.
• We obtained an end-to-end throughput (from SRB
  to UniTree) of 11 files per hour (3.2 MB/sec).

• GridFTP
• High performance, secure, reliable data transfer
  protocol from Globus
• http//www.globus.org/datagrid/gridftp.html
• SRB Storage Resource Broker
• Client-Server middleware that provides a uniform
  interface for connecting to heterogeneous data
  resources
• http//www.npaci.edu/DICE/SRB
• UniTree
• NCSAs High-speed, high-capacity mass storage
  system
• http//www.ncsa.uiuc.edu/Divisions/CC/HPDM/unitree
  

• We used the experimental DiskRouter tool instead
  of Globus GridFTP for cache-to-cache transfers.
• We obtained an end-to-end throughput (from SRB
  to UniTree) of 20 files per hour (5.95 MB/sec).

• DiskRouter
• Moves large amounts of data efficiently (on the
  order of terabytes)
• Uses disk as a buffer to aid in large data
  transfers
• Performs application level routing
• Increases network throughput by using multiple
  sockets and setting tcp buffer sizes explicitly
• http//www.cs.wisc.edu/condor/diskrouter

Unitree not responding
SDSC cache reboot UW CS Network outage
Diskrouter reconfigured and restarted
SRB server maintenance

• We skipped the SDSC cache, and performed direct
  SRB transfers from SRB server to NCSA cache.
• We described each data transfer with a three
  stage pipeline, resulting in a 3x2611 node
  workflow (DAG).
• We obtained an end-to-end throughput (from SRB to
  UniTree) of 17 files per hour (5.00 MB/sec).

PDQ Expedition
Unitree maintenance
SRB server problem