CMS Catalog Wrapper

1
CMS Catalog Wrapper Fast Data Transfer

• Faisal Khan
• Data 30 October 2006

2
Abstract

• Discovering data using the official CMS catalogs
• Effectively mirroring data to a place close to
  mass storage where data is located.
• Once the data is mirrored
• Data discovery can still be shown
• But we can switch to our mirrored dataset when
  doing physical file name (pfn) mappings
• Wide area transfer using FDT
• This is quite useful when doing a live demo where
  we want to show interaction with official cms
  catalog and at the same time do a high
  performance transfer.

3
Overview

• Ideally located close to the place where the
  actual data is stored
• Developed in python to interface it with CRAB
  wrapper classes.
• Ability to lookup Catalog for requested File
  Blocks
• Contact DLS for the storage location.
• Contact remote storage server file catalog for
  lfn pfn mappings
• Download the file using srmcp.

Mirror-Server
local srm-copy
e.g CERN
Mass Storage Server
Fast Data Transfer (WAN FDT)
Destination
e.g T2 at Caltech
4
Details
FDT config file
Mirror Server
Destination server Destination path List of files
to transfer. etc.
Python FDT Wrapper
(Catalog parameters) Requested file
block Requested site FDT using local copy etc.
POLL
Spawn FDT Java client process for each Config
file.
Transfer Config Dir
Wrapper
Select matching File Blocks (collection of
file blocks make up the dataset)
Select storage server(s) for above File Blocks
Fast Data Transfer
lfn -gt pfn mappings
srmcp
A boolean flag indicates weather to first
download (srmcp) the file on mirror server before
initiating FDT Transfer or use the local copy of
the same file block we downloaded earlier
(pre-fetched) Wrapper is written in python
using the CRAB DBS/DLS interface classes.
5
Next Step

• Install on a server at CERN
• Which is connected to our Ultralight network
• Get read access to the mass storage devices we
  want to test with
• Find suitable file-blocks (between 200 500 GB)
• Transfer test with this real CMS data
• Set it up for SC06