Concatenation @ remote sites

1
Concatenation _at_ remote sites
Rob SnihurSimulation mtg., Jan. 13, 2005.

• Proposed a way to concatenate at remote sites in
  the past month
• Today new proposal. Request feedback.
• Old proposal
• Requires use of durable cache (i.e., disk)
• Carefully split sample to generate into N
  projects, such that each project will produce X
  GB of output
• 2. Cron job to monitor simulation output submit
  concatenation jobs

2
New proposal

• One DCAF segment simulation makes many (small)
  output files, totaling X GB (where X gt 1)
• At the end, concatenate all small files within
  the DCAF segment
• MCfull
• For (iistartiltiedi)
• Call MCProd
• cp outfile outdir
• Concatenate outdir

3
Features

• Simple
• self-contained
• No need for durable cache
• instead use disk available on worker nodes how
  much ?
• No need for separate submission of concatenation
  jobs
• Less book-keeping
• Average user can do it
• Fewer segments to submit
• Drawbacks
• Need to know event size for each sample in
  advance
• Could run a 100-event test job

4
Splitting a sample into projects

• The plan for generating a sample is controlled by
  a dataset definition file
• fcdflnx2 560 cat mcProduction/book/cdfptop/ttopui
  
• -------------------------------------------------
  ----------------------
• TTOPTI Pythia ttbar
• 5 events per 1nb-1 1.15M events
• -------------------------------------------------
  ----------------------
• DSID ttopui
• BOOK cdfptop
• DSNAME Pythia_ttbar_incl_178_multi
  -runs_v532_v01
• GENERATION_MODE 0
• MC_PROCESS_TCL mc_Pythia_ttbar_178.tcl
• FILTER_TCL mc_postgenNoFilter.tcl
• MINBIAS_TCL
• NEV_PER_INV_NB 5
• NEV_PER_SECTION 2000
• FIRST_RUN 141544
• LAST_RUN 179056
• RUN_LIST runlist_summer2004
• N_SECTIONS 997

5
Splitting a sample into projects

• To determine N_SECTIONS, run a script
• fcdflnx2 557 ./make_joblist.pl
  ../book/runlist_summer2004 5 2000
• RUN NUM EVENTS NUM SECTIONS
• 141572 362 1
• 141576 474 1
• 141597 799 1
• 141598 489 1
• 141618 519 1
• 141619 413 1
• ...
• 179054 674 1
• 179055 6906 4
• 179056 2515 2
• Summary
• ---------
• Number of runs requested 734
• Total luminosity of runs
  233826.04

6
Splitting a sample into projects

• Wrote a script to assign each output file to a
  project, such that each project has at least X
  GB
• fcdflnx2 507 DSdef2joblist_proj.sh
  mcProduction/book/cdfptop/ttopui 1
• FILE PROJECT TOTAL KB NUM EVENTS
• 1 ti022904.0001topu 1 61540 362
• 2 ti022908.0001topu 1 142120 474
• 3 ti02291d.0001topu 1 277950 799
• 12 ti022b1a.0001topu 1 921230 361
• 13 ti022b1e.0001topu 1 1029520 637
• 14 ti022b58.0001topu 2 132940 782
• 15 ti022b7a.0001topu 2 286960 906
• 16 ti02328d.0001topu 2 340000 312
• 23 ti023448.0001topu 2 977670 720
• 24 ti023449.0001topu 2 1016940 231
• 989 ti02bb6f.0004topu 175 494020 906
• 990 ti02bb70.0001topu 175 834020 2000
• 991 ti02bb70.0002topu 175 921570 515

7
Questions

• How much disk space is available on DCAF worker
  nodes?
• 8 GB/job _at_ FNAL (quotas to be turned on soon)
• 10 GB/job _at_ San Diego (will request quota
  enforcement)
• gt2 GB/job _at_ other sites
• No standard!
• What is the min/max output file size?
• Min 1 GB
• Max 2 GB
• Can we make small (100 MB) run sections?
• Yes

8
Error recovery

• Simulation failure
• Need to catch core dumps within simulation phase.
  Two options
• Either try it again
• Ignore it proceed
• Proposal Try simulation 2 times max then
  proceed
• Concatenation failure.
• Ditto