Alex Sousa-MINOS Meeting

1
The MINOS Farm Processing Infrastructure
Work done in collaboration with Howie, Liz and
Jon.

Overview

• Develop an infrastructure for the MINOS
  Production Data Processing
• Automate the Production Data Processing setting
  up for two modes of operation
• Run through the full data sample
• Run on data files as they become available
• System to be setup using the Fermilab Fixed
  Target Farm (fnsfo) NuMi, KTeV,
  HyperCP(E871)...

http//mixcoatl.phy.tufts.edu/asousa/production/pr
oduction.html
2
The MINOS Farm Processing Infrastructure


The Farm Processing Block Diagram
3
The MINOS Farm Processing Infrastructure
A simple Farm model

CPU
DISK
NETWORK
FBSNG

http//www-isd.fnal.gov/fbsng/
4
The MINOS Farm Processing Infrastructure
Farms Batch System Next Generation (FBSNG)

• Job Control
• Submission
• Monitoring
• Killing/Canceling
• Holding/Releasing
• Scheduler
• Guaranteed scheduling
• Load Balancing
• Job queue priority changes dynamically as its
  sections or sections of other jobs start/complete
  
• Application Programmers Interface (API)
• Allows development of custom tools for job
  monitoring/management (Python binding)

5
The MINOS Farm Processing Infrastructure
FBSNG Job is submitted via a Job Description File
(JDF), e.g.
SECTION input_phase EXECinput.exe arg1
arg2 QUEUEinput_queue NUMPROC1
SECTION main EXECanalysis.exe
arg1 arg2 arg3 QUEUEmain_queue
NUMPROC5 DEPENDstarted(input_phase)
STDERRJ.startup.err
STDOUTJ.startup.out NEED1 SECTION
output_phase EXECoutput.exe arg1 arg2
arg3 arg4 QUEUEoutput_queue
DEPENDdone(input_phase) done(main)
NUMPROC1
/gt fbs submit test.jdf /gt fbs status -q
queue (ltjobidgtltsectionidgt) /gt fbs
kill -s section (ltjobidgtltsectionidgt)

6
The MINOS Farm Processing Infrastructure
FBSNG Web Monitoring

(http//www-isd.fnal.gov/cgi-bin/fbsng/fbswww/fbsw
ww)

List of Queues
CPU Utilization by Project
7
The MINOS Farm Processing Infrastructure
Plans for the Immediate Future

• Start running test scripts on the WorkerNuMi
  nodes
• Study and implement the automation of the Farm
  Processing Infrastructure.
• Obtain input from the Reconstruction Group and
  Software Group on
• What production job to run.
• What ntuples to produce.
• What calibration constants to use.
• ...