ATLAS full chain exercise

1
ATLAS full chain exercise

• Chris Collins-Tooth, Glasgow
• James R Catmore, Lancaster
• NESC, Edinburgh 7-8 Feb, 2005

2
What is the full chain?

• Generation use pythia etc. to produce events.
• Simulation run products through detector.
• Digitisation (with/without pile-up) digitise
  hits.
• Reconstruction (make ESD)
  this reconstructs particles, jets, tracks
  etc.
• AOD manufacture condense the ESD.
• Analysis make histograms!!
• For all of these steps, we will use athena v9.0.3

100011110101110101100101110110100
3
Background AMI, Castor and Pool

• You will need to know a little about these during
  the excercise.
• AMI Atlas Metadata Interface
• Metadata is data about data.
• Use AMI to track down physics datasets and
  individual files.
• See http//isnpx1158.in2p3.fr8180/AMI/
• Try AMI generic search web interfaces.
• For an example, choose project DC1, process
  simulation, entity dataset
• Under logicalDatasetName, type higgs returns
  a list of higgs datasets.
• These datasets can be probed e.g. by clicking the
  partition link to retrieve filenames.
• Castor
• Large storage area which can be used for your
  input/output files.
• Simple commands rfdir (ls) rfmkdir (mkdir) rfcp
  (cp)
• Slow to stage files, so not efficient to use for
  small (lt10 Mb) files
• Pool
• Pool of persistent objects for LHC.
• Athena expects all input files you use to be
  listed in PoolFileCatalog.xml.
• Athena automatically adds output files to the
  catalogue with a unique GUID.

4
Background AMI query
5
Background AMI result
6
Background AMI, Castor and Pool

• You will need to know a little about these during
  the excercise.
• AMI Atlas Metadata Interface
• Metadata is data about data.
• Use AMI to track down physics datasets and
  individual files.
• See http//isnpx1158.in2p3.fr8180/AMI/
• Try AMI generic search web interfaces.
• For an example, choose project DC1, process
  simulation, entity dataset
• Under logicalDatasetName, type higgs returns
  a list of higgs datasets.
• These datasets can be probed e.g. by clicking the
  partition link to retrieve filenames.
• Castor
• Large storage area which can be used for your
  input/output files.
• Simple commands rfdir (ls) rfmkdir (mkdir) rfcp
  (cp)
• Slow to stage files, so not efficient to use for
  small (lt10 Mb) files
• Pool
• Pool of persistent objects for LHC.
• Athena expects all input files you use to be
  listed in PoolFileCatalog.xml.
• Athena automatically adds output files to the
  catalogue with a unique GUID.

7
Background DCx Web pages

• These web-pages are really mentioned here for
  reference.
• There are good tips, such as naming conventions,
  which will help you locate datasets (within AMI)
  produced, say, for Rome.
• http//phyweb.lbl.gov/ianh/dc/ -
• naming,
• links to physics groups Monte-Carlo (MC) pages,
• notes on other generators/hadronisers (e.g.
  Herwig, Taola)
• http//www-theory.lbl.gov/7Eianh/dc/dc2.html
• dc2 specific info
• progress of datasets etc.

8
We will require

• Access to LXPLUS
• Around 50Mb space
• cmt already set up you should have a /cmthome
  directory with requirements file
• Access to castor ensure you can see your space
  on castor and make a directory there...
• rfdir /castor/cern.ch/user/ltyour
  initialgt/ltusernamegt/
• rfmkdir /castor/cern.ch/user/ltyour
  initialgt/ltusernamegt/tutorialoutput/

9
Firstly, we must set up an environment for athena
v9.0.3

• Log in to lxplus... gt ssh -l ltusernamegt
  lxplus.cern.ch
• Make a NEW directory for the tutorial... gt mkdir
  lttutdirgt
• Edit the requirements file in /cmthome ...
• ...
• macro ATLAS_RELEASE "" \
• tutorial903 "9.0.3"
• ...
• macro devarea ""\
• tutorial903area "HOME/lttutdirgt"
• Source the setup.sh file...
• source /cmthome/setup.sh -tagopt,tutorial903,tut
  orial903area
• Go to the lttutdirgt, ready to check out
  packages... gt cd lttutdirgt
• (in theory, you should only need RecExCommon
  package, in practice others might be needed..
  keep your eyes on atlas-sw-developers mailing
  list, and the software release page for clues!)

you will need the continuation \
tags
10
Environment... check out packages

• Find the correct version of RecExCommon to check
  out for athena v9.0.3... look in the dist area,
  which is part of CMTPATH...
• echo CMTPATH

11
Environment... check out packages

• Find the correct version of RecExCommon to check
  out for athena v9.0.3... look in the dist area,
  which is part of CMTPATH...
• echo CMTPATH
• /afs/cern.ch/user/c/ccollins/scratch0/lttutdirgt/af
  s/cern.ch/atlas/software/dist/9.0.3/afs/cern.ch/a
  tlas/offline/external/Gaudi/0.14.6.12pool/afs/cer
  n.ch/atlas/offline/external/LCGCMT/LCGCMT_30

12
Environment... check out packages

• Find the correct version of RecExCommon to check
  out for athena v9.0.3... look in the dist area,
  which is part of CMTPATH...
• echo CMTPATH
• /afs/cern.ch/user/c/ccollins/scratch0/lttutdirgt/af
  s/cern.ch/atlas/software/dist/9.0.3/afs/cern.ch/a
  tlas/offline/external/Gaudi/0.14.6.12pool/afs/cer
  n.ch/atlas/offline/external/LCGCMT/LCGCMT_30
• ls /afs/cern.ch/atlas/software/dist/9.0.3/Reconstr
  uction/RecExample/RecExCommon/

list this part of CMTPATH locate the packages
13
Environment... check out packages

• Find the correct version of RecExCommon to check
  out for athena v9.0.3... look in the dist area,
  which is part of CMTPATH...
• echo CMTPATH
• /afs/cern.ch/user/c/ccollins/scratch0/lttutdirgt/af
  s/cern.ch/atlas/software/dist/9.0.3/afs/cern.ch/a
  tlas/offline/external/Gaudi/0.14.6.12pool/afs/cer
  n.ch/atlas/offline/external/LCGCMT/LCGCMT_30
• ls /afs/cern.ch/atlas/software/dist/9.0.3/Reconstr
  uction/RecExample/RecExCommon/
• In principle, one could check out
  RecExCommon-00-02-99, BUT one must also check the
  S/W release page http//atlas.web.cern.ch/Atlas/G
  ROUPS/SOFTWARE/OO/Release/Status/
• So, check out RecExCommon-00-03-00... this is
  just like TestRelease, except tailored for
  Reconstruction and Analysis.
• cmt co -r RecExCommon-00-03-00 Reconstruction/RecE
  xample/RecExCommon
• We will also need a skeleton analysis package
  UserAnalysis
• cmt co -r UserAnalysis-00-01-10
  PhysicsAnalysis/AnalysisCommon/UserAnalysis
• NB the package versions can USUALLY be found
  from the dist area, but check the Release
  Status and atlas-sw-developers too!

list this part of CMTPATH locate the packages
14
Environment... fixes

• A few fixes
• As we will not be using CBNT and Muonbox is too
  slow, set doCBNT and doMuonbox to False in the
  file lttutdirgt/Reconstruction/RecExample/RecExCommo
  n/RecExCommon-00-03-00/share/RecExCommon_flags.py
• The TruthParticleAlgs package has a couple of
  small bugs, so check it out
• cd lttutdirgt
• cmt co -r TruthParticleAlgs-00-04-07
  PhysicsAnalysis/AnalysisCommon/TruthParticleAlgs
• Edit lttutdirgt/ PhysicsAnalysis/AnalysisCommon/Trut
  hParticleAlgs/TruthParticleAlgs-00-04-07/src/SpclM
  CParticleBuilder.cxx
• Replace a line as follows
• // sc m_tesIO-gtgetMC(particles, ifs) lt----
  remove
• sc m_tesIO-gtgetMC(particles, ifs,
  "TruthEvent") lt--- add
• Edit lttutdirgt/ PhysicsAnalysis/AnalysisCommon/Trut
  hParticleAlgs/TruthParticleAlgs-00-04-07/src/SpclM
  CAODBuilder.cxx.
• Change TWO lines- replace m_key with "TruthEvent"
  (you need the quotes!)
• StatusCode sc m_tesIO-gtgetMC(particles, ifs,
  m_key)
• if ( m_storeGate-gtretrieve(mcCollptr,
  m_key).isFailure() )
• Then we must tell our RecExCommon job to use the
  UserAnalysis and TruthParticleAlgs packages
• Edit lttutdirgt/Reconstruction/RecExample/RecExCommo
  n/RecExCommon-00-03-00/cmt/requirements
• Directly after the other use statements, add
  the lines
• use UserAnalysis UserAnalysis-
  PhysicsAnalysis/AnalysisCommon
• use TruthParticleAlgs TruthParticleAlgs-
  PhysicsAnalysis/AnalysisCommon

15
Environment... fixes

• A few fixes
• As we will not be using CBNT and Muonbox is too
  slow, set doCBNT and doMuonbox to False in the
  file lttutdirgt/Reconstruction/RecExample/RecExCommo
  n/RecExCommon-00-03-00/share/RecExCommon_flags.py
• The TruthParticleAlgs package has a couple of
  small bugs, so check it out
• cd lttutdirgt
• cmt co -r TruthParticleAlgs-00-04-07
  PhysicsAnalysis/AnalysisCommon/TruthParticleAlgs
• Edit lttutdirgt/ PhysicsAnalysis/AnalysisCommon/Trut
  hParticleAlgs/TruthParticleAlgs-00-04-07/src/SpclM
  CParticleBuilder.cxx
• Replace a line as follows
• // sc m_tesIO-gtgetMC(particles, ifs) lt----
  remove
• sc m_tesIO-gtgetMC(particles, ifs,
  "TruthEvent") lt--- add
• Edit lttutdirgt/ PhysicsAnalysis/AnalysisCommon/Trut
  hParticleAlgs/TruthParticleAlgs-00-04-07/src/SpclM
  CAODBuilder.cxx.
• Change TWO lines- replace m_key with "TruthEvent"
  (you need the quotes!)
• StatusCode sc m_tesIO-gtgetMC(particles, ifs,
  m_key)
• if ( m_storeGate-gtretrieve(mcCollptr,
  m_key).isFailure() )
• Then we must tell our RecExCommon job to use the
  UserAnalysis and TruthParticleAlgs packages
• Edit lttutdirgt/Reconstruction/RecExample/RecExCommo
  n/RecExCommon-00-03-00/cmt/requirements
• Directly after the other use statements, add
  the lines
• use UserAnalysis UserAnalysis-
  PhysicsAnalysis/AnalysisCommon
• use TruthParticleAlgs TruthParticleAlgs-
  PhysicsAnalysis/AnalysisCommon

16
Environment... compilation, set up

• Then, compile...
• cd lttutdirgt/Reconstruction/RecExample/RecExCommon/
  RecExCommon-00-03-00/cmt/
• cmt broadcast cmt config
• source setup.sh
• cmt broadcast gmake
• Set up some links, create and export
  PoolFileCatalog.xml...
• cd ../run
• source RecExCommon_links.sh
• (all on one line!!!) export POOL_CATALOGxmlcatalo
  g_file/afs/cern.ch/user/ltinitialgt/ltusernamegt/lttut
  dirgt/Reconstruction/RecExample/RecExCommon/RecExCo
  mmon-00-03-00/run/PoolFileCatalog.xml
• Get a few files we will need... and make them
  writable
• get_files RecExCommon_topOptions.py chmod w
  RecExCommon_topOptions.py
• get_files optRecExToESD.py chmod w
  optRecExToESD.py
• get_files optESDtoCombAOD.py chmod w
  optESDtoCombAOD.py

17
Generation (at last!!!)

• Again, in the run/ area type
• get_files jobOptions.pythia.py chmod w
  jobOptions.pythia.py
• Edit, according to physics (see pythia manual!).
  For now, we replace the PythiaCommand line with
• --------------------------------------------
• inclusive Z production with the decay forced to
  ee-
• --------------------------------------------
• Pythia Algorithm( 'Pythia' )
• Pythia.PythiaCommand 'pysubs msel 0', 'pysubs
  msub 1 1',
• 'pypars mstp 43 2', 'pydat3 mdme 174 1 0',
• 'pydat3 mdme 175 1 0', 'pydat3 mdme 176 1 0',
  
• 'pydat3 mdme 177 1 0', 'pydat3 mdme 178 1 0',
  
• 'pydat3 mdme 179 1 0', 'pydat3 mdme 182 1 1',
  
• 'pydat3 mdme 183 1 0', 'pydat3 mdme 184 1 0',
  
• 'pydat3 mdme 185 1 0', 'pydat3 mdme 186 1 0',
  
• 'pydat3 mdme 187 1 0'

18
Generation (at last!!!)
switch off Z/photon interference

• Again, in the run/ area type
• get_files jobOptions.pythia.py chmod w
  jobOptions.pythia.py
• Edit, according to physics (see pythia manual!).
  For now, we replace the PythiaCommand line with
• --------------------------------------------
• inclusive Z production with the decay forced to
  ee-
• --------------------------------------------
• Pythia Algorithm( 'Pythia' )
• Pythia.PythiaCommand 'pysubs msel 0', 'pysubs
  msub 1 1',
• 'pypars mstp 43 2', 'pydat3 mdme 174 1 0',
• 'pydat3 mdme 175 1 0', 'pydat3 mdme 176 1 0',
  
• 'pydat3 mdme 177 1 0', 'pydat3 mdme 178 1 0',
  
• 'pydat3 mdme 179 1 0', 'pydat3 mdme 182 1 1',
  
• 'pydat3 mdme 183 1 0', 'pydat3 mdme 184 1 0',
  
• 'pydat3 mdme 185 1 0', 'pydat3 mdme 186 1 0',
  
• 'pydat3 mdme 187 1 0'

create Z bosons
switch on Z-gtee, all other Z decays off (mdme 0)
19
Generation... contd

• Add the following lines for Pool
• General Application Configuration options
• -------------------------------------------------
  --------------
• theApp.setup( MONTECARLO )
• include( 'PartPropSvc/PartPropSvc.py' )
• the following lines are added for use of POOL
  persistency
• include('AthenaPoolCnvSvc/WriteAthenaPool_jobOptio
  ns.py')
• include('GeneratorObjectsAthenaPool/GeneratorObjec
  tsAthenaPool_joboptions.py' )
• 2101 EventInfo
• 133273 MC Truth (HepMC)
• Stream1.ItemList '2101', '133273'
• include( 'AthenaSealSvc/AthenaSealSvc_joboptions.p
  y')
• AthenaSealSvc.CheckDictionary TRUE
• Stream1.OutputFile 'pythia.pool.root'
• Set no. of events run number (default 0 causes
  problems)
• Event related parameters
• -------------------------------------------------
  -------------
• theApp.EvtMax 2
• Run identifier (default is 0)

20
Generation... contd

• Add the following lines for Pool
• General Application Configuration options
• -------------------------------------------------
  --------------
• theApp.setup( MONTECARLO )
• include( 'PartPropSvc/PartPropSvc.py' )
• the following lines are added for use of POOL
  persistency
• include('AthenaPoolCnvSvc/WriteAthenaPool_jobOptio
  ns.py')
• include('GeneratorObjectsAthenaPool/GeneratorObjec
  tsAthenaPool_joboptions.py' )
• 2101 EventInfo
• 133273 MC Truth (HepMC)
• Stream1.ItemList '2101', '133273'
• include( 'AthenaSealSvc/AthenaSealSvc_joboptions.p
  y')
• AthenaSealSvc.CheckDictionary TRUE
• Stream1.OutputFile 'pythia.pool.root'
• Set no. of events run number (default 0 causes
  problems)
• Event related parameters
• -------------------------------------------------
  -------------
• theApp.EvtMax 2
• Run identifier (default is 0)

output file name
21
Generation... contd

• Add the following lines for Pool
• General Application Configuration options
• -------------------------------------------------
  --------------
• theApp.setup( MONTECARLO )
• include( 'PartPropSvc/PartPropSvc.py' )
• the following lines are added for use of POOL
  persistency
• include('AthenaPoolCnvSvc/WriteAthenaPool_jobOptio
  ns.py')
• include('GeneratorObjectsAthenaPool/GeneratorObjec
  tsAthenaPool_joboptions.py' )
• 2101 EventInfo
• 133273 MC Truth (HepMC)
• Stream1.ItemList '2101', '133273'
• include( 'AthenaSealSvc/AthenaSealSvc_joboptions.p
  y')
• AthenaSealSvc.CheckDictionary TRUE
• Stream1.OutputFile 'pythia.pool.root'
• Set no. of events run number (default 0 causes
  problems)
• Event related parameters
• -------------------------------------------------
  -------------
• theApp.EvtMax 2
• Run identifier (default is 0)

output file name
22
Run event generation

• In the run/ directory
• athena.py jobOptions.pythia.py gt mygenZee.log
• Output here is pythia.pool.root and
  mygenZee.log
• NB we could have submitted this to LXBATCH
  (takes a long time for job to start, but job will
  not be killed after 40 mins!)
• Typical batch submission script (for generation),
  see (use rfdir) /castor/cern.ch/user/c/ccollins/sl
  c-fullchain903/small/doGEN.ksh
• Example submission to LXBATCH command (dont do
  this now!). From your run/ directory, youd type
  
• bsub -q 1nh doGEN.sh
• Tip when submitting to the LXBATCH for the first
  time, submit to the short 8nm queue (8 NCU
  minutes) to ensure the job does not have a silly
  error- it might fall over after you have waited
  in a PEND state for hours!

23
Simulation... set-up and run it

• This step is VERY slow...actual time depends on
  event complexity.
• For simple Zee events, takes 15mins/event, plus
  start-up time.
• We will do the sim process on LXPLUS, but it will
  be killed (40m)
• Much better to submit to the LXBATCH machines..
  see (use rfdir) /castor/cern.ch/user/c/ccollins/sl
  c-fullchain903/small/doSIM.ksh
• For now, run on LXPLUS and copy my output hits
  file.
• Preparations In the run/ directory
• get_files RomeSimulationConfig.py chmod w
  RomeSimulationConfig.py
• get_files RomeGeo2G4.py
• Edit RomeSimulationConfig.py
• comment out EvtMax and SkipEvent- (It crashed
  unless I did this!)
• replace the input and output file names and
  locations as follows
• PoolEvgenInput"pythia.pool.root"
• PoolHitsOutput "rfio/castor/cern.ch/user/ltinitia
  lgt/ltusernamegt/tutorialoutput/g4hits.pool.root"
• Run the simulation
• athena.py -c 'EvtMax 2' RomeSimulationConfig.py
  RomeGeo2G4.py gtg4hits.log

24
Simulation... get pre-prepared output.

• This command will be killed after 40 mins of
  execution on LXPLUS.
• You will not get the output file
  rfio/castor/cern.ch/user/ltinitialgt/ltusernamegt/tut
  orialoutput/g4hits.pool.root
• Instead, get my sample file
• cd lttutdirgt/Reconstruction/RecExample/RecExCommon/
  RecExCommon-00-03-00/run
• rfcp /castor/cern.ch/user/c/ccollins/slc-fullchain
  903/small/g4hits-slc903.pool.root ./
• Now, we must put this file into your
  PoolFileCatalog.xml
• (all on one line!!!) export POOL_CATALOGxmlcatalo
  g_file/afs/cern.ch/user/ltinitialgt/ltusernamegt/lttut
  dirgt/ Reconstruction/RecExample/RecExCommon/RecExC
  ommon-00-03-00/run/PoolFileCatalog.xml
• pool_insertFileToCatalog g4hits-slc903.pool.root
  
• (This will give you lots of Warnings about
  objects not being in the dictionary- just ignore
  them. If there are any FATAL or ERROR messages,
  these are a problem!)

25
Digitisation (without pile-up)

• Thankfully, this step is usually gt10 times faster
  than Sim.
• Due to batch lead-times, we will still do this
  interactively.
• To perform using LXBATCH, see (use rfdir)
  /castor/cern.ch/user/c/ccollins/slc-fullchain903/s
  mall/doDIGI.ksh.
• Make a small options file called
  myDigiOptions.py, and add the following lines (in
  run/myDigiOptions.py)
• PoolHitsInput "g4hits-slc903.pool.root"
• PoolRDOOutput "g4digi.pool.root"
• run through EvtMax events, if -1 run until end
  of file
• EvtMax -1
• Run the digitisation
• athena.py -b myDigiOptions.py Digitization/AtlasDi
  gitization.py gtg4digi.log
• The output will be g4digi.pool.root
• We are now ready to run the Reconstruction (make
  ESD).