Online Monitoring of L1CT in Run IIa

1
Online Monitoring of L1CT in Run IIa

• bonus experience from Run I
• Philippe Laurens 20-Feb-2003

2
L1 Cal Monitoring

• Monitoring can mean many things
• ? Anything related to observation of systems
  operation
• Different phases and different customers
• Commissioning/debugging
• a ton of work (probably more for Run IIa than Run
  I)
• will all have to be redone for Run IIb
• Check for run-readiness between stores
• Shifters Support
• watch for hot towers, non-uniformity, etc
• Instantaneous view of operation (programming,
  rates)
• Online Hardware Verifier
• Different time scales hours, weeks, months

3
Run IIa Monitoring Tools

• 1) Bob Kehoe's examine/offline analysis
• Analyze Events
• Physics runs or No Beam runs
• Look at noise in towers
• Compare to Precision Readout
• Detect problems
• Wiring, electronics, noisy channels, timing, etc.
• Problems are found in L1 hardware AND in BLS
  hardware
• Understand/improve
• energy scale, resolution, etc.
• Generate new L1CT gain constants
• Generate Plots for Shifters

4
Run IIa Monitoring Tools

• 2a) L1 TCC serving monitoring information
• Collects a fresh block of information every 5
  sec.
• From a triggered crossing, if possible
• EM and HD Et for 8 consecutive crossings
• and for all Trigger Towers.
• 2b) Simple display program
• ASCII dump of the data in each time slice
• or a coarse average of the 8 slices
• or a coarse standard deviation
• use colors to highlight ranges of values
• cf. "L1 Cal Trig Trigger Tower ADC Monitor" in
  trics_support_programs.txt in www.pa.msu.edu/hep/d
  0/ftp/tcc/trics_ii/

5
Run IIa Monitoring Tools

• 2) L1TCC monitoring information (cont.)
• This is how we currently get quick system
  appraisal after L1 or BLS hardware changes
• check all pedestals adjusted to 8 counts
• no obviously bad channels, etc
• ? system is (probably) ready for data taking
• L1CT sill missing its "CTRO" cards
• For L3 (and L2) readout and monitoring of final
  quantities
• Energy and Momentum Sums and Reference Set Counts
  carried over whole TT coverage.
• And-Or Terms sent to the L1FW.

6
Run IIa Monitoring Tools

• 3) Find_DAC
• Because L1CT adds constant voltage to each
  channel
• Using Pedestal DAC programmable by TCC
• To compensate for each trigger tower individual
  analog offset.
• To allow processing of negative energies
• Find_DAC tunes the Pedestal DAC to produce
  uniform 8 counts of digitized energy for analog
  input of zero GeV for all EM and HD Towers.
• This happens before the memory lookup for
  correction, low energy cut, Px/Py Scaling.

7
Run IIa Monitoring Tools

• 3) Find_DAC (continued)
• Runs on TCC independently of the DAQ system
• Scan through specified Trigger Towers
• Start with sanity check of front-end
• Uses fast VME access to build set of histograms
• Produces ASCII file later used to initialize
  system
• Also report amount of noise in each tower
• Companion tools to compare separate result files
• Helpful during commissioning to find problems
• Helpful during maintenance phase as diagnostics
  between stores

8
Run I Monitoring Tools Two types

• A) Low Level
• test the electronics by itself,
• prove that the system is operating as designed,
• independently of any beam physics.
• Independently of DAQ
• B) High Level
• look at the collected physics and special run
  data
• try and prove that it is all consistent with
  itself,
• has the right symmetries,
• the right distributions,
• the right physics, etc.

9
Run I Monitoring Tools Low Level Tests (BEFORE
beam)

• 1) L1CT system Test/Exerciser/Diagnostics
• Run on TCC between stores
• WITHOUT interfering with rest of DAQ system
• Randomly modify the simulated trigger tower
  energies AND randomly modifies the L1CT
  programmable COOR resources
• Built-in hardware assistance to change systems
  inputs for one single crossing and capture result
  of THAT Crossing in monitoring registers
• Test loop Compromise between changing more
  conditions and processing more crossings
• Configurable in TT coverage, what gets checked

10
Run I Monitoring Tools Low Level Tests (BEFORE
beam)

• 1) L1CT System Tester/Exerciser (continued)
• Crucial points to design such tests
• Maximize amount of electronics switching at once
• Hardware runs at full speed
• Capture response at first chance to process THIS
  information (as opposed to DC response as TCC
  gets around to reading thousands of registers).
• Needs hardware assistance built into the system
• Capturing multiple time slices is even better
• Monitoring data captures the switch to different
  inputs for one single crossing and go back to the
  background conditions

11
Run I Monitoring Tools Low Level Tests (BEFORE
beam)

• 2) Pulser runs for precision calibration of Cal
• Dan Owen wrote test program and led the L1CT
  commissioning effort for Run I.
• The pulser injects charge at pre-amp of Cal
  Front-End
• We see pulser patterns in the L1 CT data.
• Set the pulser to mid-scale on a few channels at
  a time
• Go through a list of pulser patterns
• Use mapping of calorimeter cells into trigger
  towers
• Predict where and about how much energy L1CT
  should see
• Find noisy channels, dead channels, mis-wiring,
  cross-talk, channels were half the differential
  signal was missing
• Cannot calculate exactly how much energy the L1CT
  should measure (Qualitative, not Quantitative)
  but LEARN how much we get then be alerted by
  departure from nominal.

12
Run I Monitoring Tools Low Level Tests (BEFORE
beam)

• 3) Find_DAC
• cf. Description for Run IIa
• Compare successive runs
• e.g. slow drift detect problems with L1CT
  analog front-end capacitors slowly going bad, and
  replaced them before they completely failed.

13
Run I Monitoring Tools Low Level Tests (BEFORE
beam)

• Usage
• Between stores, especially after we had worked on
  L1CT, or after BLS cards were fixed or replaced
• Run a number of loops of the L1CT Exerciser,
• take a special L1CT pulser run and analyze data.
• Know right then we are ready for physics
• First line of defense BEFORE next store, and
  before any physics runs.

14
Run I Monitoring Tools High Level Tests
(after-the-fact tests)

• 1) Analysis of L1CT readout data
• Kathy Streets led this effort
• calculate energy scale, saturation, resolution
• studied and watched L1CT on scale of weeks.
• Understood and improved L1CT energy scale over
  the course of Run I and replaced all 1280 hand
  made precision resistor networks
• (note energy gain is now programmable).

15
Run I Monitoring Tools High Level Tests
(after-the-fact tests)

• 2) Calorimeter Examine
• Jan and Joan Guida were also routinely building
  L1CT histograms to track the L1CT output as part
  of their watch of Cal electronics
• alert us when pedestals would drift
• or when we had noisy or dead towers
• Jan and Joan kept watching the system on the
  scale of months/years.

16
Run I Monitoring Tools High Level Tests
(after-the-fact tests)

• 3) L1CT verifier
• Amber and Jill Perkins put together a L1CT
  Verifier running on Examine
• An extension of L1SIM (L1FWL1CT simulator)
• Compare the computations made by the hardware
  L1CT to the results from the software simulator.
• run this program on a subset of the most recent
  data.
• e.g. alerted us of intermittent Missing Pt
  problem and we found a cable that had worked
  itself loose

17
Run I Monitoring Tools High Level Tests
(after-the-fact tests)

• 3) L1CT verifier (continued)
• Verifier was an important bit-level check of BOTH
  the simulator and the L1FWL1CT hardware,
  mutually against each other
• also confirmation of validity of trigger and MC
  studies based on L1SIM.