HCAL Readout and DAQ using the ECAL Readout Boards

1
HCAL Readout and DAQ using the ECAL Readout Boards

• Paul Dauncey
• Imperial College London, UK

2
ECAL readout electronics overview

• CALICE ECAL has 30 layers, 18?18 channels/layer,
  9720 total
• Each gives analogue signal, 14-bit dynamic range
• Very-front-end (VFE) ASIC (Orsay) multiplexes 18
  channels to one output line
• VFE-PCB handles up to 12 VFEs (216 channels)
• Cables from VFE-PCBs go directly to UK VME
  readout boards

3
VFE-PCB control and timing
VFE-PCB needs several control and timing signals

• Analogue 12 differential input, 2 differential
  output from readout board
• Digital 4 LVDS input, 9 LVDS output from readout
  board, plus 7 uncommitted LVDS spares (could be
  in or out assume out here)

4
Readout board overview

• Readout board structure
• Front End (FE) FPGA controls all signals on
  VFE-PCB cable
• Back End (BE) FPGA gathers and buffers all event
  data from FE and provides interface to VME
• Trigger logic in BE for timing and backplane
  distribution only active in one board
• First readout board prototypes this summer, final
  boards end 2003 cost per board 10k

5
Readout board features

• Board clock 40 MHz
• Almost all signals edges timed to 25ns periods
• Sample-and-hold timing needs to be better than
  10ns
• Effectively the VFE-PCB trigger VFE shaping time
  of 180ns sets maximum trigger latency also
• Fine-tune rising edge of signal on ?8 clock, 3ns
  steps
• Each cable can be timed independently
• Dual 16-bit ADCs and 16-bit DAC
• DAC fed back for internal as well as VFE
  calibration
• No data reduction planned in hardware
• Read out all ECAL channels for every event
• 2 Bytes ? 1728 channels/board 3.5 kBytes/board
• 2 Bytes ? 9720 channels 19 kBytes total
• On-board buffer memory 8 Mbytes
• Allows up to 2k events for ECAL during beam spill

6
Trigger/DAQ logic
Idea is to implement trigger/DAQ interface for
all subsystems
7
Trigger handling

• Multiple (4) trigger and activity (background)
  inputs
• Can be enabled and disabled through VME
• Need clean events with no pile-up
• Activity prevents subsequent triggers within
  configurable time period
• Trigger records following activity read out with
  event
• Trigger latches off logic, preventing further
  triggers
• Reset through VME single reset for system so no
  synchronisation issues
• Trigger is fanned out and sent to backplane
  connector
• Distribution to other boards in crate using
  custom cable
• Point-to-point LVDS (probably)
• Trigger also configurably delayed and output to
  connector
• For distribution to other systems (beam
  monitoring, HCAL?)
• Assuming 16 outputs, LVDS we have a LVDS?NIM
  converter available
• Beam on signal allows buffering during spill,
  readout after spill

8
Motivations for reuse of readout boards

• We have tried to keep design flexible enough to
  allow readout boards to be used for HCALs also
• Assuming the on-detector electronics is
  compatible
• Aim for single crate (or at least a single DAQ
  PC) if possible
• No need for inter-PC communication
• Single VME interface reduces amount of code
  needed
• Makes offline analysis more uniform for different
  HCALs also
• Trigger use and distribution is more uniform
• Particularly if single VME crate
• Common buffering method if not reading every
  event before next trigger
• One downside may be less flexibility during
  testing before beam
• (Semi-)custom 9U crate (CMS tracker
  standard)would need to buy specific crate for
  tests 6k
• Is rate high enough with single VME crate? See
  later

9
Use for analogue HCAL

• Tile AHCAL has 1500 analogue channels (AFAIK)
• Each ECAL readout board has 96 ADCs 16 boards,
  need 2 crates
• Multiplex AHCAL to fit into smaller number of
  boards?
• E.g. 8-fold multiplexing could fit into two
  readout boards
• Are systems compatible?
• Can something like twelve (multiplexed) signals
  be fed into each cable?
• Is multiplex control compatible with number of
  LVDS signals available?
• What is maximum allowed trigger latency for
  AHCAL?
• Are ADC input, DAC output ranges compatible?
• Trivial if using (slightly modified) VFE chip
• But who is designing equivalent of VFE-PCB?
• Otherwise would probably need (non-UK) firmware
  development
• FE FPGA only rewrite cable signal handling, keep
  BE interface
• BE and VME would be identical retains common VME
  interface

10
Use for digital HCAL

• Digital HCAL has 350k channels (AFAIK)
• Assume zero suppression, as data volume otherwise
  is high (45 kBytes/event), and data concentrator
  on detector
• Use LVDS lines for clock, trigger, configuration,
  control and data readout of concentrator
• What is maximum allowed trigger latency for
  DHCAL?
• Clearly requires FE firmware development, as for
  AHCAL
• But BE and VME are still uniform across system
• Are the number of lines compatible with
  concentrator interface?
• Minimum would be clock, trigger, serial in,
  serial out per concentrator
• Each cable could handle 4 concentrators, i.e. 12
  outputs, 4 inputs per readout board, so each
  readout board handles 32 concentrators
• E.g. with 1k channels per concentrator, see talk
  by Gary Drake Feb 6, this requires 11 readout
  boards (What sets number of channels per
  concentrator?)
• Leaves three free VME slots in the crate for beam
  monitoring readout

11
Transfer rates to readout board

• ECAL
• 18 channels multiplexed per line all lines read
  in parallel
• ADC maximum rate is 500kHz actually needs around
  3ms per channel
• Total time around 50ms per event
• Analogue HCAL
• With multiplexing at or below 18 channels per
  line, will be less than or equal to the above
  (assuming able to multiplex output at 500kHz)
• Digital HCAL
• Time set by maximum data volume of all
  concentrators for each event
• For 4 Bytes/channel, assume average maximum data
  volume per concentrator is 1 kByte 250 channels
  (pessimistic?)
• With single serial line at 40 MHz, rate is
  40Mbits/s 5 MBytes/s, so time required is 200ms

12
Modes of operation

• Readout board can run in any of three modes
• Single trigger readout
• Read all data between each trigger slow but sure
• Semi-buffered readout
• Read minimal information from each board per
  event e.g. trigger number and buffer occupancy
• Must read any unbuffered electronics for each
  event
• Buffer rest of data and read later (after beam
  spill)
• Fully-buffered readout
• Nothing read from readout boards per trigger
  only unbuffered electronics
• All readout board data buffered until end of
  spill
• A missed trigger corrupts whole spill of data
• Semi-buffered is to avoid the need for a more
  sophisticated fast control and timing system

13
DAQ readout rates

• Assumptions
• Average event data ECAL 19 kBytes, DHCAL 5
  kBytes (?) (AHCAL 3 kBytes), other (beam
  monitoring, etc) 2 kBytes (??)
• Average time for clear/trigger/interrupt from end
  of one event to start of read of next 0.1ms.
  Implies beam rate gt10kHz
• VME data speeds non-block transfer 4 MBytes/s,
  block transfer 16 MBytes/s
• Semi-buffered readout of ECAL and HCAL is 400
  Bytes per event total, read out with non-block
  transfer so takes 0.1ms
• All beam monitoring, etc, data is always
  unbuffered and is read out with non-block
  transfer so takes 0.5ms per event
• Significantly bigger than data transfer time to
  readout board of 0.2ms
• Transfer time is not a limiting factor
• Disk write speed 20 MBytes/s always
  outperforms VME

14
DAQ readout rates (cont)

• Rates for the different modes
• Single trigger 24kBytes block transfer 1.5ms,
  total 2.3ms per event rate limited to 430Hz
• Semi-buffered 400 Bytes takes 0.1ms so 0.9ms per
  event during spill rate limited to 1.1kHz. 1.5ms
  per event outside of spill rate limited to 670Hz
• Fully-buffered Only time saved is readout of 100
  Bytes, so 0.8ms per event during spill rate
  limited to 1.2kHz. 1.6ms per event outside of
  spill rate limited to 630Hz
• No significant gain from using fully-buffered
  mode over semi-buffered mode
• Simpler trigger, better error detection and event
  verification in semi-buffered mode

15
Total data sample sizes

• Want to save unsuppressed data for noise and
  pedestal studies but also want processed data for
  analysis
• Raw event size 26 kBytes (assuming no
  significant gzip compression)
• Processed event size 4 kBytes (assuming DHCAL
  can be reduced?)
• Total disk space needed per event 30 kBytes
• Want to run in many configurations
• Particle type, energy, HCAL, preshower, angle,
  etc 100 configurations
• Need 106 events per configuration
• Total event sample 108 events 3 TBytes
• Just brought 3.2 TBytes for BaBar for 6k
• We have budgeted 12k for DAQ disk
• Processed data is 400 GBytes
• DESY have said they will host data long term

16
What if no zero suppression in DHCAL?

• How would these numbers change if DHCAL raw bits
  are read out?
• Event data size is now fixed at 45 kBytes per
  event
• Transfer time to readout board reduced to 0.03ms
  as now always transfering 1kbit 128 bytes
• Buffer memories will contain 45 kBytes spread
  over 11 boards, i.e. 4 kBytes per board, so can
  still buffer 2k events
• VME readout modes single trigger becomes 210Hz,
  semi-buffered and fully buffered become 250Hz out
  of spill but are still above 1kHz during the
  spill
• The total event size is 66 kBytes per event,
  giving a total data sample size of 7 Tbytes
  processed data size is unaffected
• Could always do zero suppression in DAQ software
  so disk space as before
• Worth considering if this simplifies (and reduces
  cost) of on-detector electronics significantly

17
Summary

• With some firmware effort, the ECAL readout
  boards may be able to be used for both the AHCAL
  and DHCAL options
• Savings in software effort, code uniformity,
  trigger distribution, etc.
• Rates of around 1kHz during a spill, 600Hz
  outside of a spill would be possible
• Buffering up to 2k events during the spill
  on-board
• Data volumes would be large but affordable
• Even if DHCAL is read out unsuppressed
• BUT important to cross check the assumptions
  made here are realistic
• Compatibility of different HCAL readouts
• Amounts of HCAL data
• Beam spill timings