Data Acquisition Systems for Future Calorimetry at the International Linear Collider

1
Data Acquisition Systems forFuture Calorimetry
at the International Linear Collider

• Matt Warren, on behalf of CALICE-UK Collaboration

2
Introduction

• Building a DAQ for multiple ILC CAL sub-detector
  prototypes
• Paying attention real ILC environment
• EUDET Testbeam in 2009! (EU funded DETector
  project shares much of CALICE)
• For economies of scale, we are attempting a
  generic DAQ for many (even SLHC??)
• Modular/Generic Structure
• Generic readout system as much as possible
• Detector specific interfaces only at ends of
  chain
• Other bespoke functionality in firmware
• Commercial components and protocols where
  possible
• Readout links use standard connectors and
  protocols
• Based on PCs with PCIe cards (backplaneless)
• Clock and Control attempts commercial hardware
  too
• Extract clock and fast signals from commercial
  signalling
• Software generic for all detectors
• Try use something off-the-shelf
• BUT first, an introduction to the ILC CAL

3
ILC Calorimetry

• ILC Calorimetry is dense and high granularity
• squeezed between large tracker expensive coil
• gt100M channels
• No room for electronics or cooling.
• Bunch structure interesting
• 200ms gaps between bunch-trains
• Trains 1ms long, 300ns bunch spacing
• Triggerless sample data from every
  bunch-crossing
• SO (the problem)
• 100M channels, analog signals
• front-end electronics inside detector
• Results in high power density
• but no room for cooling
• Long gap allows chips a 1 duty cycle
• Solution Power Pulsing

HCAL
ECAL
M. Anduze
4
Sub-Detector Geometries (ASICs)
HCAL half-octant
ECAL Module-0 (reduced-Z octant)
M. Anduze
L 150 cm

• ASICS
• Must share readout resource (daisy chain)
• Bunch rate too high for instantaneous data
  transfer.
• Too much chip resource to store all events
• SO
• Auto-trigger store only data over-threshold
  with pad id (bunch-number)
• lt5kByte / bunch-train/ASIC

ASIC (gt100 in total!)
Detector Unit (e.g. ECAL Slab)
5
DAQ architecture

• Detector Unit Sensors ASICs
• DIF Detector InterFace -connects generic DAQ
  and services
• LDA Link/Data Aggregator fanout/in DIFs
  drive link to ODR
• ODR Off Detector Receiver PC interface for
  system.
• CC Clock Control Fanout to ODRs (or LDAs)

50-150 Mbps HDMI cabling
1-3Gb Fibre
LDA
CC
Detector
Counting Room
Storage
LDA
10-100m
0.1-1m
6
DIF (Detector InterFace)
? DAQ

• FPGA detector hardware connected to Detector
  Unit.
• Two halves Generic DAQ and Specific Detector
• 3 detectors ECAL, AHCAL, DHCAL
• 1 DAQ Interface!
• Focusing on the DAQ side
• From LDA, receive, decode/regenerate and
  distribute clocks, fast commands, config data and
  slow controls.
• From ASICs, receive, buffer, package and forward
  data to LDA
• ASICs power-up and read-out power-down in turn
• ALSO USB interface
• Hardware designers already have one
• DAQ plans to integrate for stand-alone testing

Detector Unit
DIF
USB
M. Goodrick
e.g. ECAL
7
DIF-LDA link

• Serial links running at multiple of machine clock
• 50Mbps (raw) bandwidth minimum
• robust encoding (8B/10B)
• HDMI cables/connectors interface.
• Commercially available cables
• Rated gt300Mb
• Even halogen free available
• Signals (ideally just TX/RX but )
• ?Clock (diff)
• ?Control/Fast (diff)
• ?Very Fast (diff)
• ?Data (diff)
• ??single ended aux x2 (or UTP)
• LDAs serve even/odd DIFs for redundancy

LDA
LDA
8
LDA (Link/Data Aggregator)

• Located as close as possible to DIFs
• Shortest cables, but convenient location (space,
  cooling)
• Supports as many DIFs as possible considering
  bandwidth and physical constrains
• Ideally 50 (20Mbps/DIF)
• Prototype will have 10
• Aggregates front-end data and sends it
  off-detector
• Fibre optic link. 1-3Gbps, with SFP (see next
  slide)
• Fanout CCC to DIFs
• USB interface for stand-alone/top-of-chain
  testing

LDA
LDA

• Currently using a commercial FPGA dev-board
• Enterpoint Broaddown2 Xilinx Spartan3-200
• With add-on boards for our needs
• SPFSerDes for ODR link
• 10 HDMI connectors with clock fanout

9
ODR (Off Detector Receiver) Link

• Receives module data from LDA
• PCI-Express card, hosted in PC.
• 1-4 links/card (or more), 1-2 cards/PC
• Buffers and transfers to store as fast as
  possible
• Fibre optic link to detector via SFP modules (std
  networking hw)
• Currently GigE (1.25Gb), but could higher and use
  different proto.
• Sends controls and config to LDA for distribution
  to DIFs
• Interfaces to CC for synchro running
• Goal to send clock and prompt controls over optic
  link too
• Reset and reprog FPGAs
• Performance studies optimisation on-going (see
  next slide)
• Bottleneck in writing data to disk.

CC
Storage
Expansion (e.g. 3xSFP)
SFPs for optic link

• Hardware
• Using commercial FPGA dev-board
• PLDA XPressFX100
• Xilinx Virtex 4, 8xPCIe, 2x SFP (3 more with
  expansion board)
• Our own firmware and Linux driver software

10
ODR Throughput Measurements

• All measurements single requester thread, single
  IO thread (disk write),
• Each event fragment written to a separate file.
  Data written to the localdisk (fs ext3)
• e.g. WORST CASE!

Ethernet frame size
Problem with test!
A. Misiejuk
NDG Network Data Generator IDG Internal (ODR)
Data Generator
NDG plot between two separate machines,
Gigabit, copper Eth on both sides
11
Clock Control

• CC unit provides machine clock and fast signals
  to 8x ODR/LDA.
• Logic control (FPGA, connected via USB)
• Command encoders
• Remote signal enable, clock selection
• But capable of stand-alone, dumb mode
• Provision for async scintillator type signals
  (VFast)
• LDA provides next stage fanout to DIFs
• Eg CC unit -gt 8 LDAs -gt 10 DIFs 80 DUs.
• Signalling over same HDMI type cabling
• Facility to generate optical link clock
  (125-250MHz from 50MHz machine clock)

LDA
Machine
CC
Run-Control
LDA

• Commercial systems are not ideal here.
• Looking at custom protocol on fibre optic link
• Prompt signals and low jitter clock recovery
  needs further investigation

12
Software and Operation

• Software
• Looking for OTS software to cover both slow and
  fast controls
• Early days examining EPICs, ACE and DOOCS (the
  current favourite)
• - DOOCS is open source, actively developed, slow
  and fast controls, and already used by ILC
  community
• Operation
• Two modes
• 1) Configure PCs controlled over network to send
  configuration to LDAs and DIFs
• 2) Run
• LDA/DIF set to data-taking mode
• ODR configured for data reception, control handed
  to central.
• OR ODR needs no control simply waits
• Bunch-train starts/stop signals sent to LDAs
  control data flow.
• Fairly autonomous system (i.e. no trigger!)

13
Summary (an example AHCAL)
Off-Detector DAQ
LDA
DIF
Detector Unit
P. Göttlicher, DESY