The%20CMS%20HCAL%20Data%20Concentrator:%20A%20Modular,%20Standards-based%20Design

1
The CMS HCAL Data ConcentratorA Modular,
Standards-based Design
A Review of the CMS HCAL DAQ

• Gueorgui Antchev, Eric Hazen, Jim Rohlf,
  Shouxiang Wu
• Boston University
• Drew Baden, Rob Bard, Rich Baum, Hans Breden,
  John Giganti, Tullio Grassi, Aaron McQueen, Jack
  Touart
• University of Maryland
• Mark Adams, Kyle Burchesky, Weiming Qian
• University of Illinois, Chicago
• Wade Fisher, Jeremy Mans, Chris Tully
• Princeton University
• J. Elias, T. Shaw, J. Whitmore
• Fermi National Laboratory

2
Outline

• HCAL Trigger DAQ Overview
• Requirements
• Overview Diagram
• DAQ Crate
• HCAL Trigger Readout Card (HRC)
• Requirements
• Block Diagram
• Photo
• HCAL Data Concentrator (DCC)
• Motherboard
• Link Receiver
• Logic Board
• HCAL Readout Controller (HRC)
• Data Format
• System Tests
• Summary

3
HCAL Trigger/DAQ Requirements

• Digitize scintillator pulses every BX
• Pulses occupy 3 BX periods (25ns each)
• Dynamic range required is about 104 (14-15 bits)
• QIE (custom floating point ADC) used
• Reliably assign energy measurement to a single BX
  for Level 1
• F.I.R. filter implemented in FPGA (ala FERMI)
• Synchronized across HCALECAL for input to Level
  1
• Transmit data to DAQ on L1A
• Zero Supression
• Formatting

4
HCAL FE/DAQ Overview
DAQ Crate (in UXA)
Readout Box (RBX) (On detector)
Raw Front-End Data (One Charge sample per Bunch
Crossing)
D C C
H T R
H T R
H R C
H TR
H T R
HPD
T T C
Triggered L2 Data
Trigger Primitives
DAQ
Level 1 Trigger
5
HCAL Channel Counts
Region Towers GOL Fibers Trigger Towers Crates
Barrel 2,160 720 2,160 6
Outer 2,160 720 0 6
Endcap 2,160 720 1,728 2
Forward 2,016 672 144 6
Overlap 864 288 144 6
Total 9,360 3,120 4,176 26
6
HCAL DAQ Crate

• 9U x 400 VME64x Crate
• Controller (HRC)
• Commercial CPU/Bridge
• TTC Fanout
• Readout Cards (HTR)
• Front-end data in (GOL)
• Level 1 Out (Vitesse Cu)
• Level 2 Out (LVDS)
• Data Concentrator (DCC)
• Level 2 in (LVDS) x 18
• S-Link Out x 2

Front End QIE ADC GOL Optical Link Tx
Secondary DAQ For Trigger
3 channels/fiber _at_ 1.6 Gbps
D C C
H T R
H T R
H R C
H TR
H T R
LVDS 80MB/s
S-Link 64 Generic Links
T T C
1 Gb/s Vitesse Cu
Primary DAQ
Level 1 Trigger
7
HCAL Trigger Readout Card (HTR)

• Level 1 (Trigger) Path
• Energy Sum and Bunch Crossing Determination
• Output Trigger Primitives to Level 1 synchd to
  BX
• Level 2 (DAQ) Path
• Zero-supress Level 2 data (energy filter)
• Package raw data plus trigger primitives
• Send to Data Concentrator every L1A
• Implementation
• FIR filter for bunch crossing (ala FERMI)
• Single large FPGA for core logic

8
HCAL Trigger Readout Card
Crate Fanout of BC0, Clock
TTCRx
Local BC0, Clock
Level 1 Processing
Serial Link Board
FE Data
Trigger Primitives To Level 1 Trigger
Level 1 Processing
8 or 16 Fibers 24 or 48 Channels
Level 1 Processing
Energy Sum, BX Identification
Level 2 Pipeline
Level 12 Data To DCC
L1 Delay pipeline, Derandomizer, Zero-Suppression,
Data formatting
9
HCAL Trigger Readout Card
Timing Inputs (TTC)
VME Interface Config. (FPGA)
FE data Inputs 8-16 digital serial fibers 16-72
Trigger Towers 800 Mb/s - 1.6 Gb/s
DESER
OPTICAL RX
P1
DESER
...
DESER
LVDS TX
Level 1 and Level 2 data to DCC
FPGA Trigger Path DAQ Path
P2
16 bit x 40MHz
SLB-PMC (ECAL design)

Trigger Primitive outputs 1Gb/s copper link
SLB-PMC (ECAL design)
10
HTR Demonstrator Board
4 G-Link Fiber Receivers
TTCrx Board
Altera APEX Logic FPGA
Trigger Output Mezzanine Card
Status Prototypes Built/Tested New Design using
GOL Links underway
11
HCAL Data Concentrator
TTCrx
DAQ Out
Slink 64
From HTRs
Trigger Data Out
LVDS Serial 80MB/s
Slink
Spy Out
VME
Fast Status
12
HCAL DCC Prototype Architecture
Overflow Warning Fast Busy To TTS
13
HCAL DCC Motherboard
PC-MIP Card
PMC Card (oversize)
3-Channel LVDS Receivers
T
33MHz
PCI bus 1
M
PC-MIP Card
3.3V
T
32
DCC Logic Board
PC-MIP Card
Universe II
VME
VME-PCI
64x
T
PCI
Bridge Y
PC-MIP Card
3.3V
M
T
33MHz
PCI bus 2
32
PC-MIP Card
64
T
PCI
Bridge X
T
33MHz
PC-MIP Card
PCI bus 3
5V
5V
T
PCI busses give access to all devices for
monitoring
PMC Card
M/T
T
(Standard)
JTAG
Local
Spare PMC Site
Test/Config
Control
14
HCAL DCC Motherboard
PC-MIP sites
Universe II VME-PCI Bridge
VME64x (Geographical Addressing)
PMC Connectors
PCI Bridges T.I. PCI2031
Status 10 Producedand Tested. No known problems
Local control Flash memory, JTAG
15
PC-MIP 3 Channel Link Receiver

• 3 Channel Link LVDS receivers
• PCI target interface
• On-board logic
• ECC (Hamming) plus parity
• Correct 1-bit, detect multi-bit errors
• On-the-fly Event Building
• Event number checking
• Overflow warning (discard data payload on
  overflow)
• Missing header/trailer detection repair
• Monitoring
• Count of words, events, errors
• Status update on marked event for
  synchronization of monitoring
• Status 20 second-generation prototypes build
  (design is done)

LVDS Serial Receivers
512k byte buffer Per channel
(i.e. National Semi "Channel Link)
SSRAM 512kx32
DS90CR286
Link A
PCI Bus
ALTERA
(32 bit 33 MHz)
DS90CR286
Link B
(ACEX 1K130)
DS90CR286
Link C
FPGA
(PCI Interface and
readout/monitor logic)
Logic Board
Link Receiver Board(s)
1
2
3
1
2
3
1
1
2
3
Event Fragments
combined during
PCI bus transfer
Event Fragments
16
PC-MIP 3-Channel Link Receiver
Altera ACEX FPGA EP1K100FC484-1
PC-MIP Interface (PMC 33MHz 32 bit)
Programming Connector For flash memory
MT55L512V ZBT 512Kx32 SSRAM
DS90CR256A Channel-Link RX
Cypress Clock Multipliers
17
DCC Logic Board
L1AFIFO
TTCRx
2 Events max
Level 2FIFO
200 Mbytes/s Average
Data Decoder
100MHz Processing
Level 1FIFO
PCI 1
DAQFIFO
Event Builder
SummaryFIFO
TriggerFIFO
33MHz x 32 bits Two busses
Level 2FIFO
Data Decoder
SpyFIFO
Level 1FIFO
PCI 2
SummaryFIFO
Monitoring (via VME to CPU)
18
DCC Prototype Logic Board

• Features
• On-board TTCrx controls operation
• 3 Altera FPGAs for PCI interfaces (use Altera
  PCI-MT/32 core)
• Xilinx Virtex-2 contains all other logic
• Event Builder
• Monitoring
• Buffering (DDR SDRAM interface at 800Mbytes/s)
• S-Link output (32 in demo 64 final)
• On-board flash memory for FPGA initialization
• JTAG Interface
• Status
• Prototype PCB Working
• Continuous DAQ transfer at 80MB/s demonstrated
  (one PCI bus only)
• 2nd Prototype layout done

JTAG
TTCrx
S-LINK 64
XC2V1000
S-LINK
PCI 1
PCI 3
PCI 2
2Mx32 SDRAM
1Mx8 FLASH
19
DCC Prototype Logic Board
PCI-1 32 bit 33 MHz HTR inputs 1-9
PCI-3 64 bit 33 MHz
TTCRx PCB (back)
DDR SDRAM
Xilinx XC2V1000
PCI-2 32 bit 33 MHz HTR inputs 1-9
20
DCC Prototype Logic Board
TTCRx PCB
S-Link 32 LSC
21
Data Concentrator Demonstrator
LVDS Serial Link Receiver
TTCRx
S-Link LSC (Transmitter)
DCC Logic Board
Spare PMC Sitefor Testing
22
HCAL Readout Controller (HRC)

• Run Control
• Initialization, shutdown
• Slow monitoring via VME
• Error recovery
• Monitor status registers of modules via VME
• Report serious errors via DCS
• Reset/Restart on command
• TTC Fanout
• Fanout encoded TTC to all modules
• Fanout Locally Decoded CLK, BC0 to HTR modules
• I2C Control of local TTCrx

Custom 6U/9U Adapter
6U VME CPU Or Bus Bridge
LVDS (3 pairs) To HTRs And DCC
LVDS Fanouts
I2C
TTCrx
From TTC Optical F/O
23
HCAL DAQ Data Format

• Data format follows TriDAS Guidelines ? ? ?
• HCAL payload (details t.b.d)
• Raw QIE (ADC) samples
• Level 2 Filter output
• Trigger Primitives
• Zero-suppression mask
• Error summary
• Front-end errors
• Uncorrected Link errors
• Synchronization errors
• We will stay tuned for updates to the data format
  ? ? ?

24
System Tests

• Radioactive Source Calibration Test
• NOW at Fermilab
• Record data at 80Mbyte/s for detector calibration
• Uses demonstrator hardware for all system
  components and verifies basic functionality
• Test Beam Summer 02 at CERN
• Record data at realistic LHC rates
• A few hundred channels
• Uses (2nd) prototype hardware for all components
• Verify high-rate operation under realistic
  conditions

25
Summary
HCAL RBX

• Front-End
• RBX Mechanics/Cooling DesignedPrototyped
• Readout Card prototypes under test
• HTR
• 6U Demonstrator prototypes working
• New prototype with GOL under design
• DCC
• 9U Demonstrator prototypes working
• No major changes anticipated for production
  version (significant FPGA coding remains)
• HRC
• Use commercial CPU for now
• TTC Fanout Design Done (U.I.C.)
• Major Concerns
• QIE Performance(pending prototype tests now
  underway)
• Performance of 1.6 Gbit GOL link
• HTR/DCC FPGA Design Quite complex

HTRDemonstrator
DCC Demonstrator
VME Pentium CPU
26
Backup Slides
27
HCAL DAQ Buffering
HTR
DCC
128kb FIFO ?1000 events
40MHz
L1A
(2) 33MHz 32 bit PCI busses
8Mb Buffer gt 4000 events
100MHz Processing
PCI
Event Builder
PCI 32/33
PCI
Link Rx
S-Link64 Interface
Derandomizer Buffer Protected against overflow by
trigger rules
Link RX logic discards data when FIFO almost
full (block structure maintained)
S-Link LSC
LVDS link speed same as HTR output logic ? no
bottleneck 18 Links per DCC
Overflow Warning
Busy/ Ready
Level 2 Data To RUI
To TriggerThrottling System
28
HCAL Timing / L1A Distribution

• Fanout Both
• Encoded TTC signal
• For synchronization with incoming FE data
  (individual skew control)
• Decoded BC0, CLK
• For synchronization across all HCAL and ECAL of
  TPG to Level 1
• Similar to ECAL solution (J. Carlos Dasilva)

TTC Fanout (1 per crate)
PIN Diode
TTCrx
Decode FPGA
LVDS F/O
LVDS F/O
HTR Cards
BC0, CLK (TPG)
BC0, CLK (FE)
BC0, CLK (TPG)
BC0, CLK (FE)
29
HCAL Controls and Monitoring

• Fast Controls (via TTC)
• L1A, Start Run, Stop Run
• Reset (complete and partial need to define!)
• Fast Monitoring (dedicated signals to TTS)
• Overflow Warning (buffer full above preset limit)
• Busy/Ready (reset, start/stop completed)
• Slow Monitoring (link errors, loss of sync, etc)
• Counters in FPGAs collect information in real
  time
• Reported via CPU

30
HTR Details

• Input
• QIE 7 bit floating plus 2 bits ?cap?
• Lookup table (LUT)
• Convert to 16 bit linear energy
• Pipeline (?Level 1 Path?)
• Transmit to Level 1 trigger, buffer for Level 1
  Accept, 3 ms latency
• Level 2 Buffer (?Level 2 Path?)
• Asynchronous buffer, sized based on physics
  requirements

31
PCI Development

• PC-MIP Cards
• Use adapter in standard PC motherboard
• LRB Prototype was completely developed before
  motherboard
• VME Motherboard
• Test all sites with standard PC-MIP and PMC cards
• Integration
• Integration of motherboard and mezzanine cards
  was quite smooth
• PCI Interface logic
• Use Altera Core (motherboard and logic boards)
• Own design simple slave for LRB

PCI Test Setup
PC-MIP to PCI Adapter (Commercial)
PC-MIP card Under test
Standard PC Motherboard
32
Flash ADC Quantization