FEEDAQ Demonstrator

1
FEE/DAQ Demonstrator

• Walter F.J. Müller, GSI, Darmstadtfor the CBM
  Collaboration
• 3rd FutureDAQ WorkshopGSI, Darmstadt, October
  11, 2005

2
Mission Statement

• Provide a platform to
• demonstrate essential architecture elements of
  the CBM FEE-DAQ concept
• FEE self-triggered, data push, conditional RoI
  based readout
• CNet combined data, time, control, and RoI
  traffic
• TNet low jitter clock and synchronization over
  serial links
• BNet high bandwidth, RDMA based architecture
• E/DCS integrated approach for DCS/ECS
• provide test bed for all future FEE/DAQ
  prototyping in
• hardware
• firmware
• controlware
• software
• perform beam tests with detector prototypes
• form basis for medium-scale applications in
  intermediate-term experiments
• Be operational by end 2006
• avoid cathedrals, go for the bazaar, try and
  learn

3
System Layout
FEE boards
FEE
CNet links
CNet
DCB data combiner boards
TNet
Clock/Trigger distribution
CDL CBM detector links
ABB active buffer boards
BNet
EB Switch
Backend processors
HNet
General Network
to archive
4
G1 Demonstrator BNet
FEE

• Key architectural element in modern high speed
  data transport is RDMA
• e.g. Infiniband
• e.g. iWARP, iSCSI
• Separate
• data transport via zero-copy RDMA
• status management via reliable messages

CNet
TNet
BNet

• Proposal
• Architecture choice CBM BNet based on RDMA
  (e.g. uDAPL or other API)
• Implementation choice use InfiniBand

HNet
to archive
5
G1 Demonstrator TNet
FEE
CNet

• TNet proposal
• reuse CNet components
• TNet link proposal
• just a CNet link
• run unidirectional
• possibly use optical splitters
• TNet controler proposal
• just a DCB with other firmware

TNet
BNet
HNet
to archive
6
TNet Controller and Setup
......
......
DCB
DCB
DCB
DCB
passive optical splitter or as alternative another
layer of DCBs
Use DCB as TNet controller - different
firmware - little extra hardware - trigger
inputs - clock generator
DCB-TC
optionally connect to ABB
hardware trigger inputs and likely also 1-2
outputs
Trigger
DATA
7
Clock and Jitter Propagation
Bottom Line Clock distributed over 2 or 3 hops
......
DCB
DCB
DCB
DCB
Is a passive splitterfeasible or even desirable
?If not, an extra layer ofDCBs need for clock
fan-outfor more than 64 FEE boards
Use DCB as TNet controller - different
firmware - little extra hardware - trigger
inputs - master clock generator
DCB-TC
optionally connect to ABB
hardware trigger inputs and likely also some
outputs
Trigger
DATA
8
G1 Demonstrator FEE
FEE

• Get started with a
• general purpose
• chamber-mountable
• modest density
• ADC FPGA based
• board
• Plausible parameters
• 16 channels
• 100 MHz sampling
• 10 bit
• If needed, a pipeline TDC based FEE module to
  support Time-over-Threshold based systems might
  be added too

CNet
TNet
BNet
HNet
to archive
9
DCB Connectivity
bi-directional optical linkdata, trigger, RoI,
control, clockOptions OASE or V4 MGT
............
Link
Link
Link
Link
for processing use Virtex-4 - FX have PPC,
Ethernet MAC, MGT - but V-4 FX are hard to get
these days
FPGA
MEM
enough SDRAM memory to allow - operation of both
PPCs - data buffering
CPLD
NVRAM
support remote reconfiguration
Link
uni-directional optical link trigger, aux.
control, clock
Ether
Ether
SFP
SFP
primary data interface MGTSFP
Ethernet as control interfaceand auxiliary data
interface
CLK/TRG
DATA
CNTL/DATA
10
Micro Setup 1 DCB 1 PC
FEE
FEE
The minimal test bench setupmaximal 8 FEElocal
clock from DCBmodest data date via GE
PC
GE
11
Mini Setup Ethernet based
FEE
FEE
FEE
FEE
FEE
FEE
FEE
FEE
Next to minimal test bench setupfew
DCBclock/trigger from DCB-TCmodest data date
via GE
GE Switch
PC
PC
GE
GE
12
Target Setup Low Performance
FEE
FEE
FEE
FEE
FEE
FEE
FEE
FEE
FEE
FEE
FEE
FEE
GE Switch
GE Switch
PC
PC
PC
PC
GE
GE
ABB
GE
GE
GE
ABB
GE
GE
ABB
Use GE for event building
Controls
DAQ Event Building Selection
13
Target Setup High Performance
FEE
FEE
FEE
FEE
FEE
FEE
FEE
FEE
FEE
FEE
FEE
FEE
GE Switch
IB Switch
PC
PC
PC
PC
GE
HCA
ABB
GE
GE
HCA
ABB
GE
HCA
ABB
Use IB for event building
Controls
DAQ Event Building Selection
14
Status

• ABB/DCB Prototype
• wait for J. Gläß talk in a few minutes
• OASE Status
• see presentations of R. Tielert and S. Tontisirin
  on Wednesday
• DAQ Controls
• just heard H. Essel
• InfiniBand Test-Cluster
• 4 node cluster ordered, will be probably be
  delivered next week
• Contacts established to larger IB based clusters
  to allow scaling tests with realistic cluster
  sizes in the future.

15
The End
Thanks for your attention
We acknowledge the support of the European
Community-Research Infrastructure Activity under
the FP6 "Structuring the European Research Area"
programme (HadronPhysics, contract number
RII3-CT-2004-506078).