GlueX Data Acquisition

1
GlueX Data Acquisition

• Graham Heyes May 2008

2
Overview

• Background
• Requirements
• Challenges
• Solution
• Status

3
Background

• CODA is a data acquisition framework for all
  halls.
• Version 1 of CODA developed in early 1990s.
• Used in detector tests and hall-C startup.
• Version 2 of CODA based on lessons learned and
  targeted at hall-B.
• Version 3 is based on lessons learned from
  earlier versions and is targeted at 12 GeV
  operation.
• Current implementations in halls A, B and C.
• The CLAS detector is the most challenging 6 GeV
  detector. Currently operating at
• 8 kHz event steady state rate.
• 10 kHz peak (limited by old Fastbus ADCs).
• 35 MBy/s to storage at 8 kHz.

4
CODA 2 data acquisition
5
(No Transcript)
6
CODA version 2 limitations

• Scalability
• Up to 32 front ROCs.
• One EB.
• One ER.
• Throughput
• Event-by-event trigger, one CPU interrupt per
  event.
• Single data path.
• Single event building thread.
• Archaic code
• CODA version 2 uses obsolete third-party
  software.
• Code was written prior to multi-core and 64-bit
  architectures.
• Code was written before languages like Java
  reached maturity.

7
12 GeV Requirements

• Rates
• Low Luminosity
• 20 kHz L1 trigger rate
• up to 20 kB average event size
• 400 MB/s off detector to event builders
• no L3 rejection, prototype farm for monitoring
  only
• 400 MB/s to storage
• High Luminosity
• 200 kHz L1 trigger rate
• up to 20 kB average event size
• 4 GB/s off detector to event builders
• x10 L3 rejection, full farm for rejection and
  monitoring
• 400 MB/s to RAID

8
12 GeV requirements

• Long lead time for final system.
• Dont want to lock down some technologies too
  early.
• Short lead time for DAQ test systems.
• Scalability
• Single crate testbed
• Approximately 60 (or more) crate system.

9
Challenges

• Front-end
• 200 kHz trigger rate.
• (only a few MB/s average data rate per ROC).
• Back-end
• partial events from 100 ROCs at 200 kHZ 2x106
  partial events/s.
• 2 GB/s throughput.
• Storage
• 200 MB/s to storage.

10
Data transport
11
Solutions, Front-end

• Proposal
• Internal buffers on F1 TDCs and Flash ADCs.
• Hold up to 256 events before readout required.
• Pipelined trigger system.
• ADCs and TDCs triggered on interesting events.
• ROC triggered to read out entire blocks of
  events.
• Advantages
• ROC trigger rate is scaled relative to true
  trigger.
• Front-end and back-end software introduces no
  dead-time if ADCs and TDCs can be read out before
  next trigger.
• Technology
• ADCs and TDCs designed at JLab.
• Readout using VME.
• ROC code running on fast embedded Intel processor.

12
Penalty for pipelined readout

• Traditional system
• Each event generates a trigger
• Boards read to form partial events.
• Partial events stacked up and sent to Event
  Builder
• Pipelined system
• ROC triggered every N events.
• Each board contains data from N events.
• Data from boards stacked up to form blocks sent
  to Event Builder.

13
Back-end, introducing the EMU

• DAQ system is a chain or mesh of components.
• To reduce coding effort and increase flexibility
  design one generic component that can be
  customized at run time.
• DAQ system scales by adding components in series
  or parallel.
• parallel increases bandwidth.
• series breaks complex problems into steps.
• Each generic has four core functions
• It can be controlled (and by implication be
  monitored).
• It reads data from somewhere.
• It processes that data in some way.
• It sends the processed data somewhere.

14
Example CODA 3 system
15
How does the EMU solve our problems?

• High number (2x106) partial events per second
• Use EMUs in series and parallel to divide event
  building into manageable chunks.
• High data rate (2 GB/s)
• Use strings of EMUs in parallel.
• High storage rate (200 MB/s)
• Use parallel EMUs configured as data writers.
• Small DAQ test systems and the short term needs.
• Use EMUs in the classical CODA version 2
  ROC-EB-ER model
• Long lead time for final system
• configuration can be mapped onto future hardware.
• data transport and other hardware dependent
  software is decoupled from the EMU by an abstract
  interface.

16
Data transport

• The EMU software defines an abstract layer that
  interfaces to the software/hardware used to move
  events.
• This allows the DAQ to be prototyped without
  forcing an early choice of data transport
  technology.
• Initial implementation based on the ET Event
  Transport system.

17
ET, the Event Transport system

• The ET system is a system for moving events
  between processes.
• Data consumers attach to stations that are
  arranged in a ring.
• A master station contains empty buffers.
• A producer fills the empty buffer and puts it
  back.
• Each station has a buffer selection algorithm.
• If a buffer matches it is given to the consumer
  (A).
• Consumers can share a station (B,C,D).
• Consumers can access over the network (E).
• After the last station the buffer is freed.
• Any consumer can free a buffer.

18
EB
ER
18
19
cMsg, CODA Messaging system

• CODA needs the ability to manage messages.
• Messages between programs.
• Messages from programs to operators.
• Messages to and from programs outside CODA.
• There are many message protocols.
• Legacy CODA version 2 protocol.
• Channel Access protocol used by slow control.
• Experiment control and monitoring protocol.
• cMsg is a software layer that sits on top of
  other protocols.
• Uniform interface for all of CODA.
• Easy to add new protocols.
• Publish and subscribe model.
• Messages can cross domain boundaries.

20
AFECS, Experiment Control

• DAQ systems based on CODA version 3 are more
  complex than CODA 2 systems
• More individual components.
• More links between components.
• Individual components have more complicated
  behaviors.
• Solution is AFECS, a multi-Agent Framework for
  Experiment Control Systems.
• Rather than a large monolithic program that does
  everything AFECS is based on the concept of small
  programs called agents that specialize in solving
  a small part of the problem.
• Agents collaborate to control the experiment.
• The system was developed by JLab but is based on
  industry standards
• AFECS is configured by a language called COOL
  that allows the user to describe their
  experiment.
• AFECS can communicate with other systems such as
  EPICS.

21
Visual editor for AFECS

• User doesnt need to write in COOL configuration
  language.

22
Status
23
Summary

• CODA v3 is a natural evolution of CODA v2.
• Optimized to be modular and scalable to meet
  GlueX requirements.
• Parts of CODA v3 are already ready and tested
  with CODA v2 in the 6 GeV program.
• To do
• Convert the toolkit into a GlueX DAQ prototype.
• Interface with the trigger.
• Interface with the front-end hardware.
• Make a parallel event builder using EMUs.
• Convert the prototype into the GlueX DAQ.
• As hardware becomes available gain experience
  with detector test DAQ systems.
• Make critical decisions and purchases for final
  design.
• Experience with hall-B shows that final
  integration takes about two years and the final
  DAQ evolves from the detector test systems.
• To a certain point time is our friend as advances
  in technology make the task easier.