CBM DAQ BNet - PowerPoint PPT Presentation

1 / 28
About This Presentation
Title:

CBM DAQ BNet

Description:

Transmission and reception within and across subsystem boundaries without regard ... 2189, Elsevier Science North-Holland, 2001 (contact Frans.Meijers_at_cern.ch) ... – PowerPoint PPT presentation

Number of Views:49
Avg rating:3.0/5.0
Slides: 29
Provided by: hgessel4
Category:
Tags: cbm | daq | bnet | meijers

less

Transcript and Presenter's Notes

Title: CBM DAQ BNet


1
DAQ Controls CMS XDAQ BTeV RTES, GME,
ARMORs EPICS
H.G.Essel, J.Adamczewski, B.Kolb, M.Stockmeier
2
Example CMS event building and filtering
40 MHz Collision rate
Detectors
1 Terabit/sec50000 channels
100 kHz output from trigger
DAQ Cluster
500 Readout Units Buffer incoming data and Serve
it upon request
500500 ports switching fabric
800 Gb/sec full connectivity
500 Builder Units combine the data from the RUs
100 Mbytes/sec output toPetabyte archive and
output to world-wide data grid
5-10 TeraOpsprocessing sub-cluster
3
CMS blueprint for clustered DAQ
4
CMS DAQ requirements
  • Communication and Interoperability
  • Transmission and reception within and across
    subsystem boundaries without regard of the used
    protocols
  • Addition of protocols without a need for
    modifications in the applications
  • Device Access
  • Access to custom devices for configuration and
    readout
  • Access to local and remote devices (bus adapters)
    without the need for modifications in
    applications
  • Configuration, control and monitoring
  • Make parameters of built-in or user defined types
    visible and allow their modification
  • Allow the coordination of application components
    (define their states and modes)
  • Allow the inspection of states and modes
  • Provide services for recording structured
    information
  • Logging, error reporting
  • Interface to persistent data stores (preferrably
    without the need to adapt the applications)
  • Publish all information to interested subscribers
  • Device allocation, sharing and concurrent access
    support
  • Maintainability and Portability
  • Allow portability across operating system and
    hardware platforms
  • Support access for data across multiple bus
    systems
  • Allow addition of new electronics without changes
    in user software
  • Provide memory management functionality to
  • Improve robustness
  • Give room for efficiency improvements
  • Application code shall be invariant with respect
    to the physical location and the network
  • Possibility to factorise out re-usable building
    blocks
  • Scalability
  • Overhead introduced by the software environment
    must be constant for each transmission operation
    and small with respect to the underlying
    communication hardware in order not to introduce
    unpredictable behaviour
  • Allow applications to take advantage of
    additional resource availability
  • Flexibility
  • Allow the applications to use multiple
    communication channels concurrently
  • Addition of components must not decrease the
    systems capacity

5
CMS XDAQ
  • XDAQ is a framework targeted at data processing
    clusters
  • Can be used for general purpose applications
  • Has its origins in the I2O (Intelligent IO)
    specification
  • The programming environment is designed as an
    executive
  • A program that runs on every host
  • User applications are C programmed plug-ins
  • Plug-ins are dynamically downloaded into the
    executives
  • The executive provides functionality for
  • Memory management
  • Systems programming
  • queues, tasks, semaphores, timers
  • Communication
  • Asynchronous peer-to-peer communication model
  • Incoming events (data, signals, ) are
    demultiplexed to callback functions of
    application components
  • Services for configuration, control and
    monitoring
  • Direct hardware access and manipulation services
  • Persistency services

6
XDAQ Availability
http//cern.ch/xdaq
Current version 1.1 Next releases V 1.2 in
October 2002 (Daqlets) V 1.3 in February 2003
(HAL inspector) Change control Via sourceforge
http//sourceforge.net/projects/xdaq Version
control CVS at CERN License BSD style
7
XDAQ References
J. Gutleber, L. Orsini,  Software Architecture
for Processing Clusters based on I2O , Cluster
Computing, the Journal of Networks, Software and
Applications, Baltzer Science Publishers,
5(1)55-64, 2002(goto http//cern.ch/gutleber
for a draft version or contact me) The CMS
collaboration, CMS, The Trigger/DAQ Project,
Chapter 9 - Online software infrastructure, CMS
TDR-6.2, in print (contact me for a draft), also
available at http//cmsdoc.cern.ch/cms/TDR/DAQ/ G.
Antchev et al., The CMS Event Builder
Demonstrator and Results with Myrinet, Computer
Physics Communications 2189, Elsevier Science
North-Holland, 2001 (contact Frans.Meijers_at_cern.ch
) E. Barsotti, A. Booch, M. Bowden, Effects of
various event building techniques on data
acquisition architectures, Fermilab note,
FERMILAB-CONF-90/61, USA, 1990.
8
XDAQ event driven communication
  • Dynamically loaded application modules (from URL,
    from file)
  • Inbound/Outbound queue (pass frame pointers,
    zero-copy)
  • Homogeneous frame format

Readout component Generates a DMA completion event
Computer
Executive framework Demultiplexes incoming events
to listener application component
foo( )
Application component Implements callback function
Peer transport Receives messages from network
9
XDAQ I2O peer operation for clusters
  • Application component device
  • Processing node IOP
  • Controller node host
  • Homogeneous communication
  • frameSend for local, remote, host
  • single addressing scheme (Tid)
  • Application framework

Messaging Layer
Messaging Layer



Peer Transport Agent
Peer Transport Agent

I2O Message Frames

Executive
Executive



Application

Application
Peer Transport
10
XDAQ available peer transports
11
XDAQWin client
Configuration tree XML based configuration of a
XDAQ cluster
Daqlet window Daqlets are Java applets that can
be used to customize the configuration, control
and monitoring of all components in the
configuration tree
12
XDAQ component commands
Configuration tree Predefined commands can
be performed on selected XDAQ components
(multiple selection allowed) Commands can be
issued to groups of components
executives,applications, peer-transports
thisenables true cluster operation.
13
XDAQ component properties
Component Properties Allows the inspection
andmodification of componentsexported
parameters.
14
XDAQ interface export
Interface Query Returns all exported
commands.Transition to WSDL planned
Cluster commands Results from multiple
componentscan be displayed.
User Commands Allows the execution of
anyexported command, including thecommand to
reflect the interface
15
BTeV a 20 THz real-time system
  • Input 800 GB/s (2.5 MHz)
  • Level 1
  • Lvl1 processing 190?s
  • rate of 396 ns
  • 528 8 GHz G5 CPUs
  • (factor of 50 event reduction)
  • high performance interconnects
  • Level 2/3
  • Lvl 2 processing 5 ms
  • (factor of 10 event reduction)
  • Lvl 3 processing 135 ms
  • (factor of 2 event reduction)
  • 1536 12 GHz CPUs commodity networking
  • Output 200 MB/s (4 kHz) 1-2 Petabytes/year

16
BTeV The problem
  • Monitoring, Fault Tolerance and Fault Mitigation
    are crucial
  • In a cluster of this size, processes and daemons
    are constantly hanging/failing without warning or
    notice
  • Software reliability depends on
  • Physics detector-machine performance
  • Program testing procedures, implementation, and
    design quality
  • Behavior of the electronics (front-end and within
    the trigger)
  • Hardware failures will occur!
  • one to a few per week
  • Given the very complex nature of this system
    where thousands of events are simultaneously and
    asynchronously cooking, issues of data integrity,
    robustness, and monitoring are critically
    important and have the capacity to cripple a
    design if not dealt with at the outset BTeV
    needs to supply the necessary level of
    self-awareness in the trigger system.
  • Real Time Embedded System

17
BTeV RTES goals
  • High availability
  • Fault handling infrastructure capable of
  • Accurately identifying problems (where, what, and
    why)
  • Compensating for problems (shift the load,
    changing thresholds)
  • Automated recovery procedures (restart /
    reconfiguration)
  • Accurate accounting
  • Extensibility (capturing new detection/recovery
    procedures)
  • Policy driven monitoring and control
  • Dynamic reconfiguration
  • adjust to potentially changing resources
  • Faults must be detected/corrected ASAP
  • semi-autonomously
  • with as little human intervention as possible
  • distributed hierarchical monitoring and control
  • Life-cycle maintainability and evolvability
  • to deal with new algorithms, new hardware and new
    versions of the OS

18
RTES deliverables
  • A hierarchical fault management system and
    toolkit
  • Model Integrated Computing
  • GME (Generic Modeling Environment) system
    modeling tools
  • and application specific graphic languages for
    modeling system configuration, messaging, fault
    behaviors, user interface, etc.
  • ARMORs (Adaptive, Reconfigurable, and Mobile
    Objects for Reliability)
  • Robust framework for detection and reaction to
    faults in processes
  • VLAs (Very Lightweight Agents for limited
    resource environments)
  • To monitor/mitigate at every level
  • DSP, Supervisory nodes, Linux farm, etc.

19
RTES Development
  • The Real Time Embedded System Group
  • A collaboration of five institutions,
  • University of Illinois
  • University of Pittsburgh
  • University of Syracuse
  • Vanderbilt University (PI)
  • Fermilab
  • NSF ITR grant ACI-0121658
  • Physicists and Computer Scientists/Electrical
    Engineers at BTeV institutions

20
RTES structure
Modeling
Analysis
Resource
Reconfigure
Performance Diagnosability Reliability
Synthesis
Design and Analysis
Fault Behavior
Feedback
Algorithms
Synthesis
Runtime
Region Operations Mgr
ExperimentControl Interface
L2,3/CISC/RISC
L1/DSP
Soft Real Time
Hard
21
GME simple meta model example
22
GME simple model made from this meta model
23
GME data type modeling
  • Modeling of Data Types and Structures
  • Configure marshalling-demarshalling interfaces
    for communication

24
RTES GME modeling environment
  • Fault handling
  • Process dataflow
  • Hardware Configuration

25
RTES system architecture
  • RunControl Manager
  • Router Information
  • How many regions ?
  • How many worker nodes inside the region?
  • Node Identification information

26
RTES GME fault mitigation modeling language (1)
C
  • Configuration of ARMOR Infrastructure (A)
  • Modeling of Fault Mitigation Strategies (B)
  • Specification of Communication Flow (C)

B
A
27
RTES GME fault mitigation modeling language (2)
  • Model translator generates fault-tolerant
    strategies and communication flow strategy from
    FMML models
  • Strategies are plugged into ARMOR infrastructure
    as ARMOR elements
  • ARMOR infrastructure uses these custom elements
    to provide customized fault-tolerant protection
    to the application

28
RTES user interface modeling language
  • Enables reconfiguration of user interfaces
  • Structural and data flow codes generated from
    models
  • User Interface produced by running the generated
    code

29
RTES user interface generation
30
ARMOR
  • Adaptive Reconfigurable Mobile Objects of
    Reliability
  • Multithreaded processes composed of replaceable
    building blocks
  • Provide error detection and recovery services to
    user applications
  • Hierarchy of ARMOR processes form runtime
    environment
  • System management, error detection, and error
    recovery services distributed across ARMOR
    processes.
  • ARMOR Runtime environment is itself self
    checking.
  • 3-tiered ARMOR support of user application
  • Completely transparent and external support
  • Enhancement of standard libraries
  • Instrumentation with ARMOR API
  • ARMOR processes designed to be reconfigurable
  • Internal architecture structured around
    event-driven modules called elements.
  • Elements provide functionality of the runtime
    environment, error-detection capabilities, and
    recovery policies.
  • Deployed ARMOR processes contain only elements
    necessary for required error detection and
    recovery services.
  • ARMOR processes resilient to errors by leveraging
    multiple detection and recovery mechanisms
  • Internal self-checking mechanisms to prevent
    failures from occurring and to limit error
    propagation.
  • State protected through checkpointing.
  • Detection and recovery of errors.
  • ARMOR runtime environment fault-tolerant and
    scalable
  • 1-node, 2-node, and N-node configurations.

31
ARMOR system basic configuration
32
EPICS overview
  • EPICS is a set of software components and tools
    to develop control systems.
  • The basic components are
  • OPI (clients)
  • Operator Interface. This is a UNIX or Windows
    based workstation which can run various EPICS
    tools (MEDM, ALH, OracleArchiver).
  • IOC (server)
  • Input Output Controller. This can be VME/VXI
    based chassis containing a Motorola 68xxx
    processor, various I/O modules, and VME modules
    that provide access to other I/O buses such as
    GPIB, CANbus.
  • LAN (communication)
  • Local area network. This is the communication
    network which allows the IOCs and OPIs to
    communicate. EPICS provides a software component,
    Channel Access, which provides network
    transparent communication between a Channel
    Access client and an arbitrary number of Channel
    Access servers.

33
Basic attributes of EPICS
  • Tool Based
  • EPICS provides a number of tools for creating a
    control system. This minimizes the need for
    custom coding and helps ensure uniform operator
    interfaces.
  • Distributed
  • An arbitrary number of IOCs and OPIs can be
    supported. As long as the network is not
    saturated, no single bottle neck is present. A
    distributed system scales nicely. If a single IOC
    becomes saturated, it's functions can be spread
    over several IOCs. Rather than running all
    applications on a single host, the applications
    can be spread over many OPIs.
  • Event Driven
  • The EPICS software components are all designed to
    be event driven to the maximum extent possible.
    For example rather than having to poll IOCs for
    changes, a channel access client can request that
    it be notified only when changes occur. This
    design leads to efficient use of resources as
    well as to quick response times.
  • High Performance
  • A SPARC based workstation can handle several
    thousand screen updates a second with each update
    resulting from a channel access event. A 68040
    IOC can process more than 6,000 records per
    second including generation of any channel access
    events.

34
IOC software components
  • DATABASE
  • The memory resident database plus associated data
    structures.
  • Database Access
  • Database access routines. With the exception of
    record and device support, all access to the
    database is via the database access routines.
  • Scanners
  • The mechanism for deciding when records should be
    processed.
  • Record Support
  • Each record type has an associated set of record
    support routines.
  • Device Support
  • Each record type has one or more sets of device
    support routines.
  • Device Drivers
  • Device drivers access external devices. A driver
    may have an associated driver interrupt routine.
  • Channel Access
  • The interface between the external world and the
    IOC. It provides a network independent interface
    to database access.
  • Monitors
  • Database monitors are invoked when database field
    values change.
  • Sequencer
  • A finite state machine.

35
Hierarchy in a flat system
tasks
IOC
  • IOCs
  • One IOC per standard CPU (Linux, Lynx, VxWorks)
  • clients
  • on Linux, (Windows)
  • Agents
  • Segment IOCs beeing also clients
  • Name space architecture!

IOC
tasks
IOC
Client
tasks
IOC
IOC
tasks
IOC
36
Local communication (node)
Node
commands
intertask
Task
statussegment
IOC
memory
Task
  • Commands handled by threads
  • Execution maybe in working thread
  • Message thread maybe not needed

messages
37
MBS node and monitor IOC
commands (text)
Task
IOC
asynchronous
External control
statussegment
Dispatcher
on request
Statusserver
Task
Messageserver
asynchronous
messages (text)
38
Screen shot FOPI
39
Kind of conclusion
  • RTES Very big and powerful. Not simply
    available!
  • Big collaboration
  • Fully modelled and simulated using GME
  • ARMORs for maximum fault tolerance and control
  • XDAQ Much smaller. Installed at GSI.
  • Dynamic configurations (XML)
  • Fault tolerance?
  • EPICS From accelerator controls community.
    Installed at GSI
  • Maybe best known
  • No fault tolerance
  • Not very dynamic
Write a Comment
User Comments (0)
About PowerShow.com