EVM Project Status

1
EVM Project Status

• S. Aziz, M. Litmaath, C. Moore, V. ODell, I.
  Suzuki
• FNAL
• S. Pavlon, K. Sumorok
• MIT
• Topics
• RCN
• EVM Test Bench
• EVBSIM results
• XDAQ Run Control
• Future work

2
Readout Control Network
Implemented on Linux with 1) Ethernet UDP/IP
2) IEEE1394 asynchronous
broadcast transmission normally point to point
with confirmation 3) IEEE1394 isochronous
broadcast being worked on IEEE1394 limited to
63 nodes with current technology Bridges should
be available in the Future Data transmitted in
a 125 us cycle gt 8K blocks/s Currently 400Mb/s
data rate Broadcast mode delivery not guaranteed
3
Reliable Broadcast Protocol
ID information from EVM to RU broadcast in
Packets

• NACK based
• Receiver side packet loss detection
• PacketX XOR of original packets
• ACKs from few clients as a pacemaker

4
Reliable Broadcast Library

• Implemented in C
• Modular design
• Capable of Using both
• Ethernet and IEEE1394
• Preliminary Benchmark
• Tests done
• Protocol Specification
• Document revised

http/home.fnal.gov/ichiro/document/evm-rcn/main.
ps
5
Benchmark Test of Library

• Test Conditions
• Two quadlets of EVT-ID data
• per Trigger
• UDP/ID on FastEthernet or
• FCP(async) on IEEE1394
• No Forward Error Correction
• No Transmission error
• Maximum Rate
• UDP 1300kHz (10.5 MB/s)
• 1394 600 kHz (5.0 MB/s)

6
Things to be done
RCN integration into XDAQ

• Add peer-transport for the RCNP
• Add Callback methods in the RUI,RM
• Modify EVT-ID Management in RUI-RUO, RM-BM

7
EVM Test Bench
EVM, RU and BU Driven by XDAQ
Myrinet used for both BCN and BDN in tests
8
Modules Brief Description (1/3)

• L1
• Sends triggers to EVM and DDU (RU)
• Receives throttle commands from EVM
• EVM
• Receives
• Triggers from L1
• Event requests and releases from BU via BCN
• Sends
• Throttle commands to L1
• Triggers to RU via RCN
• Event IDs to BU via BCN

9
Modules Brief Description (2/3)

• RU (emulates 1 DDU)
• Sends data fragments to BU via BDN
• Receives
• Triggers from L1 (DDU)
• Triggers from EVM via RCN
• Data fragment requests from BU via BCN
• BU (emulates N Filter Nodes each with M CPUs)
• Sends
• Event requests and releases to EVM via BCN
• Data fragment requests to RU via BCN
• Receives
• Event IDs from EVM via BCN
• Data fragments from RU via BDN

10
Modules Brief Description (3/3)

• BCN (no internal structure NICs do the work)
• Sends control messages to EVM, BU, RU
• Receives control messages from BU, EVM
• BDN (no internal structure NICs do the work)
• Sends data fragments to BU
• Receives data fragments from RU
• XDAQ does not distinguish BCN and BDN
• EVBSIM uses BCN also for data fragments

11
EVM Simulation

• EVBSIM new components
• BCN and BDN have no internal structure
• BCN and BDN use NIC objects for traffic
  sequencing
• NICs ensure each endpoint does only one
• send/receive/DMA at a time
• NIC can send and receive at the same time
• (easy to change if ever necessary)
• Problems
• 500x500 cannot keep up with trigger rate
• RU does not implement barrel shifter yet (not
  trivial)
• BU traffic shaping helps, but seems insufficient
• 500x500 memory usage gt 2 GB !
• There might still be a leak somewhere

12
EVBSIM Results (1)

• Comparison of 1x1 test bench varying
• Number of event requests per packet (64 bytes
  per event)
• Number of concurrent event request packets in BU
• Results tuned by varying
• DMA from host memory to NIC 10 us bytes/132 us
• Myrinet transfer 5 us bytes/160 us
• DMA from NIC to host memory 15 us bytes/132 us
• 224 byte I2O header
• Smallest message 224 64 288 bytes

13
EVBSIM Results (2)
Bandwidth EVMSIM/XDAQ in MB/s Events
per Packet 1 4 16 64 10/10 1.0
25/26 1.0 43/44 1.0 52/53 1.0 23/23
1.0 43/49 0.9 55/58 0.9 59/61
1.0 24/25 1.0 43/49 0.9 55/58 0.9
24/25 1.0
Concurrent packets
1 4 16 64
14
EVBSIM Plans

• Simple model OK for 1x1
• Lets see how it does for 2x2, 1x3, 3x1, etc.
• March, April
• Extract trends, fudge factors, if any
• Implement RU barrel shifter for LHC case
• See if it allows the DAQ to keep up with L1
• Use simple BCN for now

15
XDAQ Run Control Status

• Learning Curve for jxdaq java code
• Building code
• Booting components
• Shutdown of xdaq when using ssh instead of rsh
• Ability to Automate xdaq and obtain results
• Currently using a perl script with xdaq Tcl
  interface
• Desire same functionality but via xml/xdaq
  interface.
• First pass at automating xdaq
• Modified jxMain such that GUI appears but no
  mouse clicks required for actions(e.g. Open, boot
  and configuring IOPs). Results still displayed
  on GUI. Works but is overkill for an automation.

16
Future work

• IEEE1394 bandwidth
• Isochronous transmission
• Modification of the Linux driver
• Test Stand
• Benchmark test of 2x2 and EVBSIM comparison
• Integration of RCN into XDAQ
• Implementation of XDAQ Run Control GUI.
• Automated scripts for XDAQ with Command line
  Interface
• EVBSIM
• Implement RCN XDAQ modifications
• Extend EVBSIM to NxN
• Technical Design Report (TDR)