EVB Simulation Status

1
EVB Simulation Status

• EVBSIM code changes
• FNAL test stand comparison
• LHC case studies
• Ptolemy progress
• Plans

2
XDAQ simulation
3
EVBSIM code changes

• LHC studies use barrel shifting
• RU for event traffic ? BU
• EVM for control traffic ? BU
• BU for control traffic ? EVM, RU
• Not really needed for 64x64
• Scales nicely at least up to 256x256 (see plot)
• FNAL test stand studies do NOT have barrel
  shifting
• EVBSIM memory consumption
• Some leaks were found with Insure
• It also reported problems erroneously
• 512x512 memory usage still 2 GB
• Always crashes on cmsun1

4
FNAL NxM setup (1/2)

• XDAQ test stand and EVBSIM measure
• Event data bandwidth per BU, in MB/s
• All BUs are checked to give the same value
• Tables give EVBSIM/XDAQ bandwidth ratios
• For each setup vary 2 parameters
• N number of event IDs per control packet
• N 1, 4, 16, 64
• R number of active event ID request packets per
  BU
• R 1, 4, 16, 64
• N x R number of active event IDs per BU
• N x R ? 256 number of resources per BU

5
FNAL NxM setup (2/2)

• Event fragment 2 kB per RU
• Control packets
• 64 bytes/eventID (too large?) 224 bytes I2O
  header
• Data packets
• 2 kB event data 224 bytes I2O header
• L2 needs all RUs
• Each event rejected after 0 ms
• PCI bandwidth 132 MB/s
• Myrinet bandwidth 160 MB/s

6
FNAL 1xM comparisons (1/2)

• 1x1
• R \ N 1 4
  16 64
• -----------------------------------------------
  ----------------------------
• 1 8 / 10.7 0.7 23 / 27.2 0.8 43 /
  43.8 1.0 56 / 52.0 1.1
• 4 23 / 23.8 1.0 48 / 49.3 1.0 62 /
  68.0 0.9 66 / 68.1 1.0
• 16 25 / 25.2 1.0 48 / 49.3 1.0 61 /
  64.8 0.9
• 64 25 / 25.4 1.0
• 1x2
• R \ N 1 4
  16 64
• -----------------------------------------------
  ----------------------------
• 1 8 / 10.6 0.8 22 / 26.4 0.8 35 /
  33.7 1.0 35 / 35.0 1.0
• 4 22 / 22.3 1.0 32 / 33.9 0.9 35 /
  35.2 1.0 35 / 34.8 1.0
• 16 25 / 24.9 1.0 32 / 34.2 0.9 35 /
  35.2 1.0
• 64 25 / 25.1 1.0
• 1x4
• R \ N 1 4
  16 64
• -----------------------------------------------
  ----------------------------

7
FNAL 1xM comparisons (2/2)
8
FNAL 2xM comparisons (1/2)

• 2x1
• R \ N 1 4
  16 64
• -----------------------------------------------
  ----------------------------
• 1 14 / 18.8 0.7 34 / 41.5 0.8 54 /
  60.0 0.9 63 / 67.1 0.9
• 4 34 / 32.6 1.0 54 / 56.5 1.0 65 /
  71.6 0.9 68 / 78.0 0.9
• 16 34 / 33.4 1.0 55 / 56.5 1.0 65 /
  71.9 0.9
• 64 34 / 33.6 1.0
• 2x2
• R \ N 1 4
  16 64
• -----------------------------------------------
  ----------------------------
• 1 14 / 18.7 0.7 34 / 40.5 0.8 54 /
  53.3 1.0 63 / 55.0 1.1
• 4 34 / 31.9 1.1 54 / 55.6 1.0 65 /
  66.0 1.0 68 / 57.5 1.2
• 16 34 / 33.0 1.0 55 / 55.7 1.0 65 /
  66.3 1.0
• 64 34 / 33.1 1.0
• 2x4
• R \ N 1 4
  16 64
• -----------------------------------------------
  ----------------------------

9
FNAL 2xM comparisons (2/2)
10
FNAL 4xM comparisons (1/2)

• 4x1
• R \ N 1 4
  16 64
• -----------------------------------------------
  ----------------------------
• 1 24 / 28.9 0.8 47 / 53.1 0.9 62 /
  67.6 0.9 68 / 71.9 0.9
• 4 40 / 39.7 1.0 59 / 61.3 1.0 67 /
  73.2 0.9 70 / 78.4 0.9
• 16 40 / 40.0 1.0 59 / 61.3 1.0 67 /
  73.1 0.9
• 64 40 / 40.0 1.0
• 4x2
• R \ N 1 4
  16 64
• -----------------------------------------------
  ----------------------------
• 1 23 / 28.7 0.8 47 / 45.1 1.0 62 /
  57.2 1.1 68 / 49.9 1.4
• 4 40 / 39.2 1.0 59 / 52.1 1.1 67 /
  56.0 1.2 69 / 69.7 1.0
• 16 40 / 39.3 1.0 59 / 52.1 1.1 67 /
  50.2 1.3
• 64 40 / 39.6 1.0
• 4x4
• R \ N 1 4
  16 64
• -----------------------------------------------
  ----------------------------

11
FNAL 4xM comparisons (2/2)
12
FNAL NxM results

• Latency overheads tuned only for 1x1,
  R64, N1
• Send DMA latency 11 bytes/132 us
• Receive DMA latency 11 bytes/132 us
• Myrinet transfer latency 1 bytes/160 us
• Overall EVBSIM/XDAQ agreement not bad
• Some XDAQ values are strange
• They do not agree with the trends
• Various values have been shown to be reproducible
  only within 10 sometimes
• Slopes for R1 not understood

13
NxN LHC tests

• Each BU emulates M Filter Nodes (see plots for
  values)
• Each Node has 1 CPU for HLT calculations
• Events arrive in zero time from the BU
• A second CPU is assumed to do that in the
  background
• Event fragment 2 kB per RU
• Control packets N32, R8
• 64 bytes/eventID (too large) 224 bytes I2O
  header
• Data packets
• 2 kB event data 224 bytes I2O header
• L2 needs all RUs
• Each event rejected after X ms (see plots for
  values)
• DMA send latency 1 bytes/528 us
• Myrinet transfer latency 0 bytes/250 us
• DMA receive latency 1 bytes/528 us

14
64x64 LHC test (1/4)
15
64x64 LHC test (2/4)
16
64x64 LHC test (3/4)
17
64x64 LHC test (4/4)
18
256x256 LHC test
19
Ptolemy status at FNAL

• Started from code by Samim, Alexander and Frans
• Implemented Builder Manager
• Extra XDAQ messages
• BMControl (Allocate, Clear) and Confirm
• BM replaces the last RU
• Participates nicely in RU barrel shifting
• Implemented PCI barrel shifting in BU
• Implemented Readout Manager with rudimentary GTP
  RCN

20
Ptolemy work at FNAL

• EVM/RU
• Do we need detailed GTP/RCN handling?
• RU
• Do we need detailed FED/DDU handling?
• BU must copy events to Filter Nodes
• This will double the PCI bus load
• Check the effect
• Add more ntuples
• Study plots
• What else?

21
Plans

• Keep CNCL code mostly frozen
• We may want to run it occasionally
• http//home.fnal.gov/litmaath/evbsimrs.ps
• How does EVBSIM work?
• What are its results?
• Update CVS repositories at CERNFNAL
• Include example configurations, scripts and
  resulting numbers and plots
• Replicate full XDAQ protocol in Ptolemy
• Compare with CERNFNAL test stands
• Repeat scaling studies