Evolution of S-LINK to PCI interfaces

1
Evolution of S-LINK to PCI interfaces

• W. Iwanski
• (Henryk Niewodniczanski Institute of Nuclear
  Physics)
• M. Joos, R. McLaren, J. Petersen, E. van der Bij
• (CERN)

2
System evolution
4 S-LINKs
4 S-LINKs
S-LINK
S-LINK
S-LINK
S-LINK
SSP
SSP
FILAR
FILAR
S32PCI64
S32PCI64
slow PCI
fast PCI
fast PCI
Eth.
MEM
MEM
MEM
slow PCI
slow PCI
Eth.
Eth.
CPU
CPU
CPU
PC
PC
PC
3
Current use of S-Link to PCI interfaces in
test-beams
TileCal test beam
Muon test beam
VMEbus
VMEbus
VMEbus
VMEbus
VMEbus
SPS
SPS
SPS
SPS
SPS
S-LINK
S-LINK
S-LINK
S-LINK
S-LINK


SSPCI
SSPCI
SSPCI
SSPCI
SSPCI
PC
PC

• Features of SSP / SSPCI
• Simple hardware design (AMCC S5933 PCI
  controller)
• Max. speed 80 (RIO2) / 132 (PC) MB/s
• Event overhead 8 us
• One ROL per PMC / PCI slot

4
S32PCI64

• 32/64 bit PCI
• 33/66 MHz
• 3.3 V PCI slots only
• 3.3 V S-LINK LDC plug-in

5
Overview of S32PCI64


PCI


PCI

BURST
INPUT






32
INPUT
DMA
L



BUFFER




FIFO
I
to
BUFFER
FIFO

I



FIFO

ENGINE

64
FIFO

64






(1024 x 64
-
bit)
128x

1024 x 64-bit
128x 64

64
-
bit

REQUEST

REQUEST
FIFO


FIFO


PCI



64
-
BIT


address, length
64
-
BIT
33/66 MHz
32/64
-

32/64-bit

PCI

PCI

BACKEND
PCI

ACKNOWLEDGE


CONTROL

BACK
END
CORE



FIFO
CORE



FIFO

LOGIC
ctrl words, length


LOGIC

CONTROL,


STATUS
CONTROL,

INTERRUPT

STATUS

REGISTERS

FPGA
Local logic
Commercial IP Core


6
Features of S32PCI64

• 32/64 bit, 33/66 MHz PCI (3.3 V slots only)
• 32-bit S-LINK (3.3 V LDC plug-in only)
• 32-bit PCI-bus addressing
• No Initiator wait states during a burst
• Max. 64 Mbytes long DMA transfer
• Protocol overhead per event 5 PCI single cycles
• Based on a commercial PCI IP core (PLDA)
• Highly autonomous data reception
• Interrupt generation selectable on several
  conditions
• Data packets longer than 1kB are segmented

7
Features and applications of S32PCI64 based
systems

• Less load on the CPU to handle link protocol
• event overhead 2.5 us
• 3.3 V PCI slots required
• For applications requiring full ATLAS input rate
• Applications
• CosmoDAQ
• mainly for prototypes
• Commercially available from NE (Nowoczesna
  Elektronika)

8
S32PCI64 test set-up

• Elonex PC
• SuperMicro 370DLE motherboard
• memory bandwidth gt 528 MB/s
• SLIDAS data generator
• VMEtro PBT-515BX PCI Bus Analyser
• PCI bandwidth exerciser
• (PCI-Blaster)

SLIDAS
SLIDAS
160 MB/s
160 MB/s
S32PCI64
S32PCI64
PCI 66 MHz/ 64 bit
ServerSet III LE
CPU
MEM
PC
9
PCI-Blaster

• PCI bandwidth exerciser
• firmware designed and implemented on S32PCI64
  board to benchmark PCI bridge and memory in PC
• Features
• full speed PCI capability (528 MB/s)
• read and write capability
• read and write modes can be set-up simultaneously
  
• simple programming model
• data transfers in specified number of times or in
  infinite loop
• 32-bit and 64-bit PCI bus (3.3V PCI bus only)
• 33 and 66 MHz MHz PCI clock speed
• 32-bit PCI-bus addressing
• no interrupt capability

10
Overall throughput
11
Other measurements

• Overhead of S32PCI64 hardware in single access
  from host CPU to S32PCI64
• write -gt 30 ns (2 wait states)
• read -gt 45 ns (3 wait states)
• Minimal time interval between 2 consecutive
  single commands seen on PCI bus (limited by
  CPU/PCI bridge)
• write-write -gt 75 ns
• write-read -gt 105 ns
• read-read -gt 330 ns
• read-write -gt 345 ns
• Gap between DMA sub-bursts (packets gt 1kB) -gt
  180 ns

12
FILAR

• Four Integrated Links for ATLAS Readout

13
Main features of FILAR

• Four 2.5 Gbit/s HOLA S-LINK Destinations
  integrated
• Data channels individually enabled/disabled
• 32/64-bit 33/66 MHz PCI bus
• 32-bit S-LINK
• 32-bit PCI-bus addressing
• Protocol overhead per event and channel 2 PCI
  single cycles
• Highly autonomous data reception
• Card temperature readout
• Module scheduled for 1Q/2003

14
Steps towards FILAR

• New library written
• in ROS context
• FILAR emulator simplified specification of FILAR
  implemented in existing hardware (S32PCI64)
• Data from one S-link connector copied to all four
  data channels
• One common flow control signal
• logical OR of all four XOFF signals

15
Features of FILAR software

• Package consists of Linux driver and user library
• Detection of data packets based on interrupts
• Support for
• multiple PCI cards
• multiple channels
• Code optimised for low overhead
• 1.5 us / event
• Current API will also be valid for final FILAR

16
Performance of FILAR emulator
17
Performance (contd)

• Improved performance of FILAR emulator for small
  packets with respect to S32PCI64
• Better performance of FILAR emulator running one
  data channel than that in S32PCI64
• Performance of FILAR emulator running two, three
  or four data channels is compromised by board
  limitation
• flow control signal, working for all channels
  here, stops new data to whole interface whenever
  any of data channels is getting full. It prevents
  other, already empty buffers from being re-filled
  
• gaps up to 4.5 us seen between some DMAs
• If necessary, (still) costly single PCI cycles
  will be replaced by DMA

18
Summary

• Transition from 32-bit/33 MHz to 64-bit/66 MHz
  PCI
• full bandwidth of S-LINK (160 MB/s)
• Stable design evolution from S32PCI64 to FILAR
• great part of software re-used
• Use of FPGA and IP core in designs
• easy debugging, testing and upgrading
• re-usability of existing hardware
• PCI-Blaster
• FILAR emulator
• Powerful and cost optimised FILAR design
• integration of 4 links in one card reduces costs
  and overcomes a limit of fast 66 MHz PCI slots in
  PC
• pluggable optical transceivers further reduce
  costs when less than 4 data channels are needed