PCI Express DMA Engine f

1
PCI Express DMA Engine für Active Buffer Projekt
im CBM Experiment

• Wenxue Gao, Andreas Kugel, Reinhard Männer,
  Holger Singpiel, Andreas Wurz
• Uni. Mannheim
• DPG Tagung, Gießen
• 14 März 2007

2
Inhalt

• Einleitung
• Blockdiagramm
• Realisierung
• Leistung

2 von 15
3
Einleitung CBM Experiment
CBM TSR, Jan. 2006
4
Einleitung PCI Express

• 2,5 Gbps pro Link
• Point-to-Point
• TLP (Transaction Layer Packet)
• Post MWr (Memory Write Request),
• Non-post MRd (Memory Read Request),
• Completion CplD, Cpl,
• Message Msg

4 von 15
5
PCI Express Post TLP (MWr, )
Trn.
Host
End-Point
Rx
Tx
6
PCI Express Post TLP (MWr, )
Trn.
Host
End-Point
Rx
Tx
MWr1
7
PCI Express Post TLP (MWr, )
Trn.
Host
End-Point
Rx
Tx
MWr1
8
PCI Express Post TLP (MWr, )
Trn.
Host
End-Point
Rx
MWr1
Tx
9
PCI Express Post TLP (MWr, )
Trn.
Host
End-Point
Rx
MWr1
Tx
MWr2
10
PCI Express Post TLP (MWr, )
Trn.
Host
End-Point
Rx
MWr1
Tx
MWr2
11
PCI Express Post TLP (MWr, )
Trn.
Host
End-Point
Rx
MWr1
MWr2
Tx
12
PCI Express Post TLP (MWr, )
Trn.
Host
End-Point
Rx
MWr1
MWr2
Tx
MWr3
13
PCI Express Post TLP (MWr, )
Trn.
Host
End-Point
Rx
MWr1
MWr2
Tx
MWr3
14
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
Tx
15
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
Tx
MRd1
16
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
Tx
MRd1
17
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
MRd1
Tx
18
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
MRd1
Tx
MRd2
19
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
MRd1
Tx
MRd2
20
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
MRd1
Tx
MRd2
21
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
CplD1
MRd1
Tx
MRd2
22
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
CplD2
CplD1
MRd1
Tx
MRd2
23
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
CplD2
CplD1
MRd1
Tx
MRd2
24
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
CplD2
CplD1
MRd1
Tx
MRd2
25
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
CplD2
CplD1
MRd1
Tx
MRd2
26
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
CplD1
CplD2
MRd1
Tx
MRd2
27
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
CplD1
CplD2
MRd1
Tx
MRd2
28
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
CplD1
CplD2
MRd1
MRd1
Tx
MRd2
MRd2
29
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
CplD1
CplD2
Tag70
MRd1
MRd1
Tx
MRd2
MRd2
30
PCI Express Non-post TLP (MRd, )
Trn.
End-Point
Host
Rx
CplD1
CplD2
Tag70
MRd1
MRd1
Tx
MRd2
MRd2
31
Einleitung SG DMA

• SG(Scatter/Gather)
• Multiple-descriptor chain
• Voll-Duplex
• Downstream Host ? Endpoint
• Upstream Endpoint ? Host
• Done Zustand
• Status Register
• Interrupt

32
Blockdiagramm
33
Channel Buffer

• TLP Channel FIFO
• Breite 128
• Tiefe 15
• TLP ohne Payload
• Alles im Word
• TLP mit Payload
• Lokale Adresse
• Zusätzliche Informationen

9 von 15
34
Realisierung DMA teilen

• 4 KB Grenze verboten
• Address/Length Combination

35
Realisierung Done bestätigen

• Wann ist DMA beendet?
• Done Zustand nötig
• CplDs für unterschiedliche MRds kommen nicht
  folgend
• Mögliche Lösungen
• Tag RAM lesen
• CplD zählen
• Channel Buffer leer
• Letzten Tag triggern (x)
• Bitmap füllen
• 128-bit Register für 7-bit Tags

11 von 15
36
Leistungsparameter

• Zielbaustein
• Virtex4 XC4VFX60-11ff672
• FFs
• 9 834 out of 50 560 ( 19 )
• LUT4s
• 11 464 out of 50 560 ( 22 )
• RAMb16
• 58 out of 232 ( 25 )
• Slices
• 9 426 out of 25 280 ( 37 )
• Frequenz ( trn_clk )
• 250 MHz
• Verzögerung (Transaction layer)
• PIO 52 ns (MRd ? CplD )
• DMA 80 ns (DMA Start ? Tx TLP)
• Theoretische Bandbreite
• 2Gbps x4 8Gbps, bi-directional

12 von 15
37
4-Lane Tests
38
Offene Fragen

• Kleinerer Channel Buffer
• Meistens reichen 64-bit, statt 128-bit
• Bessere Behandlung von Fehlern
• Teilweise unvollständig
• Überschreiben von CplD zu vermeiden
• Time-out
• tag Recycling
• Höhere Bandbreite für downstream DMA

39
Zusammenfassung

• PCI Express Vorteile
• Parallelität
• Skalierbarkeit
• Virtual channels
• 2 DMA Channels
• 1 PIO Channel
• Xilinx Lösung
• 62,5 MHz für x1
• 250 MHz für x4

15 von 15
40
(No Transcript)
41
x4-ABB

• Design Summary
• --------------
• Logic Utilization
• Number of Slice Flip Flops 9,834 out of
  50,560 19
• Number of 4 input LUTs 11,464 out of
  50,560 22
• Logic Distribution
• Number of occupied Slices 9,426 out of
  25,280 37
• Total Number 4 input LUTs 12,993 out of
  50,560 25
• Number used as logic 11,464
• Number used as a route-thru 643
• Number used for Dual Port RAMs 202
• Number used as Shift registers 684
• Number of bonded IPADs 18 out of
  62 29
• Number of bonded OPADs 16 out of
  24 66
• Number of bonded IOBs 1 out of
  352 1
• Number of BUFG/BUFGCTRLs 5 out of
  32 15
• Number used as BUFGs 4

42
X4 Test
43
DMA Prozess

• Buffer-descriptor
• SA (Source Address)
• DA (Destination Address)
• NXA (Next Descriptor Address)
• Length (Length in bytes)
• Control (Control register)
• Start/Stop Befehl
• Upstream MWr MRd (dex)
• Downstream MRd
• Busy/Done Zustände erkennen
• Status Register
• Interrupt (Msg)

44
Blockdiagramm
45
Verifizieren

• PIO DMA (random)
• Transaction length
• Address-pair
• Chain length (DMA)
• Descriptor Address (DMA)
• Flow control _rdy_n
• Output checking
• tsof/teof
• Data
• Deskriptor abteilen

46
Memory Space

• BRAM
• 16KB
• FIFO
• 32 x 32
• Loop-back
• Registers
• Write / Read
• Control / Status
• Eventuelle Erweiterung
• DDR (BRAM ähnlich)
• GbE (FIFO ähnlich)