GGF4 Toronto Feb 2002

1
Initial Performance MeasurementsGigabit Ethernet
NICs64 bit PCI Motherboards(Work in progress)
CollaborationBoston Ltd. (Watford)
SuperMicro Motherboards, CPUs, Intel GE
NICsBrunel University Peter Van
SantenUniversity of Manchester Richard
Hughes-Jones
2
The Measurements (1)

• Latency
• Round trip times measured using Request-Response
  UDP frames
• Latency as a function of frame size
• Slope gives sum of individual data transfer rates
  end-to-end
• Mem copy pci Gig Ethernet pci mem copy
• Histograms of individual measurements

3
The Measurements (2)

• UDP Throughput
• Send a burst of UDP frames spaced at regular
  intervals
• Vary the frame size and the frame transmit
  spacing
• Record
• The time to send and the time to receive the
  frames
• The number received, the number lost, number out
  of order
• The received inter-packet spacing
• CPU load, Number of interrupts

4
The Measurements (3)

• PCI Activity
• Logic Analyzer with
• PCI Probe cards in sending PC
• Gigabit Ethernet Fiber Probe Card
• PCI Probe cards in receiving PC

Gigabit Ethernet Probe
CPU
CPU
NIC
NIC
chipset
chipset
mem
mem
Logic Analyser Display
5
Latency Alteon AceNIC

• Motherboard SuperMicro 370DLE Chipset
  ServerWorks III LE Chipset
• CPU PIII 800 MHz PCI64 bit 66 MHz
• RedHat 7.1 Kernel 2.4.14

6
UDP Throughput Alteon AceNIC

• Motherboard SuperMicro 370DLE Chipset
  ServerWorks III LE Chipset
• CPU PIII 800 MHz PCI64 bit 66 MHz
• RedHat 7.1 Kernel 2.4.14

7
PCI Alteon AceNIC

• Motherboard SuperMicro 370DLE Chipset
  ServerWorks III LE Chipset
• CPU PIII 800 MHz PCI64 bit
• RedHat 7.1 Kernel 2.4.14

• ALT33102
• PCI 33 MHz
• 1400 bytes sent
• Wait 16 us
• ALT66101
• 66 MHz
• 1400 bytes sent
• Wait 16 us
• NIC cannot sustain 66 MHz

Send PCI
Receive PCI
8
Latency SysKonnect

• Motherboard SuperMicro 370DLE Chipset
  ServerWorks III LE Chipset
• CPU PIII 800 MHz PCI64 bit 33 MHz
• RedHat 7.1 Kernel 2.4.14

• Latency low good
• Latency well behaved
• Slope 0.0252 us/byte
• Expect
• PCI 0.00758
• GigE 0.008
• PCI 0.00758
• 0.0236 us/byte

9
UDP Throughput SysKonnect

• Motherboard SuperMicro 370DLE Chipset
  ServerWorks III LE Chipset
• CPU PIII 800 MHz PCI64 bit 33 MHz
• RedHat 7.1 Kernel 2.4.14

• Max throughput 690Mbit/s
• No packet loss
• Packet loss during drop

10
PCI SysKonnect

• Motherboard SuperMicro 370DLE Chipset
  ServerWorks III LE Chipset
• CPU PIII 800 MHz PCI64 bit 66 MHz
• RedHat 7.1 Kernel 2.4.14

• SK300
• 1400 bytes sent
• Wait 100 us
• 8 us for send or receive

Gigabit Ethernet frame
11
PCI SysKonnect

• Motherboard SuperMicro 370DLE Chipset
  ServerWorks III LE Chipset
• CPU PIII 800 MHz PCI64 bit 66 MHz
• RedHat 7.1 Kernel 2.4.14

• SK301
• 1400 bytes sent
• Wait 20 us
• Sk303
• 1400 bytes sent
• Wait 10 us
• Frames are back-to-back
• Cannot go any faster !

Gig Eth frames back to back
12
Latency Intel

• Motherboard SuperMicro 370DLE Chipset
  ServerWorks III LE Chipset
• CPU PIII 800 MHz PCI64 bit 66 MHz
• RedHat 7.1 Kernel 2.4.14

• Latency high
• Latency well behaved
• Slope 0.0187 us/byte
• Expect
• PCI 0.00188
• GigE 0.008
• PCI 0.00188
• 0.0118 us/byte

13
PCI Intel

• Motherboard SuperMicro 370DLE Chipset
  ServerWorks III LE Chipset
• CPU PIII 800 MHz PCI64 bit 66 MHz
• RedHat 7.1 Kernel 2.4.14

• IT66M200
• 64 bytes sent
• CSR time 1.75 us
• Data time 0.25 us
• Interrupt delay70 us
• 1400 response

14
Throughput Intel

• Motherboard SuperMicro 370DLE Chipset
  ServerWorks III LE Chipset
• CPU PIII 800 MHz PCI64 bit 66 MHz
• RedHat 7.1 Kernel 2.4.14

• Max throughput 910Mbit/s
• No packet loss
• Packet loss during drop

15
Throughput Intel

• Motherboard SuperMicro 370DLE Chipset
  ServerWorks III LE Chipset
• CPU PIII 800 MHz PCI64 bit 66 MHz
• RedHat 7.1 Kernel 2.4.14

• losses occur in groups
• 50 pkts every 140

16
PCI Intel

• Motherboard SuperMicro 370DLE Chipset
  ServerWorks III LE Chipset
• CPU PIII 800 MHz PCI64 bit 66 MHz
• RedHat 7.1 Kernel 2.4.14

• IT66M212
• 1400 bytes sent
• Wait 11 us
• 4.7us on send PCI bus
• PCI bus 45 occupancy
• 3.25 us on PCI for data recv
• IT66M212
• 1400 bytes sent
• Wait 11 us
• Packets lost
• Action of pause packet?

17
Latency Intel on P4CD6

• Motherboard SuperMicro P4CD6 Chipset Intel
  i860
• CPU Dual Xeon Prestonia (2cpu/die) 2.2 GHz
  PCI64 bit 66 MHz
• RedHat 7.1 Kernel 2.4.2

• Latency high
• Slope 0.0195 us/byte
• Expect
• PCI 0.00188
• GigE 0.008
• PCI 0.00188
• 0.0118 us/byte

18
Throughput Intel on P4CD6

• Motherboard SuperMicro P4CD6 Chipset Intel
  i860
• CPU Dual Xeon Prestonia (2cpu/die) 2.2 GHz
  PCI64 bit 66 MHz
• RedHat 7.1 Kernel 2.4.2

• Max throughput 950Mbit/s
• No packet loss
• Negligible Packet loss

19
PCI Intel on P4CD6

• Motherboard SuperMicro P4CD6 Chipset Intel
  i860
• CPU Dual Xeon Prestonia (2cpu/die) 2.2 GHz
  PCI64 bit 66 MHz
• RedHat 7.1 Kernel 2.4.2

• IT66M220
• 1400 bytes sent
• Wait 1000 us
• CSR time 12.25 us
• Data time 0.25 us
• Interrupt delay79 us
• IT66M224
• 1400 bytes sent
• Wait 100 us
• Detail
• Chipset limits PCI transfers with STOPs
• Try i870 Chipset

20
PCI Intel on P4CD6

• Motherboard SuperMicro P4CD6 Chipset Intel
  i860
• CPU Dual Xeon Prestonia (2cpu/die) 2.2 GHz
  PCI64 bit 66 MHz
• RedHat 7.1 Kernel 2.4.2

• IT66M221
• 1400 bytes sent
• Wait 11 us

21
Latency Intel on IBM board

• Motherboard IBM das Chipset ServerWorks
  CNB20LE
• CPU Dual PIII 1GHz PCI64 bit 33 MHz
• RedHat 7.1 Kernel 2.4.14

• Latency high
• Latency well behaved
• Slope 0.0206 us/byte
• Expect
• PCI 0.00376
• GigE 0.008
• PCI 0.00376
• 0.0155 us/byte

22
Throughput Intel on IBM board

• Motherboard IBM das Chipset ServerWorks
  CNB20LE
• CPU Dual PIII 1GHz PCI64 bit 33 MHz
• RedHat 7.1 Kernel 2.4.14

• Max throughput 930Mbit/s
• No packet loss
• Packet loss during drop

23
PCI Intel on IBM board

• Motherboard IBM das Chipset ServerWorks
  CNB20LE
• CPU Dual PIII 1GHz PCI64 bit 33 MHz
• RedHat 7.1 Kernel 2.4.14

• uva64m02
• 1400 bytes sent
• Wait 11 us
• 9.3us on send PCI bus
• PCI bus 82 occupancy
• 5.9 us on PCI for data recv.