Design and Characterization of TMD-MPI Ethernet Bridge

1
Design and Characterization of TMD-MPI Ethernet
Bridge

• Kevin Lam
• Professor Paul Chow

2
Background

• In embedded development, Field Programmable Gate
  Arrays (FPGAs) allow mixing of hard and soft
  elements
• Issue Communication between components
• MPI is a standard for inter-process communication
  in software parallel programming
• TMD-MPI implements a hardware interface for MPI
• Unified method of communication between all
  components
• Abstracts the communication medium

3
Typical Application without TMD-MPI
4
TMD-MPI Application
5
Research Objective

• Attempt to create a module that routes MPI
  messages through on-board Ethernet hardware
• Design Objectives
• Reliability
• High Bandwidth
• Low Latency

6
Multi-Board TMD-MPI system
(your desk!)
7
Design Specification

• Behaviour
• FSL interface to on-board network (via NetIF
  module)
• Transparent inter-board communication via
  Ethernet
• FIFO behaviour
• Assumptions
• Only two systems connected, by short Ethernet
  cable
• Reliable, in-order delivery of packets

8
Design Specification
9
Design Specification

• Packet Format
• Basic Ethernet framing
• Source/Destination MAC address
• Type code set to an unused value
• CRC checksum performed by Ethernet hardware
• Length header to account for smaller-than-minimum
  packets
• Set to 0 if data is longer than minimum packet
  length
• Followed by data payload, up to maximum Ethernet
  frame length
• No handshake protocol implemented

10
Implementation

• Base design around Xilinx echo example
• First send/receive data independently
• Test using PC as second board
• Develop FSL interface, test with MicroBlaze
• Merge FSL interface to Ethernet module
• Add protocol headers, implement data packing

11
Testing

• Initial testing done using MicroBlaze processors
• High speed testing done using custom cores

Board 2
Board 1
MicroBlaze 1
MicroBlaze 2
FSL-Enet
FSL-Enet
UART
UART
PC
12
Testing

• Further testing using dedicated hardware core
• Testing high speed burst transfer from test core,
  UART displays all values received by MicroBlaze

Board 2
Board 1
MicroBlaze
Test Core
FSL-Enet
FSL-Enet
GPIO
MicroBlaze
UART
PC
13
Design Evaluation

• Reliability testing using two dedicated tester
  cores
• Transmitting sequence numbers, checking for
  mismatch/out of order

14
Design Evaluation

• Bandwidth
• Uses 2 dedicated hardware cores
• Core A writes data to the FSL-Ethernet core
  whenever the FSL is not full
• Core B reads data from the FSL-Ethernet core
  whenever the FSL has data
• Core A counts the number of clock cycles elapsed
  to write a certain amount of data to the FSL
• Result read by MicroBlaze and displayed via UART
• Measured constant 962.5 Mbit/s data throughput
  rate

15
Design Evaluation

• Latency
• Uses 2 dedicated hardware cores
• Core A sends 1 word of data to Core B via
  FSL-Ethernet
• Core B replies with 1 word
• Core A measures the clock cycles elapsed between
  sending its data and receiving the response
• Result read by MicroBlaze and displayed via UART
• Several trials yield 598-599 cycle round trip
  (125 MHz)
• One-way latency is approximately 300 cycles or
  2.4 microseconds

16
Conclusions

• Bandwidth sufficient for VGA video streaming
• Latency may be an issue for applications that
  pass data back and forth frequently
• No handshaking protocol
• Receiver in streaming applications must not be
  slower than the sender
• Messages cannot be sent until both boards are
  powered on and connected
• Future versions should attempt to implement
  handshaking
• Affect complexity and bandwidth
• Much more robust system

17
Acknowledgements

• Professor Paul Chow
• Vince Mirian
• Sami Sadaka
• Tony Zhou