UDT: UDP based Data Transfer

1
UDT UDP based Data Transfer

• Yunhong Gu Robert Grossman
• Laboratory for Advanced Computing
• University of Illinois at Chicago

2
Outline

• Background
• UDT Protocol
• UDT Congestion Control
• Implementation/Simulation Results
• Summary

3
Background

• Distributed data intensive applications over wide
  area optical networks
• Grid computing, access of bulk scientific data,
  data mining, high resolution video, etc.
• Transport protocol support
• Efficient and fair bandwidth unitization
• TCP does not work!

4
Trans-Atlantic TCP Performance

• Chicago -gt Amsterdam, 1Gbps link capacity, 110ms
  RTT
• TCP 5Mbps _at_ default setting (64KB buffer)
• TCP 100Mbps _at_ 12MB buffer (1Gbps110ms)
• Parallel TCP 800Mbps _at_ 64 TCP concurrent flows,
  with each having 1MB buffer
• Two concurrent TCP flows, 1 from Chicago to
  Amsterdam, 1 within Chicago local networks
• 2Mps vs. 940Mbps!

5
Why TCP Fails

• Discover/recover slow on high BDP links
• Increase 1 byte per RTT
• Drastic decrease in sending rate
• Fairness bias on longer RTT links
• More prone to link error in high BDP links
• B throughout in packets per second, p loss
  rate

6
Requirements to the New Protocol

• FAST
• High utilization of the abundant bandwidth either
  with single or multiplexed connections
• FAIR
• Intra-protocol fairness, independent of RTT
• FRIENDLY
• TCP compatibility

7
Use Scenarios

• Small number of sources shares abundant bandwidth
• Bulk data transfer
• Most of the packets can be packed in maximum
  segment size (MSS) in a UDT session
• MSS can be set up by applications and the optimal
  value is the path MTU

8
Whats UDT?

• UDT UDP based Data Transfer
• Reliable, application level, duplex, transport
  protocol, over UDP with congestion control
• Implementation Open source C library
• Two orthogonal parts
• The UDT protocol framework that can be
  implemented above UDP, with any suitable
  congestion control algorithms
• The UDT congestion control algorithm, which can
  be implemented in any transport protocols such as
  TCP

9
Packet Structure

• Data Packet
• Header 1bit flag 31bit sequence number
• Control Packet
• Header 1bit flag 3bit type 12bit reserved
  16bit ACK seq. no. (0 - 32n)bit control info
• Type ACK, ACK2, NAK, Handshake, Keep-alive, and
  Shutdown
• Actual size of a UDT packet can be ascertained
  from UDP header

10
Data Packet
0 Packet Sequence Number
User Data Payload User Data Payload

• Flag Bit 0
• UDT uses 31-bit packet based sequence number,
  ranging from 0 and (231 - 1)
• Sequence number may be wrapped if it exceeds the
  maximum available number

11
Control Packet
1 type reserved ACK Seq. No.
Control Information Field Control Information Field Control Information Field Control Information Field

• Flag Bit 1
• type 3-bit
• handshake (000), shutdown (101), keep-alive (001)
• ACK (010), ACK2 (110), NAK (011)
• UDT uses sub-sequencing each ACK and related
  ACK2 are assigned a 16-bit unique ACK sequence
  number

12
Acknowledgements

• Selective acknowledgement (ACK)
• Generated at every constant interval to send back
  largest continuously received sequence number of
  data packets.
• The sender sends back an ACK2 to the receiver for
  each ACK (sub-sequencing).
• Also carries RTT, packet arrival speed, and
  estimated link capacity.
• Explicit negative acknowledgement (NAK)
• Generated as soon as loss is detected.
• Loss information may be resent if receiver has
  not received the retransmission after an
  increasing interval.
• Loss information is compressed in NAK.

13
Timing

• Packet Scheduling Timer
• Tuned by Rate Control
• High precision in CPU clock cycles
• Rate Control Timer trigger rate control
• RCTP 0.01 seconds
• ACK Timer trigger acknowledgement
• ATP RCTP

14
Timing (cont.)

• NAK Timer trigger negative acknowledgement
• NTP RTT
• Retransmission Timer trigger retransmission
  based on time-out and maintain connection status
• RTP (exp-count 1) RTT ATP
• where exp-count is the number of continuous
  time-out

15
UDT Architecture
16
Congestion Control

• Rate based congestion control (Rate Control)
• RC tunes the packet sending period.
• RC is triggered periodically at the sender side.
• RC period is constant of 0.01 seconds.
• Window based flow control (Flow Control)
• FC limits the number of unacknowledged packets.
• FC is triggered on each received ACK at the
  sender side.

17
Rate Control

• AIMD Increase parameter is related to link
  capacity and current sending rate Decrease
  factor is 1/9, but not decrease for all loss
  events.
• Link capacity is probed by packet pair, which is
  sampled UDT data packets.
• Every 16th data packet and it successor packet
  are sent back to back to form a packet pair.
• The receiver uses a median filter on the interval
  between the arrival times of each packet pair to
  estimate link capacity.

18
Rate Control (cont.)

• 1. If loss rate is greater than 1, do not
  increase
• 2. Number of packets to be increased in next RCTP
  time is
• where B is estimated link capacity, C is
  current sending rate. Both are in packets or
  packets per second. MSS is the packet size in
  bytes. ß 1.5 10-6.
• 3. Recalculate packet sending period (STP).

19
Rate Control (cont.)
B 10Gbps, MSS 1500 bytes
C (Mbps) B - C (Mbps) Increase Param. (Pkts)
0, 9000) (1000, 10000 10
9000, 9900) (100, 1000 1
9900, 9990) (10, 100 0.1
9990, 9999) (1, 10 0.01
9999, 9999.9) (0.1, 1 0.001
9999.9 lt0.1 0.00067
20
Rate Control (cont.)

• Decrease sending rate by 1/9, (or equivalently,
  increase packet sending period by 1.125), only if
• Received an NAK, whose last lost sequence number
  is greater than the largest sequence number when
  last decrease occurred or
• The number of loss events since last decrease has
  exceeded a threshold, which increases
  exponentially and is reset when condition 1 is
  satisfied.
• No data will be sent out for the next RCTP time
  if a decrease occurs.
• Help to clear congestion.

21
Flow Control
BDP

• W W0.875 AS(RTTATP)0.125
• AS is the packets arrival speed at receiver side.
• The receiver records the packet arrival
  intervals. AS is calculated from the average of
  latest 16 intervals after a median filter.
• It is carried back within ACK.

22
Slow Start

• Flow window starts at 2 and increases to the
  number of acknowledged packets, until the sender
  receives an NAK or reaches the maximum window
  size, when slow start ends.
• Packet sending period is 0 during slow start
  phase and set to the packet arrival interval at
  the end of the phase.
• Slow start only occurs at the beginning of a UDT
  session.

23
Implementation Performance
24
Implementation Intra-protocol Fairness
25
Implementation TCP Friendliness
26
Implementation TCP Friendliness (cont.)
27
Implementation File Transfer
1Gbps/15.9ms
1Gbps/110ms
Canarie
StarLight
SARA
Disk R 800Mbps W 550Mbps
Disk R 800Mbps W 500Mbps
Disk R 1300Mbps W 900Mbps
To StarLight Canarie SARA
From StarLight Canarie SARA
StarLight 460 505 560
Canarie 440 502 -
SARA 441 - 660
28
Simulation UDT Throughput at Different Bandwidth
and RTT
29
Simulation Performance of Concurrent UDT Flows
30
Simulation Intra-protocol Fairness
31
Simulation RTT Independence
32
Simulation TCP Friendliness
33
Simulation Convergence/Stability
34
Simulation Complex Scenario
Link capacity Mbps
Node DropTail
Flow and its ID
Flow ID 1 2 3 4 5 6
Throughput (Mbps) 89.3 90.0 5.18 41.7 50.8 4.78
35
Simulation Multi-bottleneck
X 0.1 1 10 20 40 60
AB 198.8 189.2 180.1 170.9 152.5 137.6
AC 0.098 0.979 9.955 19.88 39.46 57.70
X 80 100 120 140 160 180
AB 108.4 104.6 100.8 101.3 100.7 100.3
AC 73.49 92.42 98.47 98.04 98.65 99.00
36
Summary

• UDT Protocol
• Application level upon UDP
• Selective acknowledgement / explicit negative
  acknowledgement
• UDT Congestion Control
• Rate Control
• Bandwidth estimation for fast probing available
  bandwidth and fast recovery
• AIMD for fairness
• Constant rate control interval
• Flow Control
• Dynamic flow window according to packet receiving
  speed

37
UDT Characters

• Good use of available bandwidth
• Application level - no changes in router and
  operating system
• No manual tuning
• Fair and Friendly intra-protocol fairness, TCP
  friendliness, and RTT independence.
• Open source

38
Thank You!

• LAC www.lac.uic.edu
• UDT sourceforge.net/projects/dataspace
• Internet Draft draft-gg-udt-01.txt