A Study of Internet Packet Reordering

1
A Study of Internet Packet Reordering

• ICOIN 2004
• Yi Wang, Guohan Lu, Xing Li
• Dept. of Electronic Engineering, Tsinghua Univ.,
  China
• China Education and Research Network (CERNET)

2
Motivation

• Packet reordering affects TCP performance
• Causes Unnecessary Retransmission (mistake for
  loss)
• Limits Transmission Speed (TCP cwnd?)
• Reduces Receiver's Efficiency (buffer burst)
• Previous studies get discrepant results of the
  prevalence of reordering
• Few work has been done with the correlation
  between reordering and network topology

3
Problem Definition

• The frequency and magnitude of packet reordering
• Any rules in distribution?
• Any characteristics at small time scale?
• The correlation between reordering and network
  topology

4
Related Work

• Prevalence
• Paxsons study (1997) 36 and 12 of sessions,
  2.0 and 0.26 of packets
• Bennett et als work (1997-1998) 90 packets
• Jaiswal et als measurement (2002) 5 packets
• Causes Local parallelism (Bennett et al, 1999
  Liu, 2002)
• Measurement
• At end point(s) Paxson, Bennett et al, Bellardo
  and Savage,
• At the gear of backbone Jaiswal et al
• Improvement SACK, DSACK (Zhang et al, 2002
  Blanton and Allman, 2002)

5
Measurement Methodology (1)

• Measurement Environment
• Single point measurement
• 10,647 web sites in CERNET (China Education and
  Research Network)
• Web page crawling (HTTP 80 port, using wget)
• Testbed setup

6
Measurement Methodology (2)

• Reorder-judging Algorithm

7
Results

• Preliminary measurement (Two categories Reorder
  Sites / Ordinary Sites)
• Three-week Measurement (May 16 - June 5, 2003)
• 8 measurements per day (every 3 hours), totally
  168 measurements
• 208 thousand connections with totally 3.3 million
  data packets
• 5.79 of all 10,647 web sites (616 Reorder Sites)
  experienced reordering at least once
• 3.197 packets of the Reorder Sites were
  reordered

8
Distribution of packet reordering
Percentage of Reordering Reorder Sites
2.39-4.27 with a mean of 3.197 Ordinary
Sites always below 0.14 with a mean of 0.017

• Huge and steady discrepancy of reordering rate
  between the two categories.
• This discrepancy indicates that reordering is
  strongly site-dependent and occurs mainly in some
  certain parts of the Internet.

9
Reordering frequency of Reorder Sites

• Reordering frequency of the 616 Reorder Sites.
• About 20 of the Reorder Sites are with a
  reordering frequency higher than 80. These sites
  contribute the bulk of reordering in our
  experiment.

10
Distribution of TTL

• Reorder Sites (with an average hops value 13.8)
  tend to be farther away from the measurement host
  than the Ordinary Sites (with average hops value
  12.9)

11
Distinguishing Reordering and Loss

• TCP would mistake reordering for loss when
  meeting sequence hole at the receiver one of
  the main reasons that packet reordering affects
  the TCP performance.
• Since loss can not be confirmed until
  retransmitted packet arrived, we studied the time
  lag and packet lag of both packet reordering and
  retransmission.
• What we found indicates that we can distinguish
  them by setting certain threshold.

12
Comparison of time lag

• Cumulative distribution of time lag of reordering
  and retransmission

• 90 of reordered packets arrive at the receiver
  with time lag less than 5.1 ms, while only 3.5
  of retransmitted packets arrive then.
• We find 12.8 ms is a relative good threshold in
  our experiment 95 of reordered packets arrived
  but only 8.3 of retransmitted packets arrived.

13
Comparison of packet lag
Distribution of packet lag

• Reordering
• Packet lag 1 86.5
• Packet lag 2 95.3

• Retransmission
• Packet lag 3 78.8

14
Reordering and Network Topology

• Use traceroute to get backward-path routing tree
  (from the remote sites to the measurement host)
• The forward-path and backward-path routing trees
  are almost identical in CERNET, thanks to the
  symmetric topology
• Introduce a metric (Reorder Ratio) to each
  routor
• Rr R/T
• R is the number of Reorder Sites that go through
  the router
• T is the total number of sites that go through it
• If a router has one of the following two
  characters, it is probably a reorder-generating
  spot in the network

15
Case 1

• The routers Rr is by far higher than its
  previous-hops Rr and other routers Rr of the
  same hop. Its next-hop routers also have got high
  and close Rr

16
Case 2

• All of the routers previous-hop routers have got
  high Rr , and its next-hop routers also get high
  Rr

17
Summary

• Conclusions
• Reordering is significantly site-dependent in the
  Internet.
• Certain threshold can be found to effectively
  help distinguish reordering and loss on some
  heavily reordering paths.
• A novel and relatively reliable approach to infer
  reorder-generating spots.
• Further work in progress
• Multi-point reordering measurement with more
  precisely data crawling
• Possible modeling?

18

• Questions or Comments? ?

Thank you!