Endtoend Internet Packet Dynamics

1
End-to-end Internet Packet Dynamics

• Vern Paxson
• IEEE/ACM Trans. On Networking

2
Introduction

• Main Objective
• Study of various network properties
• on the basis of experiments

3
Structure of the paper

• What to measure? Why?
• How to measure?
• Measurements
• Justification of possible errors/anomalies
• Conclusions drawn

4
Experimental Setup

• Two experiments conducted separated by 1 year
• Measurement framework
• Special packet monitoring daemons running at
  sites chosen for the experiment
• Restricted to TCP packets
• Pro Representative of real-life Internet traffic
• Con Highly varied transmission intervals.
  Impedes conventional frequency domain analysis

5
Details of the two experiments

• Measurements carried out at Poisson intervals
• Daemons at sender/receiver sends 100KB TCP Bulk
  Transfer
• Limited to time of 10 minutes
• Biased towards better network conditions

6
Differences in N1 and N2
7
Properties studied

• Out-of-order delivery
• Packet replication
• Packet corruption
• Bottleneck Bandwidth
• Packet loss
• TCP Retransmission
• Packet delay

8
Out-of-order Delivery

• If a packet arrives early, then the packets
  prior to this
• packet in ETA are assumed to be reordered
• Conclusions
• Data packets are re-ordered more than ack packets
• (N1 2 Data, 0.6 Ack N2 0.3 Data, 0.1
  Ack)
• Reordering is highly asymmetric
• The significant reordering shouldnt be
  misinterpreted as suggestive of erroneous packet
  transmissions in the Internet.
• Reordering isnt restricted to route changes, but
  can also be due to routing message updating
  periods

9
Packet reordering and Dup Acks

• Current dup-ack threshold is 3, i.e. sender
  retransmits packet after receiving 3 dup-acks
  from receiver (heuristic choice)
• If no reordering possible, then threshold would
  be 1.

10
Improving fast retransmission

• To improve fast retransmission
• Delayed dup-acks
• Modify threshold
• Increasing threshold Diminishes fast retransmit
  chances
• Combination of delayed dup-acks, and threshold
  reduction at sender end.
• SACK (Selective Ack)

11
Other pathologies

• Packet replication
• Extremely Rare
• Packet Corruption
• Packet corrupted during transmission possibly
  due to noise
• TCP 16-bit checksum not enough due to significant
  corruption rate.
• Acks dont suffer much corruption
• Smaller packet size

12
Bottleneck Bandwidth of a connection

• Bound on the rate at which data can be
  transmitted from sender to receiver
• ForBottleneck bandwidth B
• packet length - b
• Minimum time before next packet is
• serviced Qb b/B
• Basis of packet pair approach

13
Packet Pair

• If two packets are sent with delay lt Qb to the
• destination, then a delay of Qb is caused while
• in flight. This causes the behavior of the two
• packets to be correlated.
• Here, Qb is the bottleneck bandwidth for
• the connection.

14
Packet Pair (Contd)

• Two kinds
• Sender Based Packet Pair (SBPP)
• Measurements made at the senders end
• Cannot distinguish between the connection
  bottlenecks of forward and reverse paths
• Ack compression
• Receiver Based Packet Pair (RBPP)
• Measured at receiver end
• Receiver has full of knowledge of whether delays
  were caused other than due to queuing of the
  packets from the same flow.

15
Packet Pair - Problems

• Advantages of RBPP (Receiver Based Pkt Pair)
• Eliminates noise
• Accounts for asymmetry in delays
• Problems
• Out-of-order ?Tr (Delay seen at receiver) lt 0
• This would lead to Bs estimate being lt 0
• Out-of-order is caused if route for the two
  packets differ. Then, assumption that packets
  are queued is invalid.

16
Packet Pair Problems (Contd)

• Clock resolution
• If minimum interval receiver can discern is Cr
• If ?Tr lt Cr , then receiver makes an error in Bs
• estimate.
• Bunch k packets, so that ?Tk,r gt Cr.
• Changes in bottleneck bandwidth
• Multi-channel bottleneck links
• If there exist multiple channels, then packets
  wont
• queue behind each other -gt erroneous estimate of
  bottleneck
• bandwidth.

17
Packet Bunch Modes

• For good bottleneck estimation, PBM is adopted.
  Here, packets are bunched and sent.
• Allows for varying bottleneck values
• Allows for clock resolution
• Allows for multi-channel connections

18
Bottleneck Estimation using PBMs

• If there exist two strong modes one at start
  and one at end, then there exists bottleneck
  change
• If there exist two strong modes one for bunch
  of m, and another for gtm then m-channel link
  exists
• Both cases (1) and (2).
• Bottleneck estimates can be quite asymmetric.

19
Packet Pair vis-a-vis PBM

• Packet Pair may be evaluated as a specific case
  of PBM with packet bunch size 2
• Packet pair agrees with PBM when
• Estimation is at receiver end
• Undergoes filtering methods of PBM
• Receiver knows about out-of-order arrivals a
  priori
• Multi-channel effects are not considered

20
Packet Loss

• N1 has a lower data packet loss rate than N2
• Is this due to N2s bigger window size? (Slide
  5)
• Estimate data packet loss and ack packet loss
  rates
• N2s ack packet loss rate is still high. Hence,
  the answer is no.
• Conclusion Packet loss rates doubled during
  1995!

21
Data Packet loss Vs. Ack loss

• Waiting time calculation
• Let min and max estimates of bottleneck bandwidth
  be B- and B.
• Waiting time for 1st packet Qb. (Upper bound)
• Qb b/B- (Service time for 1 packet)
• Waiting time for ith packet
• ?i Qb max (Ti-1 ?i-1) Ti, 0
• where Ti is the time when the ith packet is sent
  

22
Data Packet loss Vs. Ack loss (Contd)

• If Ti lt Ti-1 ?i-1, then packet i is queued,
  else its unqueued.
• Queued packets are more likely to be lost than
  unqueued packets
• Acks are more likely to be lost than unqueued
  packets but less likelier than queued packets
• Acks dont adapt to network conditions
• Smaller size
• Packet loss on forward and reverse directions
  highly uncorrelated

23
Loss bursts

• Extent to which packet loss occurs in a burst
  i.e. multiple packets are lost
• Conditional Packet loss probability that a
  packet is lost on the condition that the packet
  prior to it is also lost
• plc (Cond. Packet loss probability) is more for
  queued packets than unqueued packets
• Loss correlations are short-lived if time gap
  between two packets is long, then plc ? plu

24
Loss bursts (Contd)

• Extent of bunching of packet losses
• Varies over the timescale
• Consistent with Pareto distribution with shape
  parameter ? 1.06 (? 2) ? infinite variance or
  extreme variability
• Loss bursts may be reduced by drop-tail queuing
  (e.g. RED)

25
TCP Retransmission

• Redundant retransmission
• Unavoidable All acks to data packets were lost
• Coarse feedback Lack of fine-grained acks (like
  SACK)
• Bad RTO Miscalculated RTO if waited longer,
  then ack wouldve been received
• quite rare
• (RTO Retransmission Time Out)

26
TCP Retransmission (Contd)

• Packet loss during fast recovery
• If packets lost during retransmit then connection
  timeout
• Thankfully rare!
• Significant proportion of packets lost prior to
  retransmission need to have SACK, or RED to
  reduce packet losses itself

27
Packet Delay

• OTT (One-way Transit Time) considered
• Not necessarily (RTT)/2
• Often asymmetric (at sender and receiver)
• Timing compression
• If ?Ts gt ?Tr for a flight of packets
• Could occur if the servicing time for a packet at
  a router is non-uniform giving time for other
  packets to arrive and be serviced immediately
  afterwards

28
Packet Delay (Contd)

• Metric to determine ack compression
• If (?Tr)/(?Ts) lt 1 Ack compression
• Accounting for clock resolution inaccuracies
• ? (?Tr Cr)/(?Ts - Cs)
• If ? lt 0.75, a compression event has occurred
• Cr Min. clock resolution at receiver
• Cs Min. clock resolution at sender
• All compression events are small

29
Packet Delay (Contd)

• Ack compression problems
• Data packets from sender will go out more
  frequently i.e. senders window will be advanced
  faster may cause network stress
• Sender-based measurements will interpret it as
  increased bandwidth availability

30
Packet Delay (Contd)

• Data packet timing compression
• Not reflected in the metric used for ack
  compressions, i.e. (? (?Tr Cr)/(?Ts - Cs)) lt
  0.75 if compressed after expanding due to
  bottleneck.
• Hence, compression event occurs if packet arrival
  rate Ra gt 2B (B - Upper bound on bottleneck
  bandwidth)
• Rarer than ack compression

31
Queuing time scales

• Queuing variation over a time-scale ?
• Partition TCP packet arrivals into intervals of
  time ? i.e. interval ti-1, ti) where ti ti-1
  ?
• Interval ti-1, ti)s queuing variation is ml
  mr
• where ml, mr are the median OTTs of the left
  and right halves
• Queuing variation for ? ?Q? ? median of
• ml mr over all such intervals

32
Queuing Time Scales (Contd)

• If queuing variation is maximum over a timescale
  ? thats less than connections RTT, then no
  point in adapting to queuing variations since
  feedback wont be received
• Queuing variations occur in the timescale 0.1 1
  sec (order of RTTs) but can extend to larger
  durations

33
Available Bandwidth

• OTT variations are indicative of delay caused due
  to packets having to queue behind their
  predecessors, i.e. ?i ?i - Qb for packet i
• Let ?i ? Measured OTT Minimum OTT for packet i.
  This is indicative of delay caused due to packets
  waiting to be serviced while other connections
  are also possibly being serviced
• If connection only existing connection on
  network, ?i ?i

34
Available Bandwidth (Contd)

• ? ? ?i(?i Qb)/ ?j(?j Qb)
• is the proportion of packet delay due to its
• queuing behind its predecessors and loading the
• network, and hence a good estimate of the
• proportion of bandwidth available to the
• connection
• If ? ? 1, then ?i ? ?j, or all load is due to
  this connection. If ? ? 0, then load exerted (and
  hence service received) due to this connection on
  the network is negligible

35
Available Bandwidth (Contd)

• ? approximates available bandwidth quite well
  without having to load the network with extra
  measurement daemons for this specific purpose
• ? displays a leftward density shift (lesser
  available bandwidth) for European connections
• ? is lower for higher bottleneck bandwidths
• Such paths are shared among a large number of
  connections

36
Conclusions

• In the Internet
• Packet reordering is common
• Packet loss rates doubled between 1994 and 95
• Considerable geographical differences in loss
  rates
• Packet losses are prone to occur in bursts
• TCP retransmissions perform well with SACK
• Significant routing asymmetries exist