Bandwidth estimation in computer networks: measurement techniques

1
Bandwidth estimation in computer networks
measurement techniques applications

• Constantine Dovrolis
• College of Computing
• Georgia Institute of Technology

2
The Internet as a black box

• Several network properties are important for
  applications and transport protocols
• Delay, loss rate, capacity, congestion, load, etc
• But routers and switches do not provide direct
  feedback to end-hosts (except ICMP, also of
  limited use)
• Mostly due to scalability, policy, and simplicity
  reasons

3
Can we guess what is in the black box?

• End-systems can infer network state through
  end-to-end (e2e) measurements
• Without any feedback from routers
• Objectives accuracy, speed, non-intrusiveness

4
Bandwidth estimation in packet networks

• Bandwidth estimation (bwest) area
• Inference of various throughput-related metrics
  with end-to-end network measurements
• Early works
• Keshavs packet pair method for congestion
  control (89)
• Bolots capacity measurements (93)
• Carter-Crovellas bprobe and cprobe tools (96)
• Jacobsons pathchar - per-hop capacity estimation
  (97)
• Melanders TOPP method for avail-bw estimation
  (00)
• In last 3-5 years
• Several fundamentally new estimation techniques
• Many research papers at prestigious conferences
• More than a dozen of new measurement tools
• Several applications of bwest methods
• Significant commercial interest in bwest
  technology

5
Overview

• This talk is an overview of the most important
  developments in bwest area over the last 10 years
• A personal bias could not be avoided..
• Overview
• Bandwidth-related metrics
• Capacity estimation
• Packet pairs and CapProbe technique
• Available bandwidth estimation
• Iterative probing Pathload, Pathvar and
  PathChirp
• Comparison of tools
• Applications
• SOcket Buffer Auto-Sizing (SOBAS)

6
Network bandwidth metrics

• Ravi S.Prasad, Marg Murray, K.C. Claffy,
  Constantine Dovrolis,
• Bandwidth Estimation Metrics, Measurement
  Techniques, and Tools,
• IEEE Network, November/December 2003.

7
Capacity definition

• Maximum possible end-to-end throughput at IP
  layer
• In the absence of any cross traffic
• Achievable with maximum-sized packets

• If Ci is capacity of link i, end-to-end capacity
  C defined as
• Capacity determined by narrow link

8
Available bandwidth definition

• Per-hop average avail-bw
• Ai Ci (1-ui)
• ui average utilization
• A.k.a. residual capacity
• End-to-end avg avail-bw A
• Determined by tight link
• ISPs measure per-hop avail-bw passively (router
  counters, MRTG graphs)

9
The avail-bw as a random process

• Instantaneous utilization ui(t) either 0 or 1
• Link utilization in (t, tt)
• Averaging timescale t
• Available bandwidth in (t, tt)
• End-to-end available bandwidth in (t, tt)

10
Available bandwidth distribution

• Avail-bw has significant variability
• Need to estimate second-order moments, or even
  better, the avail-bw distribution
• Variability depends on averaging timescale t
• Larger timescale, lower variance
• Distribution is Gaussian-like, if t gt100-200 msec
  and with sufficient flow multiplexing

11
E2E Capacity estimation (a)Packet pair
technique

• Constantine Dovrolis, Parmesh Ramanathan, David
  Moore,
• Packet Dispersion Techniques and Capacity
  Estimation,
• In IEEE/ACM Transactions on Networking, Dec
  2004.

12
Packet pair dispersion

• Packet Pair (P-P) technique
• Originally, due to Jacobson Keshav
• Send two equal-sized packets back-to-back
• Packet size L
• Packet trx time at link i L/Ci
• P-P dispersion time interval between last bit of
  two packets
• Without any cross traffic, the dispersion at
  receiver is determined by narrow link

13
Cross traffic interference

• Cross traffic packets can affect P-P dispersion
• P-P expansion capacity underestimation
• P-P compression capacity overestimation
• Noise in P-P distribution depends on cross
  traffic load
• Example Internet path with 1Mbps capacity

14
Multimodal packet pair distribution

• Typically, P-P distribution includes several
  local modes
• One of these modes (not always the strongest) is
  located at L/C
• Sub-Capacity Dispersion Range (SCDR) modes
• P-P expansion due to common cross traffic packet
  sizes (e.g., 40B, 1500B)
• Post-Narrow Capacity Modes (PNCMs)
• P-P compression at links that follow narrow link

15
E2E Capacity estimation (b)CapProbe

• Rohit Kapoor, Ling-Jyh Chen, Li Lao, Mario Gerla,
  M. Y. Sanadidi,
• "CapProbe A Simple and Accurate Capacity
  Estimation Technique,"
• ACM SIGCOMM 2004

16
Compression of p-p dispersion

• First packet queueing gt compressed dispersion
• Capacity overestimation

17
Expansion of p-p dispersion

• Second packet queueing gt expansion of dispersion
  
• Capacity underestimation

18
CapProbe

• Both expansion and compression are result queuing
  delays
• Key insight packet pair that sees zero queueing
  delay would yield exact estimate
• measure RTT for each probing packet of packet
  pair
• estimate capacity from pair with minimum RTT-sum

Capacity
19
Measurements - Internet, Internet2

• CapProbe implemented using PING packets, sent
  in pairs

To UCLA-2 UCLA-2 UCLA-3 UCLA-3 UA UA NTNU NTNU
To Time Capacity Time Capacity Time Capacity Time Capacity
Cap Probe 003 5.5 001 96 002 98 007 97
Cap Probe 003 5.6 001 97 004 79 007 97
Cap Probe 003 5.5 002 97 017 83 022 97
Pathrate 610 5.6 016 98 519 86 029 97
Pathrate 614 5.4 016 98 520 88 025 97
Pathrate 65 5.7 016 98 518 133 025 97
Pathchar 2112 4.0 2249 18 3 hr 34 3 hr 34
Pathchar 2043 3.9 2741 18 3 hr 34 3 hr 35
Pathchar 21.18 4.0 2947 18 3 hr 30 3 hr 35
20
Avail-bw estimation (a)Pathload

• Manish Jain, Constantine Dovrolis,
• End-to-End Available Bandwidth Measurement
  Methodology, Dynamics, and Relation with TCP
  Throughput,
• IEEE/ACM Transactions on Networking, Aug 2003.

21
Probing methodology

• Sender transmits periodic packet stream of rate R
• K packets, packet size L, interarrival T L/R
• Receiver measures One-Way Delay (OWD) for each
  packet
• D(k) tarv(k) - tsnd(k)
• OWD variations ?(k) D(k1) D(k)
• Independent of clock offset between
  sender/receiver
• With stationary fluid-modeled cross traffic
• If R gt A, then ?(k) gt 0 for all k
• Else, ?(k) 0 for all k

22
Self-loading periodic streams

• Increasing OWDs means RgtA
• Almost constant OWDs means RltA

23
Example of OWD variations

• 12-hop path from U-Delaware to U-Oregon
• K100 packets, A74Mbps, T100µsec
• Rleft 97Mbps, Rright34Mbps

• Ri

24
Iterative probing and Pathload
Avail-bw time series from NLANR trace

• Iterative probing in Pathload
• Send multiple probing streams (duration t) at
  various rates
• Each stream samples path at different time
  interval
• Outcome of each stream is either Ri gt A or Ri lt A
• Estimate upper and lower bound for avail-bw
  variation range

25
Avail-bw estimation (b) percentile estimation

• Manish Jain, Constantine Dovrolis,
• End-to-End Estimation of the Available Bandwidth
  Variation Range,
• ACM SIGMETRICS 2005

26
Avail-bw distribution and percentiles

• Avail-bw random process, in timescale t At(t)
• Assume stationarity
• Marginal distribution of At
• Ft(R) Prob At R
• Ap pth percentile of At, such that p Ft(Ap)
• Objective Estimate variation range AL, AH for
  given averaging timescale t
• AL and AH are pL and pH percentiles of At
• Typically, pL 0.10 and pH 0.90

27
Percentile sampling

• Question Which percentile of At corresponds to
  rate R?
• Given R and t, estimate Ft(R)
• Assume that Ft(R) is inversible
• Sender transmits periodic packet stream of rate R
• Length of stream measurement timescale t
• Receiver classifies stream as
• Type-G if At R I(R) 1 with probability Ft(R)
• Type-L otherwise I(R) 0 with probability
  1-Ft(R)
• Collect N samples of I(R) by sending N streams of
  rate R
• Number of type-G streams I(R,N) Si Ii(R)
• E I(R,N) Ft(R) N
• Percentile rank of probing rate R Ft(R)
  EI(R,N) / N
• Use I(R,N)/N as estimator of Ft(R)

28
Non-parametric estimation

• Does not assume specific avail-bw distribution
• Iterative algorithm
• Stationarity requirement across iterations
• N-th iteration probing rate Rn
• Use percentile sampling to estimate percentile
  rank of Rn
• To estimate the upper percentile AH with pH
  Ft(AH)
• fn I(Rn,N)/N
• If fn is between pHr, report AH Rn
• Otherwise,
• If fn gt pH r, set Rn1 lt Rn
• If fn lt pH - r, set Rn1 gt Rn
• Similarly, estimate the lower percentile AL

29
Validation example

• Verification using real Internet traffic traces

b0.05
b0.15

• Non-parametric estimator tracks variation range
  within 10-20
• Selection of b depends on traffic
• Traffic spikes/dips may not be detected if b is
  too small
• But larger b causes larger MSRE

30
Parametric estimation

• Assume Gaussian avail-bw distribution
• Justified assumption for large degree of traffic
  multiplexing
• And/or for long averaging timescale (gt200msec)
• Gaussian distribution completely specified by
• Mean m and standard deviation st
• pth percentile of Gaussian distribution
• Ap m st f-1(p)
• Sender transmits N probing streams of rates R1
  and R2
• Receiver determines percentiles ranks
  corresponding to R1 and R2
• m and st can be then estimated by solving
• R1 m st f-1(p1)
• R2 m st f-1(p2)
• Variation range is then calculated from
• AH m st f-1(pH)
• AL m st f-1(pL)

31
Validation example

• Verification using real Internet traffic traces
• Evaluated with both Gaussian and non-Gaussian
  traffic

Gaussian traffic
non-Gaussian traffic

• Parametric algorithm is more accurate than
  non-parametric algorithm, when
• traffic is good match to Gaussian model
• in non-stationary conditions

32
Avail-bw estimation (c)PathChirp

• Vinay Ribeiro, Rolf Riedi, Jiri Navratil, Rich
  Baraniuk, Les Cottrell,
• PathChirp Efficient Available Bandwidth
  Estimation,
• PAM 2003

33
Chirp Packet Trains

• Exponentially decrease packet spacing within
  packet train
• Wide range of probing rates
• Efficient few packets

34
Chirps vs. CBR Trains

• Multiple rates in each chirping train
• Allows one estimate per-chirp
• Potentially more efficient estimation

35
CBR Cross-Traffic Scenario

• Point of onset of increase in queuing delay gives
  available bandwidth

36
Comparison with Pathload

• 100Mbps links
• pathChirp uses 10 times fewer bytes for
  comparable accuracy

Available bandwidth Efficiency Efficiency Accuracy Accuracy
Available bandwidth pathchirp pathload pathChirp 10-90 pathload Avg.min-max
30Mbps 0.35MB 3.9MB 19-29Mbps 16-31Mbps
50Mbps 0.75MB 5.6MB 39-48Mbps 39-52Mbps
70Mbps 0.6MB 8.6MB 54-63Mbps 63-74Mbps
37
Experimental comparison of avail-bw estimation
tools

• Alok Shriram, Marg Murray, Young Hyun, Nevil
  Brownlee, Andre Broido, k claffy,
• Comparison of Public End-to-End Bandwidth
  Estimation Tools on High-Speed Links,
• PAM 2005

38
Testbed topology
39
Accuracy comparison - SmartBits
Direction 1, Measured AB
Direction 2, Measured AB
Actual AB
40
Accuracy comparison - TCPreplay
Actual Available Bandwidth
Measured Available Bandwidth
41
Whats wrong with Spruce?

• Spruce was proposed as fast accurate estimation
  method
• Strauss et al., IMC03 example of direct probing
• With a single-link model
• Sender sends periodic stream with input rate Ri
• Receiver measures output rate Ro
• Avail-bw A given by
• Assumptions
• Tight link capacity Ct is known
• Input rate Ri can be controlled by source
• But what would happen in a multi-hop path?
• Ri can be less than source rate (due to previous
  link with lower capacity) and, even worse, it can
  be unknown!

42
Measurement latency comparison

• Abing 1.3 to 1.4 s
• Spruce 10.9 to 11.2 s
• Pathload 7.2 to 22.3 s
• Patchchirp 5.4 s
• Iperf 10.0 to 10.2 s

43
Probing traffic comparison
44
Applications of bandwidth estimation
45
Applications of bandwidth estimation

• Large TCP transfers and congestion control
• Bandwidth-delay product estimation
• Socket buffer sizing
• Streaming multimedia
• Adjust encoding rate based on avail-bw
• Intelligent routing systems
• Overlay networks and multihoming
• Select best path based on capacity or avail-bw
• Content Distribution Networks (CDNs)
• Choose server based on least-loaded path
• SLA and QoS verification
• Monitor path load and allocated capacity
• End-to-end admission control
• Network spectroscopy
• Several more..

46
Application-1Improved TCP Throughput with BwEst

• Ravi S. Prasad, Manish Jain, Constantine
  Dovrolis,
• Socket Buffer Auto-Sizing for High-Performance
  Data Transfers,
• Journal of Grid Computing, 2003

47
TCP performs poorly in fat-long paths

• Basic idea in TCP congestion control
• Increase congestion window until packet loss
  occurs
• Then reduce window by a factor of two
  (multiplicative decrease)

• Consequences
• Increased loss-rate
• Avg. throughput less than available bandwidth
• Large delay-variation
• Example 1Gbps path, 100msec RTT, 1500B pkts
• Single loss 4000 pkts window reduction
• Recovery from single loss 400 sec
• Need very small loss probability (?lt210-8) to
  saturate path

48
TCP background

• Socket-layer buffers
• Send buffer Bs
• Receive buffer Br

• TCP windows
• Send Ws lt Bs
• Congestion Wc
• Receiver (advertised) Wrlt Br

• Ws min Bs, Wc, Wr
• Ideally, TCP send window Ws should be set to
  connections bandwidth-delay product
• Bandwidth connections fair share
• Delay connections RTT
• Achieve efficiency, fairness, no congestion

49
Can we avoid congestive losses w/o changing TCP?

• Ws min Wc, Wr
• But Wrlt S
• S rcv socket buffer size
• We can limit S so that connection
• is limited by receive window Ws S
• Non-congested path
• Throughput R(S) S/T(S)
• R(S) increases with S
• until it reaches MFT
• MFT Maximum Feasible Throughput (onset of
  congestion)
• Objective set S close to MFT, to avoid any
  losses and stabilize TCP send window to a safe
  value
• Congested path
• Path with losses before start of target transfer
• Limiting S can only reduce throughput

50
Socket buffer auto-sizing (SOBAS)

• Apply SOBAS only in non-congested paths
• Receiving application measures rcv-throughput
  Rrcv
• When receive throughput becomes constant
• Connection has reached its maximum throughput
  Rmax
• If the send window becomes any larger, losses
  will occur
• Receiving application limits rcv socket buffer
  size S
• S RTT Rmax
• With congestion unresponsive cross traffic Rmax
  A
• With congestion responsive cross traffic Rmax
  MFT
• SOBAS does not require any changes in TCP
• But estimation of RTT and congestion-status
  requires ping-like probing

51
Measurements at WAN path
Non-SOBAS
SOBAS

• Path GaTech to LBL,CA
• SOBAS flow get higher average throughput than
  non-SOBAS flow because former avoids all
  congestion losses
• SOBAS does not significantly increase paths RTT

52
To conclude..
53
Things I have not talked about

• Other estimation tools techniques
• Abing, netest, pipechar, STAB, pathneck, IGI/PTR,
  
• Per-hop capacity estimation (pathchar, clink,
  pchar)
• Other applications
• Bwest in new TCPs (e.g., TCP Westwood, TCP FAST)
• Bwest in wireless nets
• Bwest in overlay routing
• Bwest in multihoming
• Bwest in bottleneck detection
• Scalable bandwidth measurements in networks with
  many monitoring nodes
• Practical issues
• Interrupt coalescence
• Traffic shapers
• Non-FIFO queues

54
Conclusions

• We know how to do the following
• Estimate e2e capacity avail-bw
• Apply bwest in several applications
• We know that we cannot do the following
• Estimate per-hop capacity with VPS techniques
• Avoid avail-bw estimation bias when traffic is
  bursty and/or path has multiple bottlenecks
• We do not yet know how to do the following
• Scalable bandwidth measurements
• Integrate probing traffic in application data
• Many research questions are still open
• Help wanted!