Mobile Network Estimation

1
Mobile Network Estimation

• Minkyong Kim, Brian NobleMobile Software
  SystemsUniversity of Michigan

2
Adaptive distributed systems

• Many systems adapt to changes in network capacity
• media-rich applications web browsers, video
  players,
• performance enhancement caching, prefetching,
• distributed systems query planning, agent
  migration,
• All of these systems follow the same general form
• observe network traffic at one or both endpoints
• estimate the latency, bandwidth, loss rate,
• react if anything changes in an interesting way
• All of this depends on estimating network
  capacity well
• turns out to be a difficult problem

3
Networks have variable performance

• Sources of variation in mobile, wireless networks
• nodes move, leading to unpredictable topology
  changes
• often more than one connection alternative
• physical layer subject to fading, shadowing,
  multi-path
• Sources of variation in wide-area networks
• bursty congestion over all time scales
• routing changes between autonomous systems (BGP)
• Typically, adaptive systems are evaluated very
  carefully
• with respect to clean, idealized network changes
• my own work in Odyssey is guilty as charged

4
Goals of a good estimator

• Estimate metrics that matter to the system
• many network estimators focus on physical
  capacities
• link capacity is like a speed limit
• try driving the speed limit in LA during rush
  hour
• instead measure available capacities
• Provide three characteristics
• accuracy gives correct estimates in steady state
• agility detect a true shift in capacity rapidly
• stability ignore short-lived transient changes

5
Current estimators EWMA filters

• Most use exponentially weighted moving average
  filters
• at each time step, incorporate new observation
  (Ocurrent)
• with old estimate (Eold)
• using a weighted linear combination
• Ecurrent a(Eold) (1-a)Ocurrent
• The term a is called the gain
• large gain biases toward stability
• small gain biases toward agility
• gain is set statically
• You cant have your cake and eat it too

6
A tale of two estimators

• TCP a stable filter that is too stable
• estimates round trip time (RTT) segment, ACK
• stable estimator gain set to 7/8
• used to set retransmission timeout (RTO)
• under rapidly escalating congestion, RTO grows
  too slowly
• RTO adds fudge factor based on variance
• Odyssey an agile filter that is too agile
• estimates latency and bandwidth for bulk
  transfers
• applications react to change by changing fidelity
• agile filter gain set to 1/4 (latency) and 1/8
  (bandwidth)
• transient changes leads to tail-chasing
  adaptations
• applications must add hysteresis to dampen
  transients

7
The rest of this talk

• Introduce a simple fluid flow network model
• used to derive spot observations that are fed to
  filters
• Describe three filters that adapt to prevailing
  conditions
• error-based vary gain based on quality of
  estimate
• stability-based vary gain based on observed
  noise
• flip-flop use a control to select an agile or
  stable filter
• Evaluate the quality of these filters
• subject each to a variety of networking
  conditions
• compare agility and stability to TCP, Odyssey
  filters

8
A fluid-flow network model

• Our model is based on the packet-pair technique
• model network path as single, bottleneck link
• send two packets back to back from source to sink
• sink ACKs both packets as they are received
• spread between ACKs measures bandwidth along path
• We need both bandwidth and latency
• take two observations to solve for two unknowns
• Several subtle points
• depend only on passive traffic observations
• spot observations filter out self interference
• assumes symmetric network performance

9
The error-based filter

• Problem with EWMA filters comes from static gain
• Instead, vary gain based on predictive quality of
  estimates
• each estimate forms a prediction for next
  observation
• at each observation, compare prediction with
  actual value
• Scale gain with the accuracy of prediction
• predictions that are accurate deserve higher
  weight
• if inaccurate, should converge on observation
  quickly
• Tends to ignore small changes, follow large
  changes

10
Error-based filter in action
11
The stability-based filter

• The error-based filter will be pulled by large
  transients
• will tend towards instability during transient
  dips
• Instead, base gain on stability in recent
  observations
• moving range difference between adjacent
  observations
• noisy observations lead to larger moving ranges
• Scale gain with the magnitude of the moving range
• when observations are noisy, each deserves less
  weight
• when observations are stable, changes more
  significant
• Tends to ignore large changes, follow small ones

12
Stability-based filter in action
13
Subtleties in variable-gain filters

• The gain in each is based on some source metric
• Gain must be in the range 0..1
• need some way of scaling the source metric
• determine the maximum error, instability
  recently seen
• scale current error, instability relative to
  maximum
• Transient changes in source metric have drastic
  effects
• smooth observed source metrics by secondary
  filter
• secondary filter has static gain (!)
• rather than provide tertiary filter, tune
  empirically
• Sometimes, variable-gain filters are neither
  agile nor stable
• source metric places them somewhere in the middle

14
A short detour statistical process control

• Suppose you had a machine that built widgets
• widgets specified to have some size, error
  tolerance
• How do you know your machine is building good
  widgets?
• idea periodically grab k widgets, measure them
• if average size is about what you expect, things
  are OK
• if not, machine is probably out of control
• Formalizing this idea the control chart
• population mean, m
• sample standard deviation, s
• control lines m3s, m-3s
• the 3s rule stay inside the lines

15
The flip-flop filter

• Use a control chart to select for agility,
  stability
• run two static-gain EWMA filters in parallel
• maintain a control chart for each observation
• if within control limits, use agile filter (a
  0.1)
• otherwise, use stable filter (a 0.9)
• Cannot apply simple control chart directly to
  this problem
• true mean is not known, and it changes over time
• sample standard deviation is not known
• Use approximations (individual x-chart)
• m follows simple smoothed estimate of
  observations
• s approximated with 2-element moving range

16
Flip-flop filter in action
17
Evaluating candidate filters

• Can these filters be as agile as the Odyssey
  filter
• in recognizing a true change in link bandwidth?
• in reacting to the presence of cross traffic?
• in detecting a change in ad hoc topology?
• in detecting a wide-area route change?
• Can these filters be as stable as the TCP filter
• in resisting a transient change in link
  bandwidth?
• in tolerating the presence of cross traffic?
• in tolerating retransmissions in ad hoc networks?
• in tolerating noise across a real wide-area
  network?
• Can they predict in an ad hoc network with cross
  traffic?

18
Experimental methodology

• All experiments in this talk used ns, a network
  simulator
• the wide-area set are based on live network
  traces
• Extensions to support variable-link experiments
• script controls base physical performance of a
  link
• can vary latency, bandwidth over time
• Ad hoc networking simulations include Monarch
  extensions
• collision-avoidance
• link-level ACK, retransmission
• In each experiment, filters converge to same
  value
• they do not differ in accuracy
• only differences in agility, stability

19
Link changes

• First set of experiments impulse-response tests
• connect client, server with a single ns link
• vary link performance with a variant of a square
  wave
• persistent change decrease from 10Mb/s to 1Mb/s
• transient change dip from 10Mb/s to 1Mb/s and
  back
• Vary number of request/response pairs exposed to
  change
• poisson request generator, random response size
• Agility measured by settle time
• time to reach an estimate within 10 of nominal
• Stability measured by mean squared error
• penalizes large, short disturbances more than
  small, long

20
Agility for step-down waveform
21
Stability for impulse-down waveform
22
Cross traffic experiments

• Start request/response traffic between client and
  server
• at 50 seconds, inject 5Mb/s cross traffic
• All filters slightly optimistic in estimates
• not all packets see full queue delays
• Agility settle time
• Stability
• coefficient
• of variance

23
Cross traffic results agility
24
Cross traffic results stability
25
Simple ad hoc topology changes

• Place three server/router nodes in a line
• single client walks from server to end of line,
  and back
• topology changes at each stage
• Agility results do not add much new information
• similar to congestion TCP is bad, rest are
  comparable
• Stability results are useful
• coefficient of variation
• after settle time

26
Stability results topology changes
27
Summary of comparisons
28
Acid test predicting ad hoc performance

• Typical ad hoc simulation
• 50 nodes in 1500x500 meter space
• initial locations randomly distributed throughout
  space
• nodes move in random waypoint model
• Nodes are formed into 25 pairs
• one pair is our test client/server poisson
  traffic
• remaining 24 pairs exchange CBR traffic
• vary rate of congestion traffic across
  experiments
• No filter does particularly well
• two static filters are worst performers
• flip-flop is best of the bunch

29
Ad hoc accuracy results
30
Related Work

• S. Keshav introduced packet-pair, bottleneck
  bandwidth
• fuzzy estimator similar to error-based estimator
• analysis for rate-allocating servers (not FCFS)
• Packet-pair extensions
• Paxson receiver-based packet pair time at both
  ends
• Lai receiver-only packet pair time at receiver
• Active probing Bolot, Downey, Carter Crovella,
  
• measurement load can be substantial
• Lais general network model, packet tailgating
  technique
• Balakrishnans congestion manager unified RTT
  observations
• can benefit from our filters for better estimates

31
Conclusions

• Adaptive systems depend on quality of measurement
• particularly hard to estimate network capacity
• Standard filtering techniques agile or stable,
  but not both
• Adaptive filters tune for prevailing network
  conditions
• agile when possible, stable when necessary
• Best alternative flip-flop filter
• composition of two static-gain EWMA filters
• statistical process control used to select
  between them
• comparable to Odysseys agile filter in 4/5
  scenarios
• comparable to TCPs stable filter in 3/4
  scenarios
• provides best predictions in complex ad hoc
  network

32
Questions?

• Further details http//mobility.eecs.umich.edu/
• Preprint of the paper is available