Choosing an Accurate Network Model using Domain Analysis

1
Choosing an Accurate Network Modelusing Domain
Analysis

• Almudena Konrad, Mills College
• Ben Y. Zhao, UC Santa Barbara
• Anthony Joseph, UC Berkeley
• The First International Workshop on Performance
  Modelling
• in Wired, Wireless, Mobile Networking and
  Computing
• (PMW2MNC, July 2005)

2
Modeling Network Measurements

• Model-driven traces as experimental tool
• Real measurements costly to perform
• Produce models by extracting statistics from
  measurements
• Use model to generate traces w/ similar
  characteristics
• New networks (wireless) difficult to model
• Traditional models require stationary statistics
• Gilbert model, Higher order DTMC, Hidden Markov
  Models (HMM)
• Artifacts disrupt stationarity of wireless traces
• Bursty error or delays due to signal interference
  or loss
• Result traditional models generate inaccurate
  results
• Our solution
• Data preconditioning preprocess traces before
  extracting model
• Result MTA, MMTA accurately model wireless traces

3
A New Challenge

• Many mathematical models are available
• Gilbert, Higher order DTMC, HMM, MTA, MMTA
• Each optimized for certain characteristics
• Each with different computational and memory
  costs
• Different networks exhibit different
  characteristics
• IP networks, wireless (802.11), sensor nets
• The challenge
• Match up network type with more accurate, lowest
  cost model
• No tools or techniques to do this
• Our solution
• Domain analysis

4
Outline

• Motivation
• Data preconditioning and its models
• The Markov-based Trace Analysis model (MTA)
• The Multiple states MTA model (MMTA)
• Quantifying modeling accuracy
• Domain Analysis
• Conclusion

5
Modeling using Data Preconditioning

• Collect network characteristic trace
• Binary traces show occurrence of an event, e.g.
  lost or delayed frame
• Identify data patterns or states (regions with
  stationary behavior)
• Precondition the data to fit traditional models
• Calculate probability distribution for each state
• Determine transition probabilities among states

Collected Trace
Subtrace 1
Subtrace 2
Subtrace 3
6
Preconditioning Model (MTA)

• MTA Markov-based Trace Analysis Algorithm
• Two states lossy error-free states
• Create two subtraces ? lossy error-free
  subtraces
• Algorithm to optimize the change-of-state
  variable, C
• For a given trace, executes stationary test for
  large range of values C
• Choose the highest C that yields a stationary
  lossy states
• Model subtraces as Exponential distributions

Lossy
Lossy
Error-free
Error-free
States
C
C
10001110011100.0 00000000 1100110000
00000..000...
Trace
...10001110011100.0
1100110000 ...
Error-free sub-trace
Lossy sub-trace
00000000
00000..000...
7
Preconditioning Model (MMTA)

• MMTA Multiple states MTA Algorithm
• Allows for multiple states in the data
• Identifies states in original trace
• Concatenates similar states to create subtraces
• Uses a DTMC to model subtraces transition
  between states

8
Network Path Traces

• A selection of end to end wireless layer path
  traces
• 0000000 10101111 00000000000
• End-to-end IP traces (IP-1)
• End-to-end WLAN delay trace (WLAN-D)
• Wireless WLAN trace (WLAN-E)
• Wireless GSM trace (GSM-E)
• Trace collected at the Radio Link Protocol (RLP)
• Traces can be decomposed into two stationary
  states
• Lossy error-free states
• Characterize traces with three parameters (Lexp,
  EFexp, Lden)
• Exponential length distributions of lossy and
  error-free states
• Lossy error density

9
Quantifying Accuracy of Models

• Previously used metrics
• FER and simulated transmission time
• No actual correlation to error distribution
• How accurate is a particular model?
• Synthetic traces comparison (Lexp, EFexp, Lden)
• Compute correlation coefficient between
  error-burst distributions

Reference trace (Lexp, EFexp, Lden)
(0.006, 0.1, 1)
Relationship between cc and accuracy CC gt 0.96
indicates an accurate model
10
Example of Model AccuracyGSM Error Burst
Distributions

• GSM, MTA MMTA experience similar error burst
  characteristics
• Gilbert, 3rd order and HMM dont reproduce large
  error burst

CC values Gilbert 0.74 HMM 0.89
MTA 0.99 MMTA 0.99
11
Choosing the Right Network ModelDomain Analysis

• Preconditioning models also work well for
  traditional networks
• There could be a bursty nature to congestion
  loss in IP networks
• Key is the presence of bursty behavior
• Need a way to choose optimal model for given
  network
• Solution domain analysis
• Domain analysis
• Create Domain Analysis Plots (DAPs) for wide
  range of parameters
• Collect empirical packet trace T 00110110100
• 1 corrupted/delayed packet, 0 normal packet
• Calculate parameters (Lexp, EFexp, Lden) for T
• Use DAPs to lookup optimal model

12
Domain Analysis Plots (DAPs)

• Generate reference traces (T1..Tx)
• Range of values (Lexp, EFexp, Lden)
• For each reference trace Ti create
• mathematical models
• artificial traces from models
• Plot error and error-free distribution
• Calculate CC between reference and artificial
  distributions
• Optimal model for Ti gt models that yield highest
  CC value

13
DAPs for (Lexp,EFexp) 0.001 -gt 0.1

• DAP 1 Lden0.2
• Gilbert region CC values (Gilbert 0.99)
• MMTA 0.98
• MMTA region CC values (MMTA 0.97)
• Gilbert 0.96

• DAP 2 Lden 0.4
• Gilbert region CC values (Gilbert 0.99)
• MTA 0.97
• MTA region CC values (MTA 0.98)
• MMTA region CC values (M30.97)

14
DAP for (Lexp,EFexp) 0.001 -gt 0.1

• DAP 3 Lden0.7
• MMTA region CC values (MMTA0.98)
• MTA0.97
• MTA region CC values (MTA0.99)
• MMTA0.99

15
Conclusions

• The challenge
• How to choose the most accurate, least cost model
  for each network type
• The solution
• Key metric for modeling accuracy by error burst
  distributions
• Domain analysis plots show optimal model for
  different networks
• Using domain analysis
• Take your own traces for network X
• Calculate the three parameter values
• Use DAP to determine optimal model for your trace

16

• Thank you
• Questions?
• akonrad_at_mills.edu