VINT

1
VINT STRESSStatus and DirectionsOctober 26,
1998

• VINT
• Deborah Estrin and John Heidemann, USC/ISI
• Joint work with ISI (Haldar, Huang, Pryadkin,
  Yu, Xu)
• LBL (Floyd), Xerox PARC (Shenker,
  Breslau)UCBerkeley (Fall, McCanne)
• STRESS
• Deborah Estrin, Sandeep Gupta, Ahmed Helmy USC/ISI

2
Agenda

• VINT
• simulation challenges
• recent progress
• plans
• STRESS
• systematizing protocol evaluation

3
VINT Agenda

• Simulation challenges
• design exploration
• validating and quantifying results
• expanding applicability
• Recent progress
• Plans

4
Design Exploration

• Design need simple models
• Implement need clean interfaces
• Debug need visualization and measurement tools
• Compare options

VINT simulation speeds design cycle STRESS
systematizes
5
Validating and Quantifying Results

• Initial comparisons can be made in model space
• Detailed comparisons must be tied to real world
• VINT simulation helps by
• leveraging existing validated components
• providing controlled experimental environment
  with parameters set to values measured in real
  systems

6
Expanding Applicability

• Validated models can be scaled up
• more nodes and protocol agents
• larger and more varied topologies
• VINT simulation provides
• methodology for validating scaling
• virtual world which is easier to scale and vary
• (cant afford 1000 PCs or create 1000 wireless
  hops)
• wide range of topologies
• standard libraries and automatic generation

7
VINT Agenda

• Simulation challenges
• Recent progress
• design exploration
• validating and quantifying results
• expanding applicability
• Plans

8
Design Exploration

• evolving protocol support
• improved multicast support (components, LAN
  support)
• application support (new APIs, web cache)
• current integrating mobility/wireless support
  from CMU, Sun
• emulation
• real and simulated packets interact
• improved visualization tools

9
Improved Visualization

• Namgraph application-specific event graphs
• tcp time/sequence number
• srm time/event
• application customizable
• synchronized with packet view

10
Validating and Quantifying Results

• Comparisons of simulations with real world
• typically with protocol studies
• ex self-similarity in large TCP traffic,
  reliable multicast performance, etc.
• Ns test and validation suites
• TCP, multicast, queueing, intserve, routing

11
Real and Simulated TCP Flows
(simulated)

• Demonstrates

Reproduced multi-fractal traffic (from traces) in
ns simulations. (Feldmann, Gilbert, Willinger,
Huang)
12
Expanding Applicability

• Generic scale-up methodology
• Improved abstraction techniques
• hierarchical routing supports 100s-1000 node
  detailed simulations
• algorithm routing (generalized from Raman et al)
  can scale to 50k nodes on PC hardware (300MB
  mem)
• documented applicability (error understanding)

13
Scale-up Methodology
Validate at small scale
Revalidate at large scale (if possible)
Scale up in simulation
14
Example SRM
Memory Usage
Recovery Delay
30
50
25
40
20
30
MB
detailed
15
Delay in RTT
20
10
5
10
session
0
0
0
50
100
20
40
60
80

• Foo

Group Size
Group Size
Session simulation saves substantial memory
without changing results.
15
Improved Abstraction Techniques
16
Summary of Current Status

• Active development on VINT
• ...and elsewhere
• 230 institutions actively using ns/nam
• gt200 people at 3 workshops
• substantial code contributions (Daedalus,
  Monarch, Sun, Marc Greis)

17
VINT Agenda

• Simulation challenges
• Recent progress
• Plans
• design exploration
• validating and quantifying results
• expanding applicability

18
Immediate Plans

• Design exploration
• support and use mobility/wireless
• integrate Sun/CMU code
• new protocol studies
• gain experience with application-level
  simulations
• improve nam customizability
• gain emulation experience

19
Immediate Plans (cont).

• Validation and quantification
• additional, large scale validation
• expand small scale tests and validation
• Expanding applicability
• mixed-mode simulations multiple concurrent
  levels of abstraction
• validate abstraction for additional applications

20
Agenda

• VINT
• simulation challenges
• recent progress
• plans
• STRESS
• systematizing simulation

21
Systematic Testing of Protocol Robustness by
Evaluation of Synthesized Scenarios (STRESS)

• http//catarina.usc.edu/stress
• Deborah Estrin, Sandeep Gupta, Ahmed Helmy
• University of Southern California

22
Motivation Objective

• Increased complexity of protocol testing due to
  new and more frequent faults
• new fault modes with growth in number of systems
  and protocols
• increased heterogeneity of interconnect
  technologies and media
• Qualitative changes in the nature of the
  protocols due to introduction of new services,
  such as multicast
• Decrease time to deployment

23
Concepts

• Falsification vs. verification
• Falsification detect design error(s)
• Verification prove absence of errors (higher
  complexity, needs more powerful techniques)
• Robustness vs. correctness
• Robustness is correctness in presence of faults
  as defined in protocol specification
• Correctness vs. performance
• Performance includes excitation of correct
  scenarios but with worst case behavior

24
STRESS Framework
Testing
25
STRESS Framework
Much of the work will focus on Test Generation
26
Problem Dimensions

• A Test Scenario may include
• Topology (LAN, regular, random)
• Host Events (Joining, Leaving)
• Faults (message loss, crashes, failures)

Faults
Topology
Host Events
27
Approach Spectrum
Complete/ fully automatic
systematic
random
Forward search
Scenario Generator
Backward search w/topology synthesis
Simulation- based STRESS
28
Related Work

• Automata theory
• FSM Global FSM formalism
• Reachability Analysis
• Search techniques, symmetry/equivalence for
  complexity reduction
• VLSI chip testing
• fault oriented TG, implication and search
• Topology generation/graph theory

29
Modeling

• Model of the Protocol
• Simulation
• FSM model
• Model of the Topology
• For simulation use topology generators or
  heuristics
• For search techniques use global FSM for LANs

30
Modeling Global FSM Model

• FSM ltStates, Stimuli, Transitionsgt

F,NF,R,NR Join,Prune,Leave

• Global FSM
• Global State, e.g. F1,R2,NR3,R4

31
Modeling Transition Table

• Regular I/O FSM

32
Modeling Transition Table (Contd.)

• Add subscripts to denote different routers
• Add semantics to denote multicast, unicast, and
  broadcast messages
• Derive pre/post-condition table automatically

Joini
Prunej.Ri
Prunej
Leavej.NRj
33
Modeling Errors Faults

• Error Modeling
• an error is failure of the protocol to meet its
  correctness requirement
• correctness requirement
• functional e.g., no duplicated or lost data
  packets
• w.r.t. states e.g., if R exists then exactly one
  F must exist
• Fault Modeling
• a fault is a physical failure (e.g., link/node
  failure, packet loss), or error in a lower layer
  (e.g. unicast route loops)

34
Scenario Generator (ScenGen)

• Topology Generator
• flat, transit-stub, N-level hierarchy
• Agent (sender/receiver) generator
• spatial distribution e.g., clustering,
  sparseness
• temporal distribution traffic models, membership
  dynamics, connection start/end
• Fault Generator
• loss, crashes, link failures, congestion models
• Protocol Independent
• difficult to achieve completeness

35
Simulation-based STRESS

• Topology
• derive protocol specific equivalent topologies
  (LAN-based with simple extensions)
• Host events
• use heuristics to reduce explored host scenarios
• Fault
• automate faults through simulation (e.g.
  selective loss)
• Not fully automated, more complete than ScenGen
• For PIM-SM detected register loops, Join, Prune,
  and Assert blackholes

36
Work In Progress

• Automate Scenario and Topology Generation using
  search techniques
• Forward Search
• Fault-oriented backward search
• Achieve better completeness and coverage

Error
Error
37
Forward Search-based Technique

• Start from initial states and expand the space
• Reduce complexity using equivalence classes
• exploiting symmetry, e.g.
• F1,R2,NR3,R4 R1,NR2,R3,F4 F1,R2,NR1
• Case study for PIM-DM variant
• achieved reduction from O(4n) to O(n4), where n
  is the number of routers
• detected Join/Prune blackholes, Assert bandwidth
  overhead

38
Forward Search

• Challenges
• dealing with asymmetry
• defining equivalence classes
• Limitation
• topology is an input

39
Fault-oriented Test Generation (FOTG)

• Start from the fault (e.g. the message to be
  lost)
• Construct a global state needed to trigger the
  message
• Use forward search to check for errors in
  presence of message loss
• Search backward from the error to obtain a
  sequence from an initial state

40
Transition Table
Joini
Prunej.Ri
(Fk
Prunej
NFk). Ri.Joini
Leavej.NRj
41
Test Generation

• Loss of a Join message

Joini
Prunej.Ri
Prunej
Leavej.NRj
Leavej
Host Event
Constructed Topology
No loss

NFk
GI1NRj,Ri,Fk
NRj,
Ri,
GI
loss
Prunej
GI1NRj,Ri,NFk
GI-1NRj,Ri,Fk
Initial State
Error state
.
.
.
time
42
FOTG Results

• Case study for PIM-DM variant
• synthesized topologies for all given mechanisms
• detected Join/Prune blackholes, Assert overhead,
  Graft blackhole
• more powerful reasoning about interleavings
• Limitations
• no guarantee to inspect only reachable states

43
Summary of Initial Results

• Developed and simulated several test generation
  (TG) algorithms
• Scenario Generator Simulation-based STRESS
• Fault-independent TG (reduced forward search)
• Fault-oriented TG (forward/backward w/topology
  synthesis)
• Studied variants of PIM-SM PIM-DM
• Detected and fixed 8 correctness violations
• Mechanisms Register, Join/Prune, Assert, Graft
• Violations loops, duplicates, black holes, extra
  overhead

44
Key Contributions

• Integration of automatic TG and robustness
  analysis into the protocol design
• Integration of fault modeling into FSM model
• Extension of GFSM to capture multicast semantics
• Topology synthesis using FOTG
• Case studies
• multicast routing (PIM-DM/SM)
• end-to-end multicast protocols e.g. SRM (in
  progress)

45
Extensions for Search Techniques

• Forward Search
• Add topology synthesis
• Systematize equivalence classes detection

• FOTG
• More powerful implication techniques
• Add reachability detection
• Use heuristics/symmetry to increase efficiency

46
Extensions for Search Techniques

• Remove false alarms those that do not satisfy
  correctness, but do not affect packet delivery,
  e.g. they recover with next packet
• Use composite/symbolic state representation to
  aggregate topology information, e.g. repetition
  constructs F2 gt two or more forwarders
• Evaluate quantitatively completeness,
  complexity, and test quality for the algorithm
  w.r.t. the protocol under study

47
Extensions New Domains

• End-to-end vs. network layer
• rich timing/delay semantics (protocol timers, and
  network delays)
• variables (e.g. sequence numbers, ttl, unicast
  metric)
• emphasis on performance/worst case behavior
• more explicit topology representation
• redefinition of symmetry/equivalence

48
Extensions Problems revisited

• Terrestrial vs. Wireless
• wireless, ad hoc, satellite, mobile
• channel characteristics loss, asymmetry,
  uni-directional
• topology dynamics
• MicroNets
• Consider development of similar methodology to
  aid design of micronets

49
Directions Future Contributions

• Ripple effect studies effect of errors in
  network protocols on end-to-end applications
• Sensitivity analysis performance change due to
  perturbation in network or protocol parameters
• Diagnosis library build library of behavior
  patterns to aid in network diagnosis