Simulationbased STRESS Testing Case Study: A Multicast Routing Protocol Systematic Testing of Robust

1
Simulation-based STRESS Testing Case Study A
Multicast Routing ProtocolSystematic Testing of
Robustness by Examination of Selected Scenarios

• Ahmed Helmy, Deborah Estrin
• Computer Science Department/ISI
• University of Southern California
• ahelmy,estrin_at_usc.edu,
• http//catarina.usc.edu/ahelmy,estrin

2
Outline

• Motivation / Problem Statement
• Related Work
• The STRESS framework
• Problem dimensions solution for PIM-SM
• host events, topology, faults
• Simulation framework
• Case study results
• Summary Future Work

3
Motivation and Problem

• Approach
• Develop systematic methodologies for testing
  Internet protocol robustness
• Programmatically simulate failure modes to
  analyze behavior

4
Related Work
Tend to abstract out network failures Complexity
increases with multicast
5
STRESS Framework
6
STRESS block diagram
Loss Failures
Link Tracing
Routed Topology
Simulation set-up
Simulation Engine
Host Scenarios
Simulation Tracing
Scenario Generation
Output Analysis
7
Problem Dimensions

• A Test Scenario may include
• Routed Topology
• Host Events
• Faults (selective message loss)

Faults
Topology
Host Events
8
Representative Host Scenarios

• Host Events simple scenario reduction filter
• Send (S), Join (J1,J2), Leave (L1,L2)

9
Representative Topologies

• Remove topologies that experience same errors and
  transitions
• Transitions are dependent on protocol-specific
  mechanisms
• Need protocol-specific analysis to get the
  representative topologies

10
Unicast vs. Multicast
S
S
R1
R1
R2
R2
R3
R3
R4
R4
Multiple unicasts
Multicast
11
PIM-SM Overview
S
R1
Root
R2
R3
R2
R1
R4
12
Topological Equivalence

• N-router LAN topology, running PIM-SM

13
Equivalent Topologies
S1
R1
R2
R2
S2,R1
Topology 2
Topology 1
14
Simulation

• Used Network Simulator (NSv2) and the VINT system
  (http//catarina.usc.edu/vint)
• Added LANs, selective loss, protocol tracing and
  profiling support
• Implemented PIM-SM shared tree mechanisms
  (http//catarina.usc.edu/pim)
• Join/Prune, Assert, Register

15
Simulation Framework
16
Results Faulty Join Scenario
C
A
B
R1
R2
17
Results Fault Register Scenario
C
A
S2, R1
18
Results Coverage
Number of Scenarios with Selective Loss
Total 352 (1st topo) 296 (2nd topo) 648
Scenarios
Protocol Procedure Coverage
of 91 traced procedures
19
Summary

• Presented STRESS framework for systematic testing
  of protocol robustness
• Used representative scenarios and equivalent
  topologies to reduce space searched
• Automated fault investigation through simulation
• Captured several design errors in PIM-SM and
  suggested fixes

20
Future Work

• Automate Scenario and Topology Generation using
  search techniques
• Forward Search
• Fault-oriented backward search
• Achieve better completeness and coverage
• Extend the framework for performance analysis of
  end-to-end multicast protocols

21
Thank You!
Merci
Ahmed Helmy ahelmy_at_usc.edu http//catarina.usc.edu
/ahelmy