Verisim: Formal Analysis of Network Simulations

1
Verisim Formal Analysis of Network Simulations

• Karthikeyan Bhargavan, Carl A. Gunter, Moonjoo
  Kim, Insup Lee, Davor Obradovic, Oleg Sokolsky,
  Mahesh Viswanathan
• University of Pennsylvania

2
Internetwork Routing Protocols

• Internetwork routing protocols enable
  interoperability between physical networks.
• Routing protocols for the Internet RIP, OSPF,
  and BGP.
• Routing protocols for packet radio AODV and DSR.
• Routing protocols and software have growing
  importance and complexity.

3
Current Analysis Techniques

• Rigorous mathematical models and proofs limited
  by complexity.
• Testing.
• Testbed expensive.
• Operational risky and inconvenient.
• Simulation.
• Performance attributes throughput, latency,
  reliability, etc.

4
Limitations of Performance Analysis of Simulations

• Flaws may not be detected if no expected
  performance can be used for comparison.
• When flaws are suspected, finer means of analysis
  are useful.
• Some flaws do not manifest themselves as
  performance problems (e.g. most security gaps).

5
Verisim

• Verisim provides support for logical analysis
  of network simulations.
• This talk describes its architecture and logic.
• We provide a series of experiments aimed at
  assessing the approach.

6
Monitoring and Checking (MaC) Framework
NS
MEDL
7
Meta Event Description Language (MEDL)

• Expresses properties of traces.
• Extension of Linear Temporal Logic (LTL) with
  auxiliary variables.
• More expressive than LTL.
• Properties classified into (instantaneous) events
  and (enduring) conditions.

S Kannan, M Kim, I Lee, O Sokolsky, M Viswanathan
98
8
NS Network Simulator
Instrumented Protocol Code P C
Protocol Agents
P
P
P
Scenario
Configuration Parameters OTcl
Network Model
N
N
N
Topology OTcl
Traffic Agents
src/sink
src/sink
src/sink
Traffic Model OTcl
NS Trace
VINT Project
9
Verisim
Properties MEDL
Instrumented Protocol C
Checker
NS
Trace
Metatrace
Scenario OTcl
10
Ad Hoc Networks

• Routing for a wireless internetwork without the
  aid of a central base station.
• Connections are low-bandwidth, lossy, and highly
  transient.
• Unique routing assumptions
• Most routes are seldom used.
• Bandwidth must be protected.

11
Illustration Part 1 of 2
Movement
Routing
12
Illustration Part 2 of 2
New Routing
13
AODV Protocol

• If a node S needs a route to a destination D and
  does not have one, it floods a route-request
  (RREQ) packet through the network.
• Each recipient R of this RREQ keeps a return
  pointer.
• R broadcasts the request to its neighbors if it
  is not D and does not have a route to D.
• If R is D, or has a route to D, it responds with
  a route-reply (RREP) packet using the return
  pointers for S.

Perkins and Royer 99
14
Loop Freedom in AODV

• Routing loops are undesirable.
• AODV uses sequence numbers to indicate freshness
  of link information.
• Key Invariant If next(n) n, then
• seqno(n) ? seqno(n), and
• if seqno(n) seqno(n), then hops(n) gt hops(n).
• The invariant ensures that there are no loops.

15
Outline of Experiment

• Run a scenario of modest complexity.
• Analyze it in Verisim using a list of 9
  properties of AODV expressed in MEDL.
• First pass Repair First Bug (RFB).
• Second pass tune the MEDL formulas to avoid
  rerunning the simulation discovering bugs in the
  metatrace.
• We instrumented simulation code for AODVv0
  supplied by the CMU Monarch Project.

16
Experiment Scenario
1
2
3
4
5
17
Experiment Scenario
1
2
3
4
5
18
Experiment Scenario
1
2
3
4
5
19
Experiment Scenario
1
2
3
5
4
20
Experiment Scenario
1
5
2
3
4
21
Sample MEDL Alarm
alarm LoopInvatnxtdst sendroutatdst
when ((at?nxt) ? (at?dst) ? (nxt?dst) ?
(obs_nexthopatdst nxt) ?
((obs_seqnoatdst gt obs_seqnonxtdst) ?
((obs_seqnoatdst obs_seqnonxtdst)
? (obs_hopcontatdst lt
obs_hopcntatdst))))
This is the negation of the fundamental invariant
ensuring no loops in AODV If the next hop from
node at toward destination dst is node nxt then
the sequence number (for dst) of at is less than
or equal to that of nxt or they are equal and the
hop count (for dest) of at is less than or equal
to that of nxt.
22
Repair First Bug Experiment

• Let ? be the MEDL property set.
• Run the simulation to get a trace T.
• Run the checker to get a metatrace T?.
• Repair the first bug in the metatrace to get new
  protocol code.
• Rerun the simulation with the new protocol code
  to get a new trace U.
• Rerun the checker to get a new metatrace U?.
• Continue until an empty metatrace is obtained.

23
RFB Experiment Statistics
Meta- trace Dest Rep Detect RErr Node Rep Loop Env Total Alarms
T? 4 54 38 43 220
U? 0 54 38 43 216
V? 0 48 39 44 206
W? 0 0 0 0 1
24
Tuning Experiment Statistics
Meta- trace Dest Rep Detect RErr Node Rep Loop Env Total Alarms
T? 4 54 38 43 220
T? 0 54 38 43 216
T? 0 0 38 50 166
T? 0 0 21 0 21
No new simulation traces.
25
Analysis of Off-The-Shelf (OTS) Simulations

• The prior experiment involved fewer than 10,000
  events, and we designed it to exercise key
  scenarios.
• Can we do useful analysis with OTS performance
  simulations?
• Aim Verisim can be added with small
  modifications, run alongside OTS simulations, and
  find logical failures in a practical period of
  time.

26
Experiment

• Took largest available OTS simulation for AODV
  and ran it to create a trace.
• Simulation from Monarch uses 50 mobile nodes on
  1500x300m grid moving 20 m/s.
• 5220 seconds (1.5 hours) to complete simulation.
• 6,446,316 events.
• Naïve effort ran MEDL with MonSeqNo test on all
  nodes (2500 relations) using 550Mhz dual
  processor machine with 1GB of memory.
• Aborted the experiment after 4 days estimate 100
  days to complete this analysis.

27
Optimizations For Network Routing Simulations

• Population Abstraction test the property for a
  subset of the routers.
• Packet Type Abstraction prune the trace to
  include only relevant events.

28
Sample Experiment Results for MonSeqNo

• Population Abstraction for 5 nodes.
• Trace size 6,446,316 events
• Property size 14,543 bytes
• Time 51,054 seconds
• Rate 0.54 micro seconds per event per property.
• Population and Packet Type Abstractions.
• Trace size 6812 events
• Property size 14,543 bytes
• Time 51 seconds (found failures)
• Rate 0.51 micro seconds per event per property.

29
Sample Experiment Results for LoopInv

• Population Abstraction for 5 nodes.
• Trace size 69,411 events
• Property size 75,508 bytes
• Time 8064 seconds
• Rate 1.54 micro seconds per event per property.
• Population and Packet Type Abstractions.
• Trace size 48,735 events
• Property size 75,508 bytes
• Time 5912 seconds (found failures)
• Rate 1.61 micro seconds per event per property.

30
Related Work

• Communication-based test generation systems.
  STRESS and Verisoft.
• Test oracles based on formulas or formal
  operational specifications. GIL and TETRA.
• Formal instance verification of routing
  protocols. SPIN/PITHIA for PNNI.
• Simulations based on logical specifications.
  MTSim and Maude.
• Network monitoring systems. Emerald.

31
Summary

• Verisim integrates simulation and logical trace
  analysis.
• The combination provides a more flexible approach
  to analyzing network simulations for safety
  properties.
• It is able to find bugs in existing simulator
  studies.
• Its flexibility can be exploited to improve
  debugging turn-around time.