Ymer: A Statistical Model Checker

1
YmerA Statistical Model Checker

• Håkan L. S. Younes
• Carnegie Mellon University

2
Probabilistic Model Checking

• Given a model M, a state s, and a property ?,
  does ? hold in s for M ?
• Model stochastic discrete event system
• Property probabilistic temporal logic formula
• Example ?0.1? ? 5 full

3
Statistical Solution Method

• Use acceptance sampling to verify probabilistic
  properties
• Hypothesis P? ?
• Observation verify ? over a sample path
• Bounds on probability of verification error
• Probability of false negative ?
• Probability of false positive ?

4
Error Bounds
2?
Probability of errorwhen verifying P? ?
?
?
?
Actual probability of ? holding
5
Ymer at a Glance

• Supports time-homogeneous generalized semi-Markov
  processes
• Limited to time-bounded properties
• Distributed acceptance sampling (even with
  sequential acceptance sampling)
• Purely statistical approach for verifying nested
  probabilistic statements

6
DistributedAcceptance Sampling
Slave
Master
Master Acceptance Sampling
register
model property
observation
?
Slave simulation
Slave simulation
observation
?
done
7
Avoiding Sample Bias

• Process observations as they come in?
• No, bias against observations that take a long
  time to generate (long sample paths)
• Process observations according to a predetermined
  schedule

?
1
2
1
1
2
Schedule
1
1
2
Received
8
Case StudySymmetric Polling System

• Single server, n polling stations
• Stations are attended in cyclic order
• Each station can hold one message
• State space of size O(n2n)

?
?
?
?
Polling stations
9
Results
100
Machine 1 733 MHz Pentium III
Machine 2 500 MHz Pentium III
90
80
Percent of single machine
70
60
50
102
104
106
108
1010
1012
1014
Size of state space
10
Nested Probabilistic Statements Robot Grid World

• Probability is at least 0.9 that goal is reached
  within 100 seconds while periodically
  communicating
• ?0.9?0.5? ? 9 comm ? 100 goal

11
Statistical Verification ofNested Probabilistic
Statements

• Cannot verify path formula without some
  probability of error
• Probability of false negative ?'
• Probability of false positive ?'

Observation error
12
Performance Considerations

• Verification error is independent of observation
  error
• Pick observation error to minimize effort
• The same state may be visited along multiple
  sample paths
• Memoize verification results to avoid repeated
  effort

13
Robot Grid World (results)
?0.9?0.5? ? 9 comm ? 100 goal
104
103
? 0.025
? ? 10-2
102
? 0.05
Verification time (seconds)
101
100
10-1
10-2
102
104
106
108
1010
1012
Size of state space
14
Robot Grid WorldEffect of Memoization
1.0
0.9
103
0.8
0.7
0.6
Unique/visited states
Sample size
102
0.5
0.4
0.3
0.2
101
0.1
102
104
106
102
104
106
Size of state space
Size of state space
15
Availability

• Source code is released under GPL
• http//sweden.autonomy.ri.cmu.edu/ymer/