How Fast and Fat Is Your Probabilistic Model Checker

1
How Fast and Fat IsYour Probabilistic Model
Checker?

• an experimental performance comparison
• David N. Jansen3,1, Joost-Pieter Katoen1,2,
  Marcel Oldenkamp2,Mariëlle Stoelinga2, Ivan
  Zapreev1,2
• 1 MOVES Group, RWTH Aachen University
• 2 FMT Group, University of Twente, Enschede
• 3 ICIS, Radboud University, Nijmegen

2
ProbabilisticModel Checking
Probabilistic System
Probabilistic Requirement
PRISM (sparse)
PRISM (hybrid)
MRMC
VESTA
ETMCC
Probabilistic Model
Probabilistic Formula
YMER
????????
Probabilistic Model Checker
Yes
No
Probability
3
Why This Work?

• Used more often
• applications distributed systems, security,
  biology, quantum computing...
• Powerful tools
• Problem Which tool to choose?

4
ProbabilisticModel Checkers
Probabilistic System
Probabilistic Requirement
Choices made
Four examples
Probabilistic Model
Probabilistic Formula
Overall evaluation
Probabilistic Model Checker
Yes
No
Probability
5
Tools
6
Experiment Relevance

• Repeatable
• Verifiable
• Significant
• Encapsulated

7
Selected Benchmarks
8
SynchronousLeader Election

• nodes in a ring elect a leader
• each node selects random number as id
• passes it around the ring (synchronously)
• if ? unique id,node with maximum unique id is
  leader
• Itai Rodeh, 1990

9
SynchronousLeader Election
10
SynchronousLeader Election
1
4
2
2
5
3
1
5
11
RandomizedDining Philosophers

• Dining Philosophers
• pick up chopsticks in random order
• Deadlocks resolved
• if there is no second chopstick,give up eating
• Pnueli Zuck, 1986

12
BirthDeath Process

• Models a waiting queue
• Standard modelin performance evaluation
• Limit queue size to get finite model

13
Tandem Queueing Network

• Two queues after each other
• Hermanns, Meyer-Kayser Siegle, 1999

checkin counter two-phase
security check exponential
14
Cyclic Polling System

• server cycles over n stationsand serves each one
  in turn
• e.g. teacher walks through class,each pupil may
  ask a question
• Ibe Trivedi, 1990

15
Modelling
informal description
PRISM model
VESTA model
adapt syntax
.tra format model
YMER model
PRISM
ETMCC
MRMC
YMER
VESTA
16
Experiment 1Qualitative Properties

• unbounded reachability with prob 1
• Cyclic Polling System busy1 ? P1(true U
  poll1)If station 1 is busy,the server will
  poll it eventually

17
(No Transcript)
18
PRISM MTBDD Size

• Multi-Terminal BDD data structure for
  transition matrix
• size heavily depends on model
• large MTBDD ? slow

19
CPS versus SLE runtime
458.847 states 1.131.806 MTBDD
nodes
7.077.888 states 2.745 MTBDD nodes
20
VESTAsimulation problem

• actual probability close to bound Pp(...)
• estimate is almost always in p?,p?
• some irregularity stops the simulation
• 0.95 ? Prob(yes ? actual Probp) ? Prob(actual
  Probp ? yes)

21
P1(... U ...)Timing Overview
22
Analysis
23
Result Overview Timing
depends heavily on MTBDD size
depends heavily on MTBDD size
depends heavily on MTBDD size
24
Result Overview Memory
MTBDD size varies heavily
almost independent from model size
25
Experiment 2Bounded Reachability

• Tandem Queueing Network Plt0.01(true U 2 full
  )Is the probabilitythat the system gets full
  in 2 time unitssmall?

26
(No Transcript)
27
Analysis
28
Result Overview Timing
29
Result Overview Memory
30
Experiment 3Steady State Property

• Tandem Queuing Network Sgt0.2( Pgt0.1(X 2nd queue
  full ) )In equilibrium,the probability to
  satisfy is gt 0.2

Pgt0.1(X 2nd queue full )
Pgt0.1(X ... )
31
(No Transcript)
32
Simulating Steady State?

• simulation of bounded reachabilityhas clear
  stopping criterion
• simulation of unbounded reachability?
  reachability with very large bound
• simulation of steady state?? never stops

33
Analysis
34
Result Overview Timing
35
Result Overview Memory
36
Nested Formulas

• BirthDeath Process P0.8(P0.9(true U 100
  n70) U n50)The probability to reach n50 (while
  the probability to reach n70 in 100 steps
  never drops lt0.9)is 0.8

37
(No Transcript)
38
Result Overview Timing
did not terminate
39
Result Overview Memory
40
Conclusions