Statistical ModelChecking of BlackBox Systems VESTA

1
Statistical Model-Checking of Black-Box
SystemsVESTA

• Koushik Sen
• Mahesh Viswanathan
• Gul Agha
• University of Illinois Urbana-Champaign

2
Probabilistic Models (continuous-time)

• Continuous Time Markov Chains (CTMC)
• Exponential distribution
• Generalized Semi-Markov Processes (GSMPs)
• General distributions

3
CSL sub-logic

• ? true a ? Æ ? ? P_at_ p(?)
• ? ? Ultt ? X ?
• where _at_ 2 lt,gt,,
• Plt 0.5(lt10 full)
• Probability that queue becomes full in 10 units
  of time is less than 0.5
• Pgt0.98( retransmit Ult200 receive)
• Probability that a message is received
  successfully within 200 time units without any
  need for retransmission is greater than 0.98

4
Statistical Approaches
Monte-Carlo Simulator
Property
Property
5
Statistical Model Checking

• Given a model M, a set of samples S (generated
  from M) and a property ?
• A(S, s0,?)
• A(S, s0,?) yes with p-value ?
• ) PrA(S, s0,?) yes M,s0 2 ?
• A(S, s0,?) no with p-value ?
• ) PrA(S, s0,?) no M,s0 ² ?
• A(S, s0,?) dont know

yes with p-value ? no with p-value ? dont
know
6
Model-Checking Overview

• Check satisfaction of a formula
• Check satisfaction of its sub-formula
• Use the result to check satisfaction of the
  formula
• ?1 Æ ?2 is satisfied at s iff
• ?1 is satisfied at s
• ?2 is satisfied at s
• ?1 Ultt?2 is satisfied on a path s1s2 iff
• At si, ?2 is satisfied
• At sj (for all i ltj), ?1 is satisfied
• time(si) time(s1) lt t
• Pltp? is satisfied at s iff
• probability that a path from s satisfies ? is
  less than p

Easy
Easy
How??
7
Checking Plt0.6(p Ult12 q) statistically at s
Sample contains, say, 30 paths from s

• On 21 paths (p Ult12 q) is satisfied
• 21/30 gt 0.6
• can we say that Plt0.6(p Ult12 q) is violated at s
  ??
• Statistically, yes, provided we quantify the
  confidence in our decision
• p-value ?
• PrOn 21 (or more) out of 30 paths (p Ult12 q)
  hold probability that (p Ult12 q) holds on
  a path is less than 0.6
• PrX 21 where X is Binomial(30,0.6)

.
p Ult12 q
8
p-value

• Let r ( of paths on which (p Ult12 q) hold /
  of total paths)
• Let p Pr(p Ult12 q) holds on a path
• no answer (formula does not hold)
• yes answer (formula holds)

9
Nested Checking Plt0.6(?1Ult12?2) at s

• ?1 and ?2 contain nested probabilistic operators
• Checking (?1 Ult12 ?2) over a path
• Answers are not simply yes or no
• Answers can be
• yes with p-value ?
• no with p-value ?
• dont know
• Need a modified decision procedure
• Handle dont know to get useful answers
• Incorporate p-value of decision for sub-formulas

10
Checking Plt0.6(?1Ult12?2) at s (Problem)

• Solution
• Resolve dont know (?) in adversial fashion
• Observation region
• Create uncertainty region to incorporate
  p-value associated with each path.

.
?
?
?1
?3
?2
?1 Ult12 ?2
11
To check Plt0.6(?1Ult12?2) at s

• Need to check if of yes paths by of total
  paths lt 0.6
• Let, of yes paths20, of no paths 8,
  of dont know paths 3
• of yes paths lies between
• 20 resolve all dont know paths as no paths
• 23 resolve all dont know paths as yes
  paths
• Create an uncertainty region 0.6 - ?1 , 0.6
  ?2
• ?1 and ?2 depends on p-value for decision along
  all the sample paths
• Check if 20/30,23/30 falls outside 0.6 - ?1 ,
  0.6 ?2

0.6-?1
0.6?2
0.0
1.0
0.6
23/30
20/30
12
Case 1 yes answer
p-value
r
p
0.6-?1
0.6?2
0.0
1.0
0.6
13
Case 1 no answer
p-value
r
p
0.6-?1
0.6?2
0.0
1.0
0.6
14
Case 1 dont know answer
no p-value
0.6-?1
0.6?2
0.0
1.0
0.6
15
Evaluation

• Tandem Queuing Network
• Cyclic Polling System
• Grid World Example
• Answers matched the numerical model-checker
• p-value (?) of the order 10-8 in all of our
  experiments
• Very high confidence in our result
• Disadvantage Space requirement is high
• Required to store all samples before
  model-checking

16
Future Work

• Model-check liveness properties
• ?1 U ?2 (unbounded untils)
• Use Machine Learning to get rid of state
  identifiers
• Verify probabilistic properties of various
  network protocols
• Earlier intractable due to large state space