Using a Model Checker to Test Safety Properties

1
Using a Model Checkerto Test Safety Properties

• ICECCS 2001(International Conference on
  Engineering of Complex Computer Systems)
• 2001
• Paul Ammann, Wei Ding, and Daling Xu
• 2002. 10. 22.
• Presented by Cho, Jaemyung

2
Contents

• Introduction
• Background
• Framework
• Use model checker generate test set
• Conclusion

3
Introduction

• Specification-based testing
• Use a formal method of model checking
• Generate new test sets
• Analyze existing test sets
• Safety properties expressed temporal logic
• System is free from accident or loss leveson
• Failing test / Passing test
• Make mutation model for model cheking spec
• Formalize dangerous trace
• Evaluation test sets satisfy coverage criteria
• Coverage criteria
• Based on CTL operator(AX,AF, EX, EF)

4
Specification-based testing

• Determines whether an implementation conforms to
  its specification
• Consists of two main steps
• Test generation from specifications
• Test execution of an implementation

Abstraction Define/Analyze Tests
Actual System Execute Tests
5
Model checker(1/5)
True or false
System model
Model checker
witness or counterexample
Logic formula
1
2
a
a, c
b gt AX a true
a gt AX b false! Counter example 1, 3, 2, 1
b
3
6
Model checker(2/5)

• Advantages of Using a Model Checker
• Some test requirements are infeasible
• Example Maybe State x is unreachable
• Model checker identifies these
• A single test may satisfy multiple requirements
• Straightforward to identify
• Additional payoff from modeling effort
• Modeling is expensive, so amortize the cost
• Very expressive mechanism

7
Model checker(3/5)

• Example traffic light
• ! EF(NtoS.light G StoN.light G
• E(EW.light R U NtoE.light G
• E(EW.light R U StoN.light G)))

NtoS.light G StoN.light G
NtoE.light G
StoN.light G
...

6
9
2
EW.light R
EW.light R
8
Model checker(4/5)

• Counterexample for traffic light
• State 1 NtoS.light R, StoN.light R,
  NtoE.light R, EW.light R
• State 2 NtoS.light G, StoN.ligtht G
• State 4 StoN.light Y -- not explicitly
  specified
• State 5 StoN.light R -- not explicitly
  specified
• State 6 NtoE.light G
• State 7 NtoE.light Y -- not explicitly
  specified
• State 8 NtoE.light R -- not explicitly
  specified
• State 9 NtoS.light G

9
Model checker(5/5)

• Counterexample
• Appears to be a sequence of states
• This looks a lot like a test case
• In Computational Tree Logic (CTL)
• Root of some computational tree that shows
  formula does not hold
• Usually decorated with path information to reach
  this root

10
Mutation testing

• Small changes are made in a module and then the
  original and mutant modules are compared
• Mutant a program (P) that is similar to the
  program that you are testing (P)

Input 1
Input 1
Original P
Mutant P
Output 2
Expected output 1
11
Passing test / Failing test
Passing test
failing test
implementation
specification
implementation
specification
!p
p
p
p
p
12
Frame work(1/6)

• Test set generation

SMV system specification
model
counter example
formula
Check model
Derive test reqs
reduce
Test set
Test Criteria
13
Frame work(2/6)

• Test set evaluation

Make machines
Test set
model
formula
SMV system specification
Derive test reqs
Check model
Coverage report
Test criteria
14
Frame work(3/6)

• Test set evaluation

1
2
3
5
4
7
6
Existing Test Set Example
15
Frame work(4/6)

• Test set evaluation

Evaluate Test Requirements Against This Finite
State Machine
16
Framework(5/6)

• D(Dangerous)-trace

M Union of M and M
M-M In mutant but not in original machine
M original state machine
A dangerous trace
Turn point
P is violated
17
Framework(6/6)

• Dangerous EX Trace with Passing/Failing Tests
• EX trace t a, b, c
• Passing test for EX trace t a, b, c, d
• Failing test for EX trace t a, b, c, e

d
a
b
c
e
e
Violate P
18
Use model checker generate test set (1/4)

• Combining the Original and Mutant State Machine

Original State Machine(SM)
Mutated State Machine(SM)
a1
0
0
1
agt0
a0
a1
a0
Change a0 to agt0
init(x) 0 next(x) case a00 a11 esac
init(x) 0 next(x) case agt00 a11 esac
Original State Machine(SM) (with lost traces)
init(x) 0 next(x) case a0agt0 0 a1
1 esac
0
1
a1
a0
a0
a1
agt0
19
Use model checker generate test set (2/4)

• Combining the Original and Mutant State Machine

Original State Machine(SM) (with lost traces)
init(x) 0 next(x) case a0agt0 0 a1
1 esac
0
1
a1
a0
a0
a1
agt0
Original State Machine(SM) (include all states)
init(x) 0 next(x) case a1agt01,0 a0agt
0 0 a1 1 esac
0
1
a1
a0
a0
a1
agt0
20
Use model checker generate test set (3/4)

• Expressing test reqs in T/L
• To cover an EX trace
• -- Failing
• EF(original EX(!original !P))
• -- Passing
• EF(original EX(original) EX(!original !P))
• SMV input
• -- Failing
• SPEC !EF(original EX(!original !P))
• -- Passing
• SPEC !EF(original EX(original) EX(!original
  !P))

21
Use model checker generate test set (4/4)

• To cover an EX trace

1
-- Failing EF(original EX(!original !P)) --
Passing EF(original EX(original) EX(!original
!P))
2
3
M1
M2
M3
4
-- Failing test set (1,2) -- Passing test set
(1,3)
22
Conclusion

• Model checker
• Test set generation
• Mutation testing
• Test set evaluation
• Checking safety property
• Passing failing test