Automated test of LustreSCADE programs Virginia Papailiopoulou virginia'papailiopoulouimag'fr Ioanni

1
Automated test of Lustre/SCADE programsVirginia
Papailiopoulou (virginia.papailiopoulou_at_imag.fr
)Ioannis Parissis
(ioannis.parissis_at_imag.fr)Laboratoire
dInformatique de Grenoble

• TAROT Summer School 2008
• June 23-27, Bath United Kingdom

2
Synchronous / safety-critical software
high level of criticality
control/command programs
3
Synchronous languages

• Esterel, Signal, Lustre
• SCADE (Safety-Critical Applications Development
  Environment)
• Based on Lustre
• Related testing tools (lab)
• LUTESS, LUSTRUCTU (LIG lab, Grenoble)
• GATeL (CEA-LIST, Paris-Saclay)
• Lurette (Verimag, Grenoble)
• SCADE, GATeL, LUTESS LUSTRUCTU in the SIESTA
  project

4
The SIESTA project (jan 2008 jan 2011)
http//www.siesta-project.com/ (in french)
Paris
Grenoble
Valence
Toulouse
Pau
5
SIESTA projectAutomation of testing in SCADE
based development
Informal requirements
Test objectivesTest specification Test generation
Lutess V2
Test execution
Quality evaluation
Structural coverage for SCADE
6
Outline

• Structural coverage of Lustre/SCADE programs
• Automating test generation with Lutess V2

7
Lustre overview
E (E1, E2, E3, )
data flow
cyclic behavior
declarative language
synchronism hypothesis
node never (E bool) returns (S bool) let S
not(E) -gt (not(E) and pre(S)) tel
graphical representation operator network
8
Structural Coverage Assessment
Informal requirements
Test data selection
Test execution
Define when the test stops
Structural Coverage criteria
Quality evaluation
Evaluate the test data quality
9
Structural Coverage Assessment
test objective
test data
test data
SUT functional requirements
no formal relation
system specification
system specification
? How thoroughly the test data
has tested the specification?
10
Structural Coverage Assessment

• Coverage criteria
• adapted to the Lustre paradigm
• measure the test thoroughness and the test effort
• Tool LUSTRUCTU
• automate the coverage assessment process

11
Coverage Criteria

• defined on the operator network
• based on the path activation conditions

(a false) or (b true)
(l true) or (c false)
? AC (a false or b true) and (l true or c
false)
12
Coverage Criteria

• ? Complexity w.r.t. the path length
• ? Complexity w.r.t. the strength of the criterion

Basic Criterion (BC)
Elementary Conditions Criterion (ECC)
Multiple Conditions Criterion (MCC)
satisfy the activation conditions
satisfy all input variations
satisfy all edges variations
true
false
a
false
a
true
a
false
true
l1
l
l1
l
s
s
b
b
c
c
13
LUSTRUCTU (LUstre STRUCTUral)
LUSTRE program
Criterion selection
Operator network
Activation condition for P2
Execution
Activation condition for P3
...
Activation condition for Pn
14
Case Study

• Alarm Management System (Avionics)
• software component involved in flight control
• LUSTRE code
• 20 nodes
• Main node 850 LOC -- 28 inputs 1 boolean
  output
• Operator network (SCADE)
• ? 200 operators (boolean, arithmetic, temporal)
• ? 300 edges (boolean integer)

15
Case Study
16
Objectives

• Criteria demonstration
• Applicability
• nodes of different size and complexity
• required effort in function of the criterion
  strength
• Usefulness
• fault detection ability

17
Results Activation conditions computation cost

• the stronger the criterion ? the more the
  conditions
• affordable cost for small nodes
• increased cost in memory time for larger nodes

18
Results Test effort VS Criteria satisfaction

• Random testing ? independent from human factors
• Selected data average of 10 test suites
• Sequence length 1 to several thousands

19
Results Fault detection ability

• Mutation-based technique
• changes/replacements of operators
• mutants generated automatically
• 400 mutants for the main node
• Mutation score coverage ratio comparison

20
Results Fault detection ability
21
Case Study Summary

• Applicability on nodes with small size - unit
  test context
• Methodological suggestions
• First, achieve 100 of BC
• Ensure high ratio (70-80) of ECC and MCC
• 100 of ECC and MCC (if possible)
• ? program analysis important test effort
• Random testing insufficient to evaluate the
  effectiveness
• ? test cases defined by human testers

22
Perspectives

• Integration testing

23
Perspectives

• Parallel use of data-flow and control flow graphs
  (SCADE V6)

24
Perspectives

• Extend the criteria to multiple clocks
• Use
• prevent useless operations
• save computational resources
• Operators when and current

25
Other approaches

• C code coverage
• link with the corresponding Lustre code coverage
• Requirements-based coverage Hemdal Rajan
• criteria defined on formal requirements
• generate requirements-based tests

Informal requirements
Requirements-based Test cases
Formalized requirements
Formalized requirements
coverage
26
Outline

• Structural coverage of Lustre/SCADE programs
• Automating test generation with Lutess V2

27
SIESTA projectAutomation of testing in SCADE
based development
Informal requirements
Test objectivesTest specification Test generation
Lutess V2
Test execution
Quality evaluation
Structural coverage for SCADE
28
Automating the test generation

• Model-based testing
• Several models can be used
• Lustre/SCADE code
• Testing for structural coverage
• LUSTRUCTU GATeL (not detailed here)
• Behavioural specifications
• Both code behavioural specifications

29
A (very) simple air conditionner controller
30
Lustre implementation of the AC controller
node Program_AC(Button bool Tamb, Tuser
int) returns (OnOff bool Tout int)
let OnOff Button -gt ((pre OnOff) and not
Button) or (not(pre
OnOff) and Button) Tout Tuser (Tuser -
Tamb) / 3 tel
31
Lustre implementation of the AC controller
node Program_AC(Button bool Tamb, Tuser
int) returns (OnOff bool Tout int)
let OnOff Button -gt (pre OnOff and
not(Button)) or (not(pre
OnOff) and Button) Tout Tuser (Tuser -
Tamb) / 3 tel
32
Testing from behavioural specifications

• Behavioural specifications can be
• Assumptions on the controlled environment
• Physical constraints, absence of hardware
  failures
• Behaviours that must be tested (scenarios)
• What could happen if the train speed is over
  200mph?
• Expected software properties
• The program should never send a bad command
• Assumptions on the software behaviour
• The will never ask for takeoff if the plane has
  no fuel.

33
Lutess V2
Program_AC
Constrained randomtest generator
Lutess V2(based on constrain programming)
Test model
34
Lutess V2
Program AC
Constrained randomtest generator
Lutess V2
Test model
testnode TestAC() returns () let --
specification tel
Test modeling methodology with Lutess V2
35
Physical invariants domain definition

•  There are constraints on the temperature
  range 

36
Physical invariants domain definition

• testnode TestAC(OnOff bool Tout int)
  returns (Button bool Tamb, Tuser int)
• let
• environment((Tamb gt -20) and (Tamb lt 60) and
  (Tuser gt 10) and (Tuser lt 40))
• tel

37
(No Transcript)
38
(No Transcript)
39
Methodology

• First step Domain definition

40
Physical invariants system dynamics

•  No big variations of the ambient temperature 

41
Physical invariants system dynamics

• testnode TestAC(OnOff bool Tout int)
  returns (Button bool Tamb, Tuser int)
• var dT int
• let
• environment((dT gt -1) and (dT lt 1) and
  (Tamb gt -20) and (Tamb lt 60) and
  (Tuser gt 10) and (Tuser lt 40))
• dT 0 -gt Tamb - pre Tamb
• tel

42
(No Transcript)
43
Methodology

• First step Domain definition
• Second step System dynamics

44
Scenarios

•  If the AC is off, then the user will turn it
  on 

45
Scenarios

• testnode TestAC () returns ()
• var dT int
• let
• environment(true-gt(implies(not(pre(OnOff)),
  Button) and
• implies(pre(OnOff), not
  Button)))
• dT 0 -gt Tamb - pre Tamb
• tel

46
(No Transcript)
47
Methodology

• First step Domain definition
• Second step System dynamics
• Third step User-defined scenarios
• Deterministic

48
Scenarios non deterministic operational profiles

•  If the AC is off, the user will (very
  probably) turn it on 

49
Scenarios non deterministic operational profiles
testnode TestAC () returns () var dT int
let prob(false -gt not(pre(OnOff)), Button,
0.9) prob(false -gt pre(OnOff), not Button,
0.2) environment() tel
50
(No Transcript)
51
Scenarios non deterministic operational profiles

•  If the AC is off and the temperature is too
  high or too low, then the user will (very
  probably) turn the AC on 

52
Scenarios non deterministic operational profiles
prob(false -gt not(pre(OnOff)) and
((Tuser-Tamb gt 5) or (Tuser-Tamb lt -5)),
Button, 0.99) prob(false -gt (pre(OnOff)) or
((Tuser-Tamb lt 5) and (Tuser-Tamb gt -5)),
not Button, 0.01)
53
Scenarios

•  No big user temperature variations 

54
Scenarios

• environment(
• (dTuser gt -1) and (dTuser lt 1)
• )
• dTuser 0 -gt Tuser - pre Tuser

55
(No Transcript)
56
Scenarios non deterministic operational profiles

•  If the AC is on and in cooling mode,then the
  temperature will (probably) decrease 

57
Scenarios non deterministic operational profiles
prob(false -gt pre(OnOff) and pre(Tout lt Tamb),
dT lt 0, 0.99) prob(false -gt pre(OnOff) and
pre(Tout gt Tamb), dT gt 0, 0.99) dT
0 -gt Tamb - pre Tamb
58
(No Transcript)
59
Methodology

• First step Domain definition
• Second step System dynamics
• Third step User-defined scenarios
• Deterministic
• Non-deterministic

60
Safety property guided testing

•   If the AC is on and the ambient temerature is
  too high, then the AC will be in cooling mode  

61
Safety property guided testing

• let
• safeprop(implies((OnOff and Tamb lt Tuser),
  (Tout gt Tuser)))
• environment()
• tel

62
(No Transcript)
63
Methodology

• First step Domain definition
• Second step System dynamics
• Third step User-defined scenarios
• Deterministic
• Non-deterministic
• Fourth step Safety property guided testing
• Safety property specification

64
Using hypotheses on the black-box

• let
• safeprop(implies((OnOff and Tamb lt Tuser),
  (Tout gt Tuser)))
• hypothesis(true -gt (Button (OnOff ltgt pre
  OnOff)))
• environment((dT gt -1) and (dT lt 1)
• tel

65
(No Transcript)
66
Methodology

• First step Domain definition
• Second step System dynamics
• Third step User-defined scenarios
• Deterministic
• Non-deterministic
• Fourth step Safety property guided testing
• Safety property specification
• Adding hypotheses

67
Conclusions

• Lutess V2
• Generator of test generators
• Based on CLP
• Assessments on case studies
• The steam-boiler specification (Abrial 1994)
• A landing gear controller
• Full methodology learned and applied by an
  undergraduate student in few weeks.

68
General conclusion - Perspectives

• Strong demand for testing tools/methods for
  synchronous applications
• Lutess Lustructu
• Can be included in a global testing methodology
• Test quality assessment
• Test case generation
• Together with other tools (GATeL)
• Dealing with the real-world problems
• Siesta project (www.siesta-project.com)

69
Questions

• ?