RE04 keynote

1
Chapter5 Quality Assurance

• Jean-Christophe Libbrecht
• Jérôme Paul
• Julien Goffaux

2
Introduction

• The latter errors are found, the more costly
  their repair is.
• Result of this activity is a consolidated RD.

3
introduction

• Quality Assurance consists of
• Detecting defects
• Reporting them
• Analysing their cause
• Undertaking actions to fix them

4
introduction

• Mains techniques
• Inspections and reviews
• Queries on a requirement database
• Animation-based validation
• Formal verification

5
introduction

• When?
• Into requirement documentation
• As soon as portion of specification are available
• Differences with Requirement evaluation
• Not in the same time
• Not the same goal

6
Inspections and reviews

• Ask selected people to inspect the DR
  individually
• Provide list of problem to be solve together
• It works !
• Can even outperform code testing
• Proved to work well on Requirements Documents (RD)

7
Inspection process

• Inspection planning
• Size of the team
• Schedule
• Format of the reports

8
Inspection process

• Individual reviewing
• Free mode
• Checklist based
• Process based

9
Inspection process

• Defect evaluation at review meetings
• Defects found by each inspector
• are collected and discussed
• Analysis of the causes
• gt Reduce false positives

10
Inspection process

• RD consolidation
• Document revised to address all the concerns

11
Inspection guidelines

• What?
• Facts
• Constructive
• Approved by all inspectors
• Well structured
• Who?
• Inspectors ? authors
• Representative of all stakeholders viewpoints
• Peoples with different background
• gt Minimum size 3

12
Inspection guidelines

• When?
• Not too soon ...
• ... Not too late
• Where?
• Critical parts
• Safety-security-related

13
Requirement inspection checklists Defect-based
checklist

• Omission
• Contradiction
• Inadequacy
• Ambiguity
• Unmeasurability
• Noise
• Overspecification

14
Defect-based checklist

• Unfeasibility
• Ununtelligibility
• Poor structuring
• Forward reference
• Remorse
• Poor modifiability
• Opacity

15
Other checklists

• Quality-specific
• Focus on non-functional requirements
• Unspecified reponses to out-of-range values
• Are all inputs from sensors used ?
• ...
• Domain-specific
• Focus on domain-related requirements
• Provide increasingly specific guidance in defect
  search

16
Other checklists

• Language-based checklists
• Use the specific constructs of the structured
  (semi)-formal language in the RD.
• Can be automated !
• Decision tables is there a value for every
  field
• Global template check the conformance of RD
  organization to the structure presented by the
  template
• Diagrammatic language surface-level consistency
  and completeness within and between diagrams
• Formal specification languages semantically
  richer consistency and completeness

17
Conclusion of inspection

• Inspection and reviews are
• Even more effective than code inspection.
• Widest in scope and applicability ( any format
  supported)
• Costly in ressources and time

18
Queries on a requirements database
19
Queries on a requirements database

• This technique is
• Tool supported
• Simple and effective
• Adapted for large specifications
• Working on parts of the RD that are specified in
  terms of diagrammatic notations (cf. 4.3)

20
What's the idea ?

• Maintain specifications in a requirements DB
• Queries on DB

21
In practice...

• Meta-tools can perform automatic derivation of
• RDB's schema
• Language-specific database engine

22
How does it work ?

• Input
• Meta-specification of the diagram language
• Output
• A DB engine with
• A diagram-specific query language
• A processor for specification querying

23
And now... an example

• Context
• Part of the RD is specified with data-flow
  diagrams
• Complex operations are recursively refined into
  finer ones
• We want to make sure that each data flowing out a
  complex operation appears as data flowing out of
  one (or more) of its sub-operation(s).

24
example.next()

• We check the opposite
• Is there any output data flowing out of this
  operation that is not flowing out of any of the
  refining sub-operations?
• This checks
• Consistency between adjacent levels of operation
  refinement
• Completeness on the refining data-flow diagram

25
example.next()

• Same question in a DFD-specific query language

26
The result

• Empty set
• No inconsistency detected
• Non-empty set
• which indicates the output flows missing in the
  DFD refining the operation

27
Remark

• The DFD-specific query language and its
  query-engine are generated from a specification
  of the DFD language in an ER meta-language

28
Example of ER meta-specification
29
The good news of the day

• In practice, these structural checks are automated

30
Requirement validation by specification animations
31
Reminder

• We want to
• check the adequacy of requirements and
  assumptions
• see wether the system-to-be as specified
  corresponds to the stakeholders expectations

32
Validation scenarios

• With the stakeholders, we will check
• interactions among system components
• sequencing

33
Visualization

• Visualization is possible
• Scenarios enactment tools

34
The problem ...

• Identification of representative scenario sample
  from the specified requirements and assumptions,
  for comprehensive system coverage MAY NOT BE EASY

35
Another approach animating parts of the
specified system

• How ?
• Extract or generate an executable model of the
  software-to-be from the specifications
• Simulate system behaviour
• Visualize
• check with stakeholders

36
Simulation

• Submit events to the model that mimic behaviours
  of the environment
• The model software reacts to these events

37
Visualization

• Shows how suggestive representation of system
  evolves dynamically

38
Remark simulation gtlt animation

• Simulation
• execution of software model
• Animation
• Visualization, in a suggestive form, of the
  simulated model in its environment

39
Another remark prototyping VS animation

• Prototype
• Quick implementation in a very high level
  language
• Not necessarily based on a model
• Pays less attention to system visualization

40
Many tools for animations

• State-based
• Jaza (Z), B-core, B-toolkit (B), VDM-tools (VDM),
  ...
• Transition-based
• SCR animator (SCR), Statemate, Rhapsody
  (Statecharts), Nimbus (RSML), LTSA (labelled
  transition system)
• History-based
• FAUST

41
Extracting an executable model from the
specifications

• Determine the part of the specification we want
  to animate for adequacy checking
• Transform it into an executable version that is
  semantically equivalent

42
Example

• For state machine specifications, the result is
  typically a set of parallel state machines (one
  per class we want to animate)
• In our train system one state machine for the
  dynamics of
• The train doors
• The train moves
• ...

43
Simulating the model

• Instantiation of the extracted model
• Ex
• a single train and 3 stations
• 2 trains and 1 station
• Execution of the instantiated state machines
• NextState function

44
Role of the NextState function

• Control the concurrent activation of multiple
  transitions fired by the occurrence of multiple
  triggering events
• Handle clock ticking for the animator's stepwise
  execution

45
Definition of NextState
46
Remark

• Two conflicting transition are BOTH discarded
• A less strict approach
• Keep one of the conflicting transition
• Problem arbitrary choice

47
Visualizing the simulation

• Suggestive way
• Animator should make the stakeholders convenient
  to input events and watch the model's reactions

48
Visualization formats

• Textual
• I Text command
• O execution trace
• Diagrammatic
• I Event selection wrt the current state
• O Tokens processing along the model diagrams
  together with state visualization
• Domain-specific
• I domain-specific control devices
• O new values on domain-specific monitors

49
Some tools skills

• Control devices and panel
• SCR and RSML
• Domain scene
• LTSA and FAUST

50
A running example
51
Conclusion on requirement animationAdvantages

• Concrete technique for checking a specification
• May reveal subtle inadequacies and other defects
• One of the best ways of getting stakeholders and
  practitioners into the quality assurance loop
• Interesting animation scenarios may provide
  acceptance test data for free
• Animators can be coupled with other analysis
  tools
• monitors that detect property violations on the
  fly
• model checkers

52
Conclusion on requirement animationInconvenient

• Need of a formal specification
• No guarantee that rewarding animation sequences
  will be played
• The user of the animator needs to be experimented
  and to know about tricky things that can happen
  in the environment
• Animation scenarios should be carefully
  elaborated to to ensure comprehensive model
  coverage
• Gap between the animated model and the original
  specification

53
Formal verification

• Checks on a specification that can be automated
  by tools
• Specification must be formal
• Checks depend on the specific constructs of the
  specification language

54
Formal verification

• Language checks
• Dedicated consistency and completeness checks
• Model checking
• Theorem proving

55
Languages checks

• Syntax checking
• Every expression in the specification must be
  grammatically well formed according to the syntax
  rules of the language
• Example Free' Free instead of Free' Free

56
Languages checks

• Type checking
• Each variable must have a specified type and all
  uses of this variable must be convenient with the
  type declaration
• Example Free Ambulance and Free'Form

57
Languages checks

• Static semantic checking
• Each variable must be declared and have a
  specified initial value
• Variable must be used within their scope and so
  forth

58
Languages checks

• Circularity checking
• No variable is being defined in terms of itself
• Example Ambulance Ambulance \ amb?

59
Dedicated consistency and completeness checks

• Relies on input-output relations
• Relations are functions
• Relations are total

60
Dedicated consistency and completeness checks

• Checking disjointness of input cases for
  consistency
• Check that relation is a function
• Conditions entries in each input row of the table
  are pairwise disjoint No same input with
  different input
• Cond_Entry1 Cond_Entry2 false

61
Dedicated consistency and completeness checks

• Application of Checking disjointness of input
  cases for consistency
• (OnAcident OR OnRoad) AND (NOT OnAcident AND NOT
  OnRoad)
• (OnAcident AND NOT OnAccident AND NOT OnRoad) OR
  (OnRoad AND NOT OnRoad AND NOT OnAccident)?
• (false AND NOT OnRoad) OR (false AND NOT
  OnAccident)?
• false

62
Dedicated consistency and completeness checks

• Checking coverage of input cases for
  exhaustiveness
• Check that function is total
• The conditions entries cover all possible cases
• Cond_Entry1 v ... v Cond_EntryN true

63
Dedicated consistency and completeness checks

• Checking coverage of input cases for
  exhaustiveness
• (OnAcident OR OnRoad) OR (NOT OnAcident AND NOT
  OnRoad)
• (OnAcident OR OnRoad OR NOT OnAcident) AND
  (OnAcident OR OnRoad OR NOT OnRoad)
• (true OR NOT OnAcident) AND (true OR NOT
  OnRoad) true

64
Model Checking

• Verify that a formally specified model satisfies
  some desired property.
• Explore the model in a systematic way to search
  for property violations

65
Model Checking

• Brute-force search algorithm builds the model's
  reachability graph
• Reachability graph and algorithm
• Contains all possible reachable states
• Explored by recursively generating next nodes
• Testing node if it violate the property
• Visited state are marked to avoid cycling
• Algorithm terminates when a bad states is reached
  or when all states has been visited
• If a bad node is reached, violating path is
  produced

66
Model Checking

• Property
• MovementState 'moving' -gt DoorsState 'closed'
• A possible counterexample
• Init (doorsClosed,trainStopped)?
• Start (doorsClosed,trainMoving)?
• Speed 0 (doorsClosed,trainStopped)?
• Opening (doorsOpen,trainStopped)?
• Start (doorsOpen,trainMoving)?

67
Model Checking

• Properties verifiable by checker
• Reachability simply inspect reachability graph
• Safety undesirable condition may never happen
  (?P)?
• Liveness desirable condition must eventually
  hold (?P)?

68
Model Checking

• Issues
• Combinatorial explosion of states valuesVars
• Length and comprehensibility of counterexample

69
Model Checking

• Best-known variants
• SPIN
• properties formalized in TTL
• state machine model must be expressed in PROMELA
• the visited states are maintained in a tunable
  hash table
• SMW
• properties formalized in CTL
• branching temporal logic where histories have a
  tree structure with branching to alternative
  successor states
• sets of visited states are represented
  symbolically by binary decision diagrams

70
PROMELA
spin -a increment.spin Translate the
model to C cc -DSAFETY -o pan pan.c
Compile the model ./pan
Run the model
pan assertion violated ((sumlt2)(counter2))
(at depth 20)? pan wrote
increment.spin.trail (Spin Version 4.2.5
-- 2 April 2005)? Warning Search not
completed Partial Order
Reduction Full statespace search for
never claim - (none
specified)? assertion violations
cycle checks -
(disabled by -DSAFETY)? invalid
end states State-vector 40 byte,
depth reached 22, errors 1 45
states, stored 13 states, matched
58 transitions ( storedmatched)?
51 atomic steps hash
conflicts 0 (resolved)? 2.622 memory
usage (Mbyte)?
71
Model Checking

• Alloy analyzer
• Input model is specified in Alloy,a state-based
  specification language
• Property specified in Alloy
• Model is first instantied by restricting the
  range of the model variables to a few values, in
  order to avoid state explosion
• The analyzer tries to satisfy the negation of the
  claim within that bounded model, using an
  efficient SAT solver

72
(No Transcript)
73
Model Checking

• Strengths
• Verification fully automated
• No human intervention is required during
  verification
• Analysis is exhaustive
• Help debugging with the counterexample

74
Theorem Proving

• Verify property deductively rather than
  algorithmically
• Target property (Theorem) can be derived form the
  the formal specification (axioms) by a sequence f
  applications of inference rules
• User may be required to confirm or reject
  generated lemmas that the tool cannot verify

75
Theorem Proving

• Inferences rules
• Modus Ponens
• Chaining rules
• First-order functional substitutivity
• First-order predicate substitutivity
• Z-specific rule

76
Theorem Proving
77
Theorem Proving
78
Theorem Proving

• Assertion to verify
• Verification

79
Theorem Proving

• Strengths
• Soundness and completeness
• Infinite state spaces can be handled
• Issues
• Difficulty to use it
• No concrete counterexample produced