Benoit Baudry

1

• Benoit Baudry
• 2000 Masters degree at the Univ. de Rennes 1 in
• june 2003 PhD thesis, Testable assembly and
  component validation  with Yves Le Traon and
  Jean-Marc Jézéquel in the Triskell group at the
  Univ. de Rennes 1
• Next Postdoc position at the CEA-Saclay on MDA

2
Test in the Triskell group

• Triskell
• Model Driven Engineering for Component Based
  Software (http//www.irisa.fr/triskell/welcome.htm
  )
• UML-based OO testing (Yves Le Traon)
• Test order, integration strategies
• Vu Le Hanh. -- test et modèle UML stratégie,
  plan et synthèse de test. -- PhD thesis
• Test generation from the requirements (Clémentine
  Nebut)
• UML-based test generation
• Alain Le Guennec. -- Génie Logiciel et Méthodes
  Formelles avec UML Spécification, Validation et
  Génération de tests. -- PhD thesis
• PhD starting on MDA and testing (Franck Fleurey)

3
Testable assembly and component validation

• Benoit Baudry
• Triskell group, IRISA, Rennes, France
• Jean-Marc Jézéquel
• Yves Le Traon

4

• Complex software systems are built with
  components as the unit for reuse
• Trustable components
• Techniques to assemble components

5
Trustable component
Specification
executables contracts
V V Test cases set
Trust based on consistency
Implementation
6
Components assembly

• Impact of design by contract
• Impact of coupling on a testability factor

7
Context

• Software components
• Object oriented design and analysis
• Specific structures in OO programs
• Need to adapt testing techniques
• R. V. Binder, "Testing Object-Oriented Systems
  Models, Patterns and Tools". Addison-Wesley 1999.

8
Summary

• Related work
• Software testing
• Mutation analysis
• Automatic test cases generation
• Design by Contract and testing
• Component testability

9
Software testing

• Two types
• Structural based on implementation
• Functional based on specification of
  functionalities
• Objectives
• Examine or execute a program searching for errors
• Possibly
• robustness
• performances
• safety properties

10
Le test de logiciel
Generation Test data
Execution Test case
Diagnostic
verified
correct
Oracle
correct
Test criterion
verified
Stop
11
Mutation analysisR. DeMillo, R. Lipton and F.
Sayward, "Hints on Test Data Selection Help For
The Practicing Programmer". IEEE Computer 11(4)
34 - 41 April 1978.

• Technique that aims at validating the quality of
  a test cases set
• Errors injected in the program under test
• Compute the proportion of errors detected by the
  test cases

12
Mutation analysis

• Several error types mutation operators
• Operators defined from analysis of errors sets
  observed during development
• J. Offutt, A. Lee, G. Rothermel, R. H. Untch and
  C. Zapf, "An Experimental Determination of
  Sufficient Mutant Operators". ACM Transactions on
  Software Engineering and Methodology 5(2) 99 -
  118 April 1996.
• Recent work Ma02, Alexander02 propose OO
  specific operators

13
Mutation analysis

• Faulty program mutant
• Test cases detect mutants
• Test cases kill mutants
• Mutation score
• Proportion of killed mutants ? quality of test
  cases
• Two oracles
• Traces difference
• Executable contracts

14
Mutation analysis
P
TC
Deleted from the set of mutants
Add contracts
15
Automatic test cases generation and optimisation
16
Automatic test cases optimisation

• Average mutation score easy to reach
• By hand or random generation
• Unit testing
• Class testing in an OO context
• Component testing
• Classes assembly with a main interface class

17
Genetic algorithm
Build an initial population
Compute the fitness value for each individual
? Mutation score
Reproduction
Crossover
Mutate one or several individuals
test_hour is do time.set_hour(0)
time.set_hour(1) end test_out is do
time.set(02,02,02) end test3 is do
time.set(0,0,0) end
test_hour is do time.set_hour(0)
time.set_hour(1) end test_out is do
time.set(02,02,02) end test3 is do
time.set(0,0,10) end
Several stop criteria good mutation score,
number of generations
18
Genetic algorithm

• Tools development (Java, C)
• For mutation analysis JMutator, NMutator
• Framework for the genetic algorithm
• Test driver
• Experiments
• Several classes or components

19
C case study
20
Genetic algorithm

• Fixed size for the test cases set
• Crossover not much useful
• Reproduction not efficient to keep memory

21
Bacteriologic algorithmRosenzweig , Species
diversity in space and time, CUP,1995

• Delete
• The notion of individual
• Crossover operation
• Introduc
• The notion of bacterium ? a test case
• A memory set of good bacteria

22
Bacteriologic algorithm
Build an initial medium
Compute the fitness value for each bacterium
? Mutation score
Memorise the best ones
Mutation
test_hour is do time.set_hour(0)
time.set_hour(1) end
test_out is do time.set(15,59,59) end
test_out is do time.set(15,60,59) end
test_out is do time.set(02,02,02) end
Several stop criteria good mutation score,
number of generations
test_out is do time.set(15,59,59) end
test3 is do time.set(0,0,0) end
23
Bacteriologic algorithm

• Tools development (Java, C)
• Experiments
• Several case studies
• Tuning / validating the model

24
C case study
25
Results

• Original algorithm for automatic test cases
  generation
• Tools development
• Work in progress with new programs and new
  fitness functions

26
Design by contract for robustness and
diagnosability
27
Design-by-contract

• A design method for OO components (B. Meyer)
• Boolean assertions
• pre et post conditions for each method
• invariants for global properties
• A broken contract indicates the presence of a
  bug
• Precondition violated ? a client has broken the
  contrat
• Postcondition violated ? an error in the method

28
Two quality criteria

• Robustness
• Ability for a component to detect a faulty
  internal state
• Diagnosability
• effort for the localization of a fault and the
  preciseness allowed by a test strategy on a given
  system, knowing there is a fault

29
Robustness
contracts
Local robustness ability of contracts to
detect errors
Det(A,C)
Global robustness
30
Robustness

• Global robustness for a components assembly
  depends on
• Local robustness of components
• The Det(i,j) probability a component i detects
  errors in j
• Mutation analysis test cases set with a
  100 mutation score
• local robustness ? mutation score of contracts
• Det(i,j) probability ? mutation score for
  contracts in i with mutants of component j

31
Robustness
32
Diagnosability
33
Diagnosability
34
Results

• Qualitative study of design-by-contract
• Summary
• Adding contracts, even if they are weak,
  improves the components quality
• Efficiency of contracts has more impact than
  the density

35
Testability anti-patterns in a UML class diagram
36
Testability of OO software

• Control is distributed
• Numerous interactions between objects
• Ambiguities in the design can lead to hard points
  for testing
• Class diagram
• Test criterion
• Detecting / deleting testability anti-patterns

37
Example
38
Example
39
Example
40
Example
41
Example
42
Anti-patterns

• Two anti-patterns on the design
• Self-usage

43
Anti-patterns

• Class interaction

44
Improve testability

• Add preciseness on the design to make it closer
  to implementation
• Refactoring for testability
• Use interfaces when possible
• Use stereotypes
• Specify the roles of associations consult,
  create, modify

45
Improve testability
46
Improve testability
47
Improve testability
48
Improve testability
create
create
49
Methodology for testability
Testability analysis
Class diagram
Accept and implement the design
Improve the design
Check properties on implementation
Refuse the design
Test
50
Results

• Testability analysis on the design
• Methodology to improve the class diagram
• Catalogue for the testability of design patterns

51
Conclusion

• Work for the validation and design of software
  components
• Algorithms for test generation
• Models to measure the quality of components
• Test tools
• Publications (JSS, ISSRE, Metrics, ASE)

52
Approach
 Testing can prove the presence of bugs, but
never their absence  Dijkstra

• Mutation analysis
• Evolutionist algorithm
• Design by contract
• Testability

53
Future work

• Test generation for efficient diagnosis
• Diagnosis algorithms
• Test criteria and bacteriologic algorithm
• Mutation analysis for security
• Design by contract for test oracle
• Testability of design patterns
• The addition of stereotypes is seen as a model
  tranformation