TOBIAS and some ASE challenges

1
TOBIASand some ASE challenges

• Y. Ledru
• P. Bontron, O. Maury,
• L. du Bousquet, M.L. Potet,
• C. Oriat , S. Beghdadi, H. Bouldjedri
• LSR/IMAG Grenoble
• ASE Workshop at Irvine, June 2002

2
Overview

• Two challenges of ASE
• TOBIAS
• How TOBIAS is confronted to these challenges

3
Overview

• Two challenges of ASE
• TOBIAS
• How TOBIAS is confronted to these challenges

4
Signature of ASE tools
Formal or structured document
Formal or structured document
ASE tool
5
1st challenge process documents

• Complexity of input/output documents
• Algorithmic complexity of processing
• Combinatorial explosion
• Undecidable problems

Challenge 1 Process documents
Formal or structured document
Formal or structured document
ASE tool
Domain specific tools
6
2nd challenge produce documents

• Software engineers only like to write code! XP
• Millions of amateur software engineers
• how can we learn them category theory?

Challenge 1 Process documents
Challenge 2 Produce documents
Formal or structured document
Formal or structured document
ASE tool
Capture information
7
Elements of solutions

• Dont underestimate software engineers provide
  useful/efficient/scalable tools!
• Provide easier formalisms (graphical formalisms,
  GUI tools,)
• Integrate your tools in the process of the
  company!

Challenge 1 Process documents
Challenge 2 Produce documents
Formal or structured document
Formal or structured document
ASE tool
Capture information
costs vs benefits
8
Overview

• Two challenges of ASE
• TOBIAS
• How TOBIAS is confronted to these challenges

9
TOBIAS
Class signatures
Test data/ Test cases/ Test purposes
TOBIAS
Capture information
Test schemas

• Generates a large number of tests
• from a generative pattern (test schema)
• Should be used in combination with tools that
  provide the test oracle (VDMTools, TGV).

10
A simple example

• An integer set classwith two operations
• manual tests
• Add(0)
• Add(0),Add(1),Add(2)
• Add(0),Add(0)
• Add(0),Remove(0)
• Add(0),Add(1),Remove(0)
• Add(0),Remove(1)

11
A TOBIAS test schema

• Add(x)1..3Remove(y)0..2
• Where
• x 0,1,2,3
• y 0,1,2

• Add(x1)
• Add(x1),Add(x2)
• Add(x1),Add(x2),Add(x3)
• Add(x1),Remove(y1)
• Add(x1),Add(x2),Remove(y1)
• Add(x1),Add(x2),Add(x3), Remove(y1)
• Add(x1),Remove(y1),Remove(y2)
• Add(x1),Add(x2),Remove(y1),Remove(y2)
• Add(x1),Add(x2),Add(x3), Remove(y1),Remove(y2)

(444444)(1333) 1092 sequences
12
Results of TOBIAS

• TOBIAS generates the manual tests lots of
  similar tests
• for the same design cost
• A recent experiment
• 45 manual tests
• 4320 TOBIAS tests
• same test design effort
• 2 new errors detected
• TOBIAS helps detecting more errors!

13
Overview

• Two challenges of ASE
• TOBIAS
• How TOBIAS is confronted to these challenges

14
1st challenge complexity

• Intrinsic to the tool
• generates a large number of tests
• because IUTs are complex
• Still too many tests are useless
• Require untractable resources
• May not detect more errors
• Keep the number of tests small enough
• Avoid redundant test cases
• Extend the test schemas
• (constraints, hypotheses)
• Remove non-conform test cases
• Verify vs specification

15
TOBIAS and complexity
Class signatures
Solutions to the first challenge
Test data/ Test cases/ Test purposes
TOBIAS
Capture information
Test schemas
make the second one more complex!
16
Capturing knowledge for TOBIAS
Class signatures can be extracted from existing
documents
Class signatures
UML class diag. Or Java Program
extract
Test schemas are simple and short !
Test data/ Test cases/ Test purposes
TOBIAS
Capture information
Test schemas
TOBIAS provides a GUI for test schemas
Specification (UML diagrams, Pre/post cond., )
Specifications can be used in conjunction with
generated tests (oracle, TGV,)
Specifications are not needed to generate the
first set of tests!
17
Conclusion

• TOBIAS is confronted to both ASE challenges
• complexity
• information acquisition
• Complexity addressed by additional knowledge
• Information acquired progressively
• Provide first benefits at low cost
• Motivate efforts for specification
• Multiple use of specifications (oracle, TGV)
• Fits into a classical process

18
THANKS !
19
Redundant test cases

• The implementation of the integer set may be
  independent from the actual values of integers
• Lots of tests generated by TOBIAS can be
  considered redundant
• This corresponds to the expression of test
  hypotheses.

Add(0) Add(1) Add(2) Add(3)
Add(0),Add(0) Add(2),Add(0) Add(0),Add(1)
Add(2),Add(1) Add(0),Add(2)
Add(2),Add(2) Add(0),Add(3)
Add(2),Add(3) Add(1),Add(0)
Add(3),Add(0) Add(1),Add(1)
Add(3),Add(1) Add(1),Add(2)
Add(3),Add(2) Add(1),Add(3) Add(3),Add(3)
Add(0),Add(1),Add(2) Add(2),Add(1),Add(0)
20

• Add(x)1..3Remove(y)0..2
• Where
• x 0,1,2,3
• y 0,1,2
• Forall x_i,x_j. iltj gt x_iltx_j

Add(0),Add(0) Add(2),Add(0) Add(0),Add(1)
Add(2),Add(1) Add(0),Add(2)
Add(2),Add(2) Add(0),Add(3)
Add(2),Add(3) Add(1),Add(0)
Add(3),Add(0) Add(1),Add(1)
Add(3),Add(1) Add(1),Add(2)
Add(3),Add(2) Add(1),Add(3) Add(3),Add(3)
Add(0),Add(1),Add(2) Add(2),Add(1),Add(0)
21
Overview

• Context the COTE project (COmponent TEsting)
• Why TOBIAS ?
• TOBIAS description of the tool
• Test purpose generation
• Test case generation with VDM
• Smarter generation

22
Overview

• Context the COTE project (COmponent TEsting)
• Why TOBIAS ?
• TOBIAS description of the tool
• Test purpose generation
• Test case generation with VDM
• Smarter generation

23
The COTE project

• A RNTL Project
• Partners
• SOFTEAM (tool objecteering) leader
• IRISA (tools UMLAUT/TGV)
• GEMPLUS (smart cards/software components)
• FT RD (OO technologies/components)
• LSR-IMAG (Software engineering)

24
The context of COTE

• Conformance testing of software components
• The component is tested against its UML
  specification
• Black box testing the internal behaviour of the
  product under test is hidden

25
Aim of the COTE project

• UML sequence diagrams provide an abstraction
  for tests cases
• provide tools for automatic executable test
  generation from UML test cases
• target three component technologies

26
Aim of the COTE project
27
Test case / Test purpose
An example A coffee vending machine
A test purpose a sequence of transitions to reach
A test case a complete path in the behavioural
specification
28
General structure of COTE
UML Specification
UML Test purposes
Test synthesis
Behaviour computation UMLAUT/TGV
UML Test cases
Automatic tests generation
corba
ejb
.net
29
Overview

• Context the COTE project (COmponent TEsting)
• Why TOBIAS ?
• TOBIAS description of the tool
• Test purpose generation
• Test case generation with VDM
• Smarter generation

30
Why TOBIAS ?

• Behavioural testing requires the specification of
  many tests.
• Test campaign gt (potentially) thousands of test
  cases
• UMLAUT/TGV one test purpose and one execution
  one test case
• New values gt creation of new test purposes
• Hugues MARTINs doctoral thesis (GEMPLUS)
•  Une méthodologie de génération automatique
  de suites de tests pour applets java-card. 
• A methodology for automatic generation of
  test suites for java-card applets.

31
Generation of test purposes

• Idea notion of high level test purpose
• (test pattern)
• gt abstraction of test purposes on objects,
  methods or values
• A test pattern gt a set of test purposes gt a set
  of test cases
• Significantly reduces the repetitive tasks of
  test definitions

32
TOBIAS
Test pattern
TOBIAS
UML Specification
Test synthesis
UML Test cases
33
Overview

• Context the COTE project (COmponent TEsting)
• Why TOBIAS ?
• TOBIAS description of the tool
• Test purpose generation
• Test case generation with VDM
• Smarter generation

34
Principles of TOBIAS

• TOBIAS provides
• help for test purposes design
• test pattern descriptions
• Idea
• instantiation of test patterns according to
  different criteria (objects, parameters )

35
TOBIAS inputs
UML specification
deployment diagram
class diagram
test patterns
TOBIAS
Test purpose in UML
36
A little example (1)

• UML class diagram

1
0..1
1..5
37
A little example (2)

• UML deployment diagram (one for each test
  campaign)

TestConfiguration
DrinkMachineVendingMachine
FrontHCI
CoffeeDrink
TeaDrink
CappucinoDrink
ChocolateDrink
TestorActor
ResetCancel
38
Some test purposes
5 test purposes
For - 4 different drinks - 1 cancel touch
(Reset.GiveChange) - 5 different coins 0.1,
0.2, 0.5, 1, 2 The number of different test
purposes is around (55²) x (4 1) 150 test
purposes
39
Principle of test patterns

• Provide the generation of test purposes according
  to different criteria
• the method parameters
• the objects
• the groups
• the number of group or method calls

40
Principle of test patterns

• Provide the generation of test purposes according
  to different criteria
• the method parameters
• the objects
• the groups
• the number of group or method calls

41
Principle of test patterns

• Provide the generation of test purposes according
  to different criteria
• the method parameters
• the objects
• the groups
• the number of group or method calls

42
Principle of test patterns

• Provide the generation of test purposes according
  to different criteria
• the method parameters
• the objects
• the groups
• the number of group or method calls

43
Principle of test patterns

• Provide the generation of test purposes according
  to different criteria
• the method parameters
• the objects
• the groups
• the number of group or method calls

44
An example
GroupAction Drink.ClickButton(),
Cancel.GiveChange()
Using the following test pattern
insertCoin(x)1..2 GroupAction

• TOBIAS generates the 150 test purposes previously
  showed

45
Advantages of using TOBIAS

• TOBIAS is useful to generate many test purposes
  ? generating many test cases using UMLAUT/TGV
• Generating test purposes for new values is easier
• TOBIAS can be used to generate test suites for
  VDM in combination with VDMTools

46
Using TOBIAS with VDM

• VDMtools provide an environment to specify an
  application, to execute test suites and to
  evaluate the test results
• The output of TOBIAS is directly used as a test
  suite.
• Using TOBIAS for the automatic generation of VDM
  test cases - VDM Workshop 2002 at Floc02

47
Using TOBIAS with VDM
VDM specification
Test pattern
UML specification
TOBIAS
VDM test suites
VDMTools

• Results
• fault detection
• code coverage

48
Using TOBIAS with VDM

• We did the following experiment
• specify an application managing groups of
  students
• manually specify a test suite (about 15 test
  cases)
• generate a test suite using TOBIAS (about 4300
  test cases)

49
Using TOBIAS with VDM

• We obtained the following results
• the generated and the manually written test suite
  have the same code coverage
• The development effort is similar
• the generated test suite detect more errors
• one unknown error
• one known error (from reading the specification)
• new activations of a known error
• the generated test suite takes significantly more
  time to be executed
• TOBIAS generates non conforming test cases
• TOBIAS generates redundant test cases

50
Smarter generation

• Preventing non conforming test purposes / test
  cases
• using more elements from the specification
• relations between classes
• pre/post conditions
• state diagrams
• Reducing the test suite size
• eliminating redundant test purposes / test cases
• defining constraints on the schemas

51
Using more elements
TOBIAS extract classes from the UML class diagram.
Class A
Class C
Class B
Class D
the class A calls the methods of the class C
A!C ? C!A ? B!C ? C!B ? A!B ? B!A ?
A!D ? D!A ? B!D ? D!B ? C!D ? D!C ?
A!A ? B!B ? C!C ? D!D ?
52
Using more elements
Relations between classes
A!C ? C!A ? B!C x C!B x A!B x B!A x
A!D x D!A ? B!D ? D!B ? C!D x D!C x
A!A x B!B x C!C x D!D x
53
Using more elements
Deployment diagram
54
Using more elements
Deployment diagram that gives the initial state
of the application
aA
cC
bB
dD
A!C ? a!c? C!A ? c!a? B!C x C!B x A!B x B!A x
A!D x D!A ? d!a? B!D ? b!d? D!B ? d!b? C!D
x D!C x
A!A x B!B x C!C x D!D x
55
Adding constraints

• The user can associate a constraint to each test
  pattern
• Constraints deal with parameters
• Constraints are used to filter test purposes /
  test cases

56
Adding constraints
GroupAction Drink.ClickButton(),
Cancel.GiveChange()
A test pattern insertCoin(x)1..2
GroupAction
- insertCoin(x_1) GroupAction - insertCoin(x_1)
insertCoin(x_2) GroupAction
?
A constraint
Number of selected test purposes
(59)x(41) 70 test purposes
57
Conclusion

• TOBIAS provides a new abstract level for test
  design the test patterns
• A test pattern gt a set of similar test cases
• To avoid combinatorial explosion, the user can
  associate constraints to the schemas
• TOBIAS can be used to generate test sequences
• TOBIAS will be used and validated on an
  industrial test case in the COTE project

58
Prospects

• Refining the test generation
• Deleting redundant
• Using pre/post conditions on operation to
  eliminate non-conforming sequences
• Helping the decision to stop test campaign
• Specification coverage by test patterns ? using
  more elements from the specification