BAiT: Adaptive Test Case Execution in Practice

1
BAiT Adaptive Test Case Executionin Practice

• Jens R. Calamé
• Centrum voor Wiskunde en Informatica, Amsterdam

2
Agenda

• Introduction
• The Toolset for BAiT
• An Exemplary Case The ATM
• Structure of BAiT
• Conclusion

3
Introduction

• BAiT Behavior Adaptation in Testing
• Toolset to generate test cases and test oracles
  as well as execute tests on nondeterministic
  systems with data

4
Agenda

• Introduction
• The Toolset for BAiT
• An Exemplary Case The ATM
• Structure of BAiT
• Conclusion

5
BAiT Test Case Oracle Generation

• Test generation with TGV (enumerative approach)
• Chaotic data abstraction for input and output
  parameters of events
• Test oracle generation for Eclipse Prolog
  constraint solver in order to select test data
• Automatic generation of variable names for the
  test case

6
BAiT Test Proxy Generation

• Test Proxy for Java-based SUTs
• Instantiation of test object within test proxy
• Procedure-basedbidirectional communication
  (publisher-subscriber pattern)
• Mapping of datatypes from specification language
  to target platform

7
BAiT Test Execution

• Test execution of pre-calculated test trace
  (behavior and data)
• Adaptation to the IUT's reactions, rather than
  immediate assignment of verdict
• BAiT framework provides adapters for user-defined
• trace selection and
• test data selection
• algorithms

8
Agenda

• Introduction
• The Toolset for BAiT
• An Exemplary Case The ATM
• Structure of BAiT
• Conclusion

9
An Exemplary Case The ATM
ATM specification (partially)
ATM test purpose (completely)
10
An ATM Test Case
!getAmount(100)
?
??
Photos Wikipedia and ECB
11
An ATM Test Case (2)
1. Precalculation of possible test trace
May 25, 2007 30619 PM nl.cwi.sen2.bait.steps.Tra
ceImpl merge FINE Pruning planned test
trace. May 25, 2007 30619 PM nl.cwi.sen2.bait.Ru
nTest findTrace FINER Examining
init,initPin(Pin),getPin(PinUser),pinCorrect,
initBalance(Bal),getAmount(Amt),tau,retFifty(Fif
),emitBankNotes May 25, 2007 30619 PM
nl.cwi.sen2.bait.steps.TraceImpl solve FINEST
init(G1),initPin(G1,G2,lparam(nat(Pin))),... Pi
n in -Infinity .. Infinity gt 1234 May 25,
2007 30621 PM nl.cwi.sen2.bait.steps.TraceImpl
solve FINEST init(G1),initPin(G1,G2,lparam(nat(12
34))),... ... May 25, 2007 30626 PM
nl.cwi.sen2.bait.RunTest executeTest INFO
Trying init,initPin(1234),getPin(1234),pinCorre
ct, initBalance(1000),getAmount(100),tau,retF
ifty(2),emitBankNotes May 25, 2007 30626 PM
nl.cwi.sen2.bait.RunTest executeTest FINE
initPin(1234) -gt OK ... May 25, 2007 30626 PM
nl.cwi.sen2.bait.RunTest executeTest FINE
retFifty(2) -gt NOT OK May 25, 2007 30626 PM
nl.cwi.sen2.bait.RunTest executeTest INFO
Expected retFifty(2), received retFifty(1) -gt
Trace failed, trying alternative.
2. Iterative data selection
3. Stepwise execution of the trace
until the IUT deviates from the precalculated
trace.
12
An ATM Test Case (3)
May 25, 2007 30626 PM nl.cwi.sen2.bait.RunTest
executeTest INFO Trace deviated executed so
far init,initPin(1234),getPin(1234),pinCor
rect,initBalance(1000), getAmount(100),tau,r
etFifty(1),retTwenty(2),retTen(1),
emitBankNotes May 25, 2007 30626 PM
nl.cwi.sen2.bait.steps.TraceImpl merge FINE
Pruning planned test trace. May 25, 2007 30626
PM nl.cwi.sen2.bait.steps.TraceImpl merge FINE
Adding retFifty(1) to trace stub. ... May 25,
2007 30626 PM nl.cwi.sen2.bait.steps.TraceImpl
merge FINE Adding emitBankNotes to trace
stub. May 25, 2007 30627 PM nl.cwi.sen2.bait.Run
Test executeTest INFO Test finished executed
init,initPin(5),getPin(5),
pinCorrect,initBalance(1000),getAmount(100),tau,
retFifty(1), retTwenty(2),retTen(1),emitBankNo
tes May 25, 2007 30627 PM nl.cwi.sen2.bait.RunTe
st executeTest INFO The test case ended with
verdict PASS.
4. Pruning not yet executed part of test trace.
5. Appending received system output to trace.
6. Solve test trace with pending system output
assign verdict or calculate new trace.
13
Agenda

• Introduction
• The Toolset for BAiT
• An Exemplary Case The ATM
• Structure of BAiT
• Conclusion

14
Structure of BAiT The Tools

• BAiT Behavior Adaptation in Testing
• Toolset to generate test cases and test oracles
  as well as execute tests on nondeterministic
  systems with data

15
Structure of BAiT Execution Part

• Variable class Codec of data to constraint
  solver, data instantiator and IUT
• Data instantiation algorithm(s)

Test Oracle
Data Mapping Constraint Solver
Data Mapping Impl. Under Test
BAiT Execution Core
Impl. Under Test
Data Selection
Constraint Solver
Trace Selection
Behavior Mapping Constraint Solver
Behavior Mapping Impl. Under Test
Test Case(s)

• Trace class trace execution management
• Step class communication with IUT
• Trace selection from complete test graph (IOLTS
  with all possible test cases for a given test
  purpose)

16
Agenda

• Introduction
• The Toolset for BAiT
• An Exemplary Case The ATM
• Structure of BAiT
• Conclusion

17
Conclusion

• Behavior Adaptation in Testing helps finding more
  reliable verdicts in testing non-deterministic
  systems
• A prototypical framework for BAiT has been
  developed procedure-based setting with atomic
  data
• Further work
• Extension of frameworkfor other settings
• Extension of frameworkto structured datatypes
• Case StudyMozilla FirefoxCSS subsystem

18
Further Information

• Background information
• Calamé, Jens R. Ioustinova, Natalia Pol, Jaco
  van de Sidorova, Natalia Data Abstraction and
  Constraint Solving for Conformance Testing (APSEC
  2005).
• Calamé, Jens R. Ioustinova, Natalia Pol, Jaco
  van de Towards Automatic Generation of
  Parameterized Test Cases from Abstractions. Techni
  cal Report, 2006.
• The toolset/framework itself
• Calamé, Jens R. Adaptive Test Case Execution in
  Practice. Technical Report, 2007.
• Calamé, Jens R. The Internals of Adaptive Test
  Case Execution. Technical Report,to appear.
• All downloadable from www.calame.de/works,publishe
  d.bib,date,.html
• Contact
• Jens R. Calamé
• CWI, Amsterdam
• jens.calame_at_cwi.nl