Test-Suite Reduction for Model Based Tests: Effects on Test Quality and Implications for Testing

1
Test-Suite Reduction for Model Based
TestsEffects on Test Quality and Implications
for Testing

• Written by Heimdahl and DevarajDept. of Comput.
  Sci. Eng., Minnesota Univ.(19th International
  Conferenceon Automated Software Engineering
  2004)
• Presented by Jae Ho Jung
• (Some slides from George Devaraj)

2
Test-suite Reduction

• Naive test set generationresults in lots of
  redundanttest cases
• Cost of maintaining andrunning large test sets
  is high
• Test suite reduction techniques based on
  structural coverage help reduce these costs
• Drawback This mightaffect the quality of test
  sets

What is the effect of test reduction on the fault
finding ability of test sets?
3
Previous Empirical Studies

• Wong et.al 1994
• Test set minimization keeping coverage constant
  results inlittle or no reduction in its fault
  detection effectiveness
• Rothermel et.al 1997
• Fault finding capability was significantly
  compromised on test-set reduction
• Same results were also observed by Jones and
  Harrold 2003
• How is this papers work different?
• Focusing on testing of formal specifications
• Addressing a wider range of test adequacy criteria

4
Model-Based Development
This paper focusedon test-suites for Conformance
testing
Specification Centered Development Process
5
Specification based test case generation using
model checkers
Test sequence generation architecture using model
checker (cited from Auto-generating test
sequences using model checkers A case study)
? RSML-e is based on the Statecharts like
language Requirements State Machine Language
(RSML). RSML-e is a fully formal and
synchronous data-flow language without any
internal broadcast events (the absence of
events is indicated by the -e).
6
Hypothesis

• Hypothesis 1
• Hypothesis 2

Test reduction of a naively generatedspecificatio
n based test-set can producesignificant savings
in terms of test-set size
Test reduction will adversely affect the
fault-finding capability of the resulting test set
7
Case ExampleFGS (Flight Guidance System)
8
Verification Framework
NIMBUS tools framework
9
The Experimental Setup

• Experimental Subject
• FGS Model expressed in the RSML-e formalism
• Fault Injection
• Creation of mutant FGS specifications
• Full Test-set generation for six test coverage
  criteria
• Reduced Test-set generation
• Simple greedy based heuristic algorithm used
• Fault Detection
• Running reduced as well as full versions against
  the mutant specs

• Fault Classes
• Variable Replacement (VR)
• Condition Insertion (CI)
• Condition Removal (CR)
• Condition Negation (CN)

• Test Coverage Criteria
• Variable Domain Coverage
• Transition Coverage
• Decision Coverage
• Decision Coverage withSingle Uses
• Modified Condition andDecision Coverage (MC/DC)
• MC/DC with Single Uses

10
Algorithm for test-suite reduction
Reduced Set O ? F Greedy choice Add a
testcase to O which improvesthe existing
coverage
11
Modified Condition/Decision Coverage
S1 a decision statement A,B,C three conditions
The truth vector FFF is not a truth value for
C(F) because it is not part of MC/DC pair for C.
FTT and TFF does not satisfy the MC/DC
Criterion for C
ltPartial Program Pgt

• MC/DC is a stricter form of decision coverage
• MC/DC require execution coverage at the condition
  level
• MC/DC requires that each condition in a decision
  be shown by execution to independently affect the
  outcome of the decision (MC/DC requires that
  MC/DC pairs be covered for each condition)

FFT FFF
ltTruth value, Truth vector, and MC/DC pairgt
12
Experimental Results and AnalysisFull Test Set
Test Size VR CN CI CR Total
Random 100 21 25 5 15 66 (92)
Variable Domain 115 14 15 2 4 32 (44)
Transition 313 20 24 5 15 64 (89)
Decision 435 23 24 5 15 67 (93)
Decision Usage 478 23 24 7 15 69 (96)
MCDC 537 22 25 7 16 70 (97)
MCDC Usage 334 23 25 8 16 72 (100)
Full test set generation for various criteria
along withtheir fault detection capability
13
Experimental Results and AnalysisTest-Suite
reduction
Criteria Full Set Run 1 Run 2 Run 3 Run 4 Run 5 AVG Reduction
VariableDomain 115 19 22 18 21 21 20.2 82
Transition 313 35 43 29 38 43 37.6 88
Decision 435 45 44 44 45 42 44.0 90
DecisionUsage 478 37 43 47 43 38 41.6 91
MCDC 537 34 33 29 34 32 32.4 94
MCDCUsage 334 30 30 33 32 33 31.6 91
Reduced test set sizes for various test reduction
runs
14
Experimental Results and AnalysisEffect on
Fault Finding Capability
Criteria Full Set Run 1 Run 2 Run 3 Run 4 Run 5 AVG Reduction
VariableDomain 32 28 29 25 28 25 27.0 15.6
Transition 64 58 58 58 59 57 58.0 9.38
Decision 67 62 61 62 62 61 61.6 8.06
DecisionUsage 69 62 63 63 62 63 62.6 9.28
MCDC 70 64 63 63 63 63 63.2 9.71
MCDCUsage 72 67 66 67 67 67 66.8 7.22
Fault finding capability of the reduced test-sets
15
Threats to Validity

• Using only one instance for investigations
• Nevertheless useful because it was performed on a
  real world case example
• Using seeded faults in experiment (not actual
  faults)
• Using a single fault per specification
• In practice, specifications may have more complex
  error patterns
• However, single fault specifications more easier
  to run and control than multiple fault
  specifications

16
Future Works

• Further experimentation needed
• More experimental instances and multiple fault
  specifications
• More robust coverage metrics needed
• Skeptical towards the use of test-suite reduction
  techniques in the formal model based domain
• More works are needed to determine how to
  prioritize test cases and empirically evaluate
• Test-case prioritization would be a better
  approach than test-suite reduction or minimization

17
Questions

• What makes the conflicting evidence in the
  previous empirical studies on this issue (the
  effect on the fault finding capability of the
  reduced test-suites)?
• Considering test-suite prioritization algorithms.
  What could be a attribute of that algorithms?

18
Conclusions

• Reduction of test-suite size has an unacceptable
  effect on the suites fault finding capability -
  test-suite reduction may not be an effective
  means of reducing testing time
• More rigorous criteria, such as MC/DC, provide a
  better fault finding capability than less
  rigorous criteria, such as variable domain and
  transition coverage