Black-Box Test Reduction Using Input-Output Analysis

1
Black-Box Test Reduction Using Input-Output
Analysis
ISSTA 00 Patrick J. Schroeder, Bogdan
Korel Department of Computer Science Illinois
Institute of Technology Chicago, IL USA
2
Presentation Overview
1) Motivation 2) Black-Box Testing 3)
Input-Output Analysis 4) Determining Input-Output
Relationships 5) Conclusion
3
Motivation for Research

• Experience in Industry as a black-box tester of
  large, complex, telecommunications systems
• Root cause analysis of customer-found faults
  often discovered ineffective or redundant
  black-box tests
• Additional information from the software
  implementation often improves black-box testing
  (after the initial release!)

4
Black-Box Testing

• Black-box testing is testing from a functional or
  behavioral perspective to ensure a program meets
  its specification
• Testing usually conducted without knowledge of
  software implementation - system treated as a
  black box
• Black-box test design techniques include
  equivalence partitioning, boundary value
  analysis, cause-effect graphing, random testing

5
Observations on Black-Box Testing

• Testing from the black-box perspective generates
  a large number of tests (e.g. ATT Definity PBX
  gt 3 million tests)
• All testers are faced with resource limitations
  the number of tests must be reduced
• Test reduction done randomly or based on
  engineering judgement may lead to ineffective
  or redundant tests

6
Combinatorial Testing

• Combinatorial testing For programs with
  multiple inputs, one must decide how to test
  different combinations of selected test data
  values
• The most thorough approach is to test all
  combinations - this is quite often impossible due
  to the large number of tests
• However, not to consider testing different input
  combinations could lead to an unacceptable number
  of undetected software faults

7
Combinatorial Explosion
If n 10 and 10 test data values are selected
for each variable, the total number of
combinatorial test cases is 1010
8
Current Approaches

• Testers may try to identify important
  combinations to test using specifications,
  historical fault data, and engineering judgement
• Orthogonal arrays, or other combinatorial design
  techniques, may be used to tests pair-wise or
  n-way combinations of inputs
• Difficult to determine impact on fault-detection
  capability of the test set using these techniques

9
Definition of the Problem

• In combinatorial testing, black-box testers face
  the situation where the number of possible test
  cases far exceeds the time and resources
  available to execute them
• How can we reduce the number of combinatorial
  tests while maintaining the fault-detection
  capability of combinatorial testing?

10
Proposed Technique

• Observations
• Current techniques do not consider program output
  variables
• For a class of programs, not all program inputs
  affect all program outputs
• We propose the use of Input-Output (IO) Analysis
  to determine which program inputs affect program
  outputs
• This information can be used to reduce the number
  of combinatorial tests while maintaining the
  fault-detection capability of the test
  set

11
Simple Example using IO Analysis
Input data selected using equivalence
partitioning
A1, 3 BN,S,E,W CTDC, BDM
W Z
Total Number of Combinatorial Tests 2 4 2
16
12
IO Analysis Applied
A1, 3 BN,S,E,W CTDC, BDM
W Z
13
IO Analysis Applied
A1, 3 BN,S,E,W CTDC, BDM
T(W) ? T(Z) (1, N, TDC) (1, N, BDM) (3, N,
TDC) (3, N, BDM) (1, S, TDC) (1, E, TDC) (1, W,
TDC)
W Z
T(W) (1, N, TDC) (1, N, BDM) (3, N,
TDC) (3, N, BDM)
T(Z) (1, N, TDC) (1, S, TDC) (1, E,
TDC) (1, W, TDC)
14
Optimization Problem

• Create the smallest possible test set which
  covers all data combinations in the individual
  output test sets
• Test set optimized using Dont Care (DC)
  conditions

Toptimized (1, N, TDC) (1, S, BDM) (3, E,
TDC) (3, W, BDM)
T(W) (1, DC, TDC) (1, DC, BDM) (3, DC,
TDC) (3, DC, BDM)
T(Z) (DC, N, DC) (DC, S, DC) (DC, E,
DC) (DC, W, DC)
15
Determining Input-Output Relationships

• Information may be gathered in a variety of ways
  (e.g. from specifications, code)
• Manual collection of information difficult and
  time consuming
• Automated collection possible through
• Static Analysis
• Dynamic Analysis
• Execution-oriented Analysis

16
Automated Analysis Techniques

• Static techniques analyze a programs source code
  - static program dependencies used to determine
  IO relationships
• Dynamic techniques analyze a programs execution
  trace across a set of test data - dynamic
  dependencies used to determine IO relationships
• Execution-oriented techniques manipulate
  individual inputs across multiple program
  executions in a test harness to determine IO
  relationships

17
Technique Advantages/Disadvantages

• Static techniques reliably determine the absence
  of relationships between inputs and outputs
  however, they may identify relationships that are
  not actually present
• Dynamic analysis and execution-oriented analysis
  reliably determine the presence of input-output
  relationships however, they cannot guarantee
  that all relationships will be detected
• Dynamic analysis and execution-oriented analysis
  may be used when static analysis, perhaps in
  over-estimation, indicates that all program
  inputs affect all program outputs

18
Experimental Study Results
1) Warehouse Management System Combinatorial
Tests 3,125 Using IO Analysis
40 2) DACS Configuration
System Combinatorial Tests 22,500 Using IO
Analysis 158 3) Liquidity Ratio
Spreadsheet Combinatorial Tests 1.95
Million Using IO Analysis 625
19
Conclusion

• For the class of program and type of problem
  defined, input-output analysis can reduce the
  number of combinatorial tests while maintaining
  the fault-detection capability of the test set
• Research continues on identifying program types
  and testing situations where our approach is
  applicable
• Additional experimentation with
  execution-oriented analysis to provide program
  dependencies, and other information useful in
  black-box testing, is underway