Unit Testing

1
Unit Testing

• OMSE 535 Lecture 7
• Manny Gatlin

2
Overview

• Unit Testing Characteristics
• Unit Test Planning
• Unit Test Case Design
• Path Reliability
• Coverage Criteria
• Functional (specification-based)
• Statement/branch/path/dataflow
• Mutation
• Special-values

3
What Is Unit Testing?

• Unit Testing
• A test designed to demonstrate that a given unit
  or component performs correctly
• Although many testing techniques apply at all
  levels, they are easiest to understand, to apply,
  and to use tool support at the unit level

4
Why Unit Test?

• Unit testing tests each component in isolation
  from the system
• Focus is on correct functionality of unit
• Detect many failures before integration
• Usually catches more obvious bugs
• Usually done by developer, if done at all

5
When to Unit Test

• Unit Test Planning should occur before coding
• Not typically done formally
• Unit testing usually unstructured, ad hoc
• Unit Testing should occur right after unit coding
  
• Rework based on test results should also be unit
  tested

6
Unit Test Planning

• Unit Test Plan contents (basic)
• Unit walkthrough (static test)
• Inspection a recommended method
• Unit test cases (dynamic test)
• Initial conditions for each unit test
• Input test data for each unit test
• Expected results for each unit test
• Actual results and pass/fail status for each unit
  test

7
Unit Test Case Design

• Techniques
• Black-box analysis of specification
• White-box analysis of program structure
• Unit Testing considerations
• Environmental limits (e.g., memory)
• State transitions that change behavior
• Historical defects (same/similar units)
• Error handling

8
Black-box Techniques

• Specification-based Path Analysis
• Control-flow testing (Beizer)
• Logic coverage (Myers)
• Functional Analysis
• Functions/output/input analysis
• Boundary Value Analysis

9
White-box Techniques

• Statement Coverage
• All source statements are executed once
• Path Coverage
• Every branch has been exercised once
• Dead code analysis

10
Unit Test Drivers

• Two styles of test driver (harness)
• Interactive drivers
• Reflect internal functions out to the
  keyboard/screen
• Can be tedious
• Scripts can help drive interactive testing
• Programmed drivers
• Use code instrumentation to drive
• Use exception reporting for feedback

11
Category-partition

• Functional units
• Categories
• Choices
• Test specification for each functional unit
• One value from each choice from each category
• Constraints
• Annotate choices with properties constraints
• error single

12
Path Reliability

• A testset T is reliable for P, specification F
  iff
• For all x in T, P(x) F(x) ? P F
• A testing strategy H is a set of subdomains Ti
• An H-testset is a vector (t1, t2, ...) with ti in
  Ti
• H is reliable for P iff
• P F, or there is an i such that for all x in
  Ti, P(x) F(x)
• P is the function that program P computes
• Caution The definition of subdomains is
  arbitrary
• Plausible subdomains do not contain only success
  points

13
Reliability Examples
Example 2 main() / find x in array / int
x,N,A51,2,3,4,5 scanf("d", x) for (N1
Nlt5 N) if (AN x) printf("OK")
exit prinf("Not OK")
Example 1 main() / compute F(x) 2x
/ float x scanf("d", x) printf("d", xx)

• Example 1
• Input 2 executes one path but does not fail
• Path testing thus not reliable for the fault
• Example 2
• The path that includes loop index 5 will get
  the bounds violation
• Path testing is reliable for this failure

14
Coverage Criteria

• Questions to ask
• How can it fail to find failures?
• Revealing examples
• Is it a true partition?
• What's left out?
• Overkill on overlap?
• How hard is it to generate test cases?

15
Statement Coverage

• Testset must cause every statement to be executed
  
• Branch to be taken both ways
• Many variants of "condition coverage"
• Overlapping subdomains
• Easy to understand generate tests
• Infeasible cases are common

16
Testing With a Coverage Tool

• "Haphazard" method
• Any old data to start
• Analyze program to extend coverage
• Adequacy method (Weyuker, Marick)
• Start with test-plan data (mostly functional)
• Analyze requirements, design, to extend coverage

17
Full Path Coverage

• There is a potential infinity of paths
• Every conditional multiplies the number by 2
• Loops introduce unlimited paths
• Lots of infeasible cases
• Meaning of many paths not obvious

18
Infeasible Path Test Example

• Example
• of basis paths EdgesNodes2
• 9-72 4
• Assume upper limit on iteration is 10
• Number of exhaustive tests
• 310 39 31 30
• 90,000 test cases

19
Dataflow Testing

• Intuition interesting paths are from value
  creation to value use
• "D-U" associations
• Def X is a place where X gets a value
• Use X is a place where a value of X is used
• A du pair for X is a def X connected to a use X
  by at least one path on which no def X occurs

20
Dataflow

• Formalism is surprisingly difficult
• Treating nodes edges, c-uses p-uses, was
  probably a mistake
• It's hard to talk about paths and parts of paths
• "Applicable" isn't quite accurate
• The "subsumes" comparison may be of academic
  interest only

21
More Dataflow

• Dataflow anomalies
• (no d)-u
• d-d
• All uses criterion
• some path from each d to u covered by testset
• All chains criterion
• du(P), du(Q), ... , du(Y), du(Z) where the def
  Q involves a use P, ..., the def Z involves a use
  Y

22
Coverage Tools

• Statement/branch coverage
• Use an existing profiler
• Pre-processor techniques work well
• No more than linear performance degradation
• Dataflow coverage
• Requires parsing the source
• Graph-traversal algorithms
• Exponential performance degradation potential

23
Research Tools -gt Practice?

• Statement-coverage tool (e.g., GNU gcov)
• "Statement 137 not executed"
• result highlighted program listing
• Dataflow-coverage tool (e.g., ASSET)
• "du association for variable X not covered, Path
  1-3-4-5-7-13-14-206 would do so"
• result draw and color flowgraph
• Issues for tool selection
• Languages supported
• Graphics
• Handling of complexity

24
Mutation Testing

• A "data coverage" (as opposed to control-flow
  coverage) idea
• For each control point in a program, does enough
  test data arrive there?
• "Enough data" must be defined by trying variant
  code
• Coverage failure means
• If the program should have been the mutant, this
  data won't find a failure

25
"Weak" Mutation Coverage

• Specific local fault targeted
• Change program, then
• Local state must change on the testset
• Adequate testsets can be deduced for some faults
• Wrong variable assigned try an assigned value
  different from all current values
• wrong comparison operator try , lt, gt values

26
ESTCA

• Error Sensitive Test Cases Analysis
• Analogy to hardware testing for "open"/"stuck-at"
  faults
• Today's terminology "fault-based" testing

27
Assessment of Mutation

• Almost no use in practice
• Too many hard-to-understand messages
• Research tools only
• Compiler-based UNIVAC 1108 language (1976)
• Interpretive FORTRAN system (1978)
• Modern C system (c. 1995)

28
Conclusions

• Advantages to testing small units
• Easier to hang onto intellectual control
• Easier to uncover failures early
• Structural coverage is better for adequacy than
  test generation
• Generate functional tests from unit specs
• Use a statement/branch/loop analyzer