6 November

1
6 November

• Software Quality and Testing (cont)

2
How important is unit test?

• The Voyager bug (sent the probe into the sun).
• 90 The ATT bug that took out 1/3 of US
  telephones (crash on receipt of crash notice).
• The DCS bug that took out the other 1/3 a few
  months later.
• 93 The Intel Pentium chip bug (it was software,
  not hardware).
• 96 The Ariane V bug auto-destruct (data
  conversion).

3
What are you trying to test?

• Most common actions?
• Most likely problem areas?
• Risk-based testing

4
Risks

• Identify criteria of concern availability,
  quality, performance,
• Risk of it not being met
• likelihood
• consequences
• If Im testing code for a grocery store, what is
  the impact of the code not being highly
  available?

5
How to identify what to test

• New features
• New technology
• Overworked developers
• Regression
• Dependencies
• Complexity
• Bug history
• Language specific bugs
• Environment changes

• Late changes
• Slipped in pet features
• Ambiguity
• Changing requirements
• Bad publicity
• Liability
• Learning curve
• Criticality
• Popularity

6
Four Parts of Testing

• Model
• Select test cases
• Execute test cases
• Measure

7
Basic Software Model
User interfaces APIs Operating system Files
environment
capabilities
Input Output Storage Processing
8
Test Case Selection

• Environments
• What happens if a file changes out from under
  you?
• Consider all error cases from system calls (e.g.,
  you cant get memory)
• Test on different platforms (software and
  hardware)
• Test on different versions and with different
  languages
• Capabilities
• Inputs
• Outputs
• States
• Processing

9
From the User Interface Inputs

• Error messages
• Default values
• Character sets and data types
• Overflow input buffers
• Input interactions
• Repeated inputs
• How easy is it to find bugs in Word?

10
From the User Interface Outputs

• Concept What inputs create interesting outputs?
• REQUIRES DOMAIN EXPERTISE
• (Example calendar invalid leap day)
• State-based outputs
• Invalid outputs
• Changeable outputs
• Screen refreshes

11
Bubble diagrams Reverse state exploration

• Define a failure state
• What would have happened to get you there?
• Repeat
• Find a way to force it down that path
• Lets try it remote access keys

12
Processing

• Same input, different initial conditions
• Too many or too few values in a data structure
• Alternative ways to modify constraints
• Invalid operator/operand combinations
• Recursive functions
• Overflow/underflow computations
• Feature interaction

13
How to select specific cases

• Data based
• Boundary conditions
• Equivalence classes
• Control based
• State transitions

14
Questions to Ask for Each Test

• How will this test find a defect?
• What kind of defect?
• How powerful is this test against that type of
  defect? Are there more powerful ones?

15
References

• Whittaker, How to Break Software
• Kaner, The Impossibility of Complete Testing at
  www.kaner.com (Articles)

16
Execution tools

• Automation often primarily scripts
• Critical for regression testing
• GUI tools are abundant, but marginal

17
Test Plan

• What needs to be tested
• Functions
• Environments
• Who will test it
• When to test