Testing Adequacy CFICSE VV10 October 1999

1
Testing AdequacyCFICSE VV10 October 1999

• Dr. Terry Shepard
• Royal Military College of Canada
• Electrical and Computer Engineering
• 613-541-6000 ext. 6031

2
Adequacy of Requirements Testing

• Largely a judgement call
• There may be diminishing return, if there is a
  high rate of requirements change
• Need to decide what actions to take to test
  requirements eg.
• how many prototypes to do
• how much usability testing to do
• Testing largely means reviews and inspections

3
Adequacy of Design Testing

• Can involve simulation of critical parts of a
  design
• Even with executable designs (or requirements),
  testing is usually limited because of slow
  execution
• Again, testing largely means reviews and
  inspections

4
Basis for deciding that code testing is complete
varies

• Rate at which failures are being found drops
  below a threshold
• can infer remaining defect count/KLOC from past
  history
• Release date occurs
• Budget or other resources run out
• Regulatory requirement met
• MTBF goal met

5
Adequacy Levels for Defect Counts/KLOC

• 3 to 10 defects/KLOC for commercial software
  BOEH88
• 1 to 3 defects/KLOC for industrial software
  BOEH88
• 0.01 defects/KLOC for NASA shuttle code HEND94

6
Reliability/Certification Testing

• Intended to predict probability of failure, or
  MTBF
• Based on random testing using operational profile
• Stop when an acceptable level of MTBF has been
  reached
• Reliability Growth Models exist, which are
  intended to allow the date at which that will
  happen to be predicted

7
R.W. Butler and G.B. Finelli, The Infeasibility
of Quantifying the Reliability of Life-Critical
Real-Time Software, IEEE TSE 19(1) pp. 3-12 (Jan
93)
8
DO178B MCDC
9
Research on coverage measures with ObjecTime and
Cong Hu
10
References

• BOEH88 Boehm, B.W. and P.N. Papaccio,
  "Understanding and Controlling Software Costs,"
  IEEE Transactions on Software Engineering,
  October 1988, Vol. 12, No. 9, pp. 929-940.
• HEND94 Henderson, Johnnie A., "Software Quality
  Assurance on Onboard Shuttle Software,"
  Proceedings of the Software Technology Conference
  (STC-94), April 1994.