The Software Process

1
CHAPTER 6
TESTING
2
Overview

• Quality issues
• Nonexecution-based testing
• Execution-based testing
• What should be tested?
• Testing versus correctness proofs
• Who should perform execution-based testing?
• When testing stops?

3
Testing

• Testing is an integral component of software
  process and an activity that must be carried out
  throughout the life cycle
• Two types of testing
• Execution-based testing
• Nonexecution-based testing

4
Testing (contd)

• V V
• Verification
• Determine if the phase was completed correctly
• Performed at the end of each phase
• Validation
• Determine if the product as a whole satisfies its
  requirements
• Performed just before the product is delivered to
  the client
• In this book, testing simply denotes V V

5
Software Quality

• Quality implies excellence of some sort
• In S/W, we do not strive for excellence but
  merely getting the software to function correctly
  is enough
• That is, does the software satisfy its
  specifications?
• Read Just In Case You Wanted to Know on page 138
• Software Quality Assurance (SQA)
• Ensures that the product is correct
• Performs testing at the end of each phase as well
  as at the end of the product development
• Managerial independence is important
• Development group
• SQA group

6
Nonexecution-Based Testing

• Underlying principles
• One should not review his or her own work
• Why not?
• Nonexecution-based testing is usually referred to
  as review
• There are two types of reviews
• Walkthroughs
• Inspections

7
Walkthroughs

• Should consist of 4-6 members
• For example, a specification walkthrough should
  include
• Specification team manager member
• Client representative
• Member that will perform the next phase (design
  team member)
• SQA team member (should chair the walkthrough)
• The material to be reviewed should be distributed
  in advance to the participants
• Each reviewer should study the material and
  prepare two lists
• items that the reviewer does not understand
• items that the reviewer believes incorrect

8
Walkthroughs (contd)

• Walkthrough process
• Usually no more than 2 hours
• Detect and record faults DO NOT correct
• Two ways of conducting walkthroughs
• Participant driven
• Participants present their lists of unclear and
  incorrect items
• The author must respond to each query
• Document driven
• The author walks the participants through the
  document
• The reviewers interrupt whenever unclear or
  incorrect items are presented
• The document-driven approach is usually more
  thorough (i.e., finds more faults)

9
Inspections

• More detailed than walkthrough and has five
  formal steps overview, preparation, inspection,
  rework and follow-up
• Inspection team should consist of 4 members
• For example, design inspection team includes
• Moderator (inspection team leader), designer,
  implementer, tester

10
Inspection Steps

• Overview
• An overview of the document to be reviewed is
  given by one of the authors
• At the end of the overview session, the document
  is distributed to the participants
• Preparation
• Participants try to understand the document in
  detail, aided by statistics of fault types
• Participants prepare the lists of
  unclear/incorrect items as well
• Inspection
• An author walks through the document with the
  inspection team
• Faults are detected and recorded DO NOT correct
  here
• Within one day, the moderator generates a written
  report containing faults detected during
  inspection

11
Inspection Steps (contd)

• Rework
• The author resolves all faults and problems noted
  in the written report
• Follow-up
• The rework is thoroughly checked
• All fixes must be checked to ensure that no new
  faults have been introduced

12
Statistics on Inspections

• 82 of all detected faults (IBM, 1976)
• 70 of all detected faults (IBM, 1978)
• 93 of all detected faults (IBM, 1986)
• 90 decrease in cost of detecting fault
  (Switching system, 1986)
• 4 major faults, 14 minor faults per 2 hours (JPL,
  1990). Savings of 25,000 per inspection
• Number of faults decreased exponentially by phase
  (JPL, 1992)

13
Metrics for Inspections

• Fault density (e.g., faults per KLOC)
• Fault detection rate (e.g., faults detected per
  hour)
• Fault detection efficiency (e.g., the number of
  major and minor faults detected per person-hour)

14
Execution-Based Testing

• Definitions
• Fault is the IEEE Standard terminology for bug
• Failure is the observed incorrect behavior of the
  product as a consequence of the fault
• Error is the mistake made by programmer
• Programming testing can be a very effective way
  to show the presence of bugs, but it is
  hopelessly inadequate for showing their absence
  Dijkstra, 1972

15
What is execution-based testing?

• Execution-based testing is a process of
  inferring certain behavioral properties of a
  product based, in part, on the results of
  executing the product in known environment with
  selected inputs. Goodenough, 1979
• Inference (i.e., deriving logical conclusion)
• Known environment
• Selected inputs
• What must be tested? gt Behavioral properties
  must be tested
• Correctness, utility, reliability, robustness,
  and performance

16
Utility

• Utility is the extent to which a users needs are
  met when a correct product is used under
  conditions permitted by its specifications
• That is, does it meet users needs?
• Ease of use
• Useful functions
• Cost-effectiveness

17
Reliability

• Reliability is a measure of the frequency and
  criticality of product failure
• How often the product fails? (Mean time between
  failures, MTBF)
• How long it takes to restore? (Mean time to
  restore, MTTR)
• But often more important is how long it takes to
  repair the results of the failure?

18
Robustness

• Robustness is a function of a number of factors
  such as
• Range of operating conditions
• Possibility of unacceptable results with valid
  input
• Effect of invalid input

19
Performance

• Extent to which space and time constraints are
  met
• Real-time systems have hard time constraints

20
Correctness

• A product is correct if it satisfies its output
  specifications
• But what if the specifications themselves are
  incorrect?

21
Correctness of specifications

• Incorrect specification for a sort
• Function trickSort which satisfies this
  specification

22
Correctness of specifications (contd)

• Incorrect specification for a sort
• Corrected specification for the sort

23
Correctness

• NOT necessary
• NOT sufficient

24
Correctness Proofs

• Alternative to execution-based testing

25
Example of Correctness Proof

• Code segment to be proven correct

26
Example of Correctness Proof (contd)

• Flowchart of code segment

27
Example of Correctness Proof
28
Correctness Proof Case Study

• Never prove a program correct without testing it
  as well

29
Episode 1

• 1969 Naur Paper
• Naur text-processing problem
• Given a text consisting of words separated by
  blank or by nl (new line) characters, convert it
  to line-by-line form in accordance with following
  rules
• (1) line breaks must be made only where given
  text has blank or nl
• (2) each line is filled as far as possible, as
  long as
• (3) no line will contain more than maxpos
  characters
• Naur constructed a procedure (25 lines of Algol
  60), and informally proved its correctness

30
Episode 2

• 1970 Reviewer in Computing Reviews
• The first word of the first line is preceded by
  blank unless the first word is exactly maxpos
  characters long

31
Episode 3

• 1971 London finds 3 more faults
• Including
• The procedure does not terminate unless a word
  longer than maxpos characters is encountered

32
Episode 4

• 1975 Goodenough and Gerhart find three further
  faults
• Including
• The last word will not be output unless it is
  followed by blank or nl

33
Proofs and Software Engineering

• Lesson
• Even if product is proved correct, it must STILL
  be tested.

34
Three Myths

• Software engineers do not have enough math for
  proofs
• Proving is too expensive to be practical
• Proving is too hard

35
Proofs and Software Engineering (contd)

• Can we trust a theorem prover?
• How to find inputoutput specifications, loop
  invariants
• What if the specifications are wrong?
• Can never be sure that specifications or a
  verification system are correct

36
Proofs and Software Engineering (contd)

• Correctness proofs are a vital software
  engineering tool, WHERE APPROPRIATE.
• If
• Human lives are at stake
• Indicated by cost/benefit analysis
• Risk of not proving is too great
• Also, informal proofs can improve the quality of
  the product
• Assert statement

37
Who Performs Execution-Based Testing?

• Testing is destructive
• A successful test finds a fault
• Solution
• 1. The programmer does informal testing
• 2. SQA does systematic testing
• 3. The programmer debugs the module
• All test cases must be
• Planned beforehand, including expected output
• Retained afterwards

38
When Testing Can Stop?

• Only when the product has been irrevocably retired