Testing EEE493 2000 See Pressman chapter 18

1
Testing EEE493 2000See Pressman chapter 18
Royal Military College of Canada Electrical and
Computer Engineering

• Major Greg Phillips
• greg.phillips_at_rmc.ca
• 1-613-541-6000 ext. 6190

Dr. Scott Knight knight-s_at_rmc.ca 1-613-541-6000
ext. 6190
2
Testing in the software lifecycle
System requirements
Software requirements
Analysis
Design
Coding
Imagine this process laid out on a calendar.
Think as a software manager. What is the problem
with this view of testing?
Testing
Maintenance
3
Testing Strategies

• specify product requirements before testing
  commences
• state testing objectives explicitly
• understand the users of the software and develop
  profiles for each user category
• this allows testing to focus on actual use
• emphasize rapid cycle testing
• build software that tests itself (antibugging)
• use formal technical reviews prior to testing
• use formal technical reviews on the test strategy
  and the test cases themselves
• develop a continuous improvement approach for the
  testing process

4
Testing and Inspections
Interaction Faults
Execution Faults
Static faults
Faults
Inspection
Unit Test
Integration Test

• Recall
• inspections are most cost effective way to
  remove faults
• inspections seldom remove all faults

5
User testing

• early testing of prototype systems
• designed to
• elicit or improve understanding of product
  requirements
• validate user interface designs
• find optimal user interface designs
• must be planned, structured, managed
• data must be analyzed
• must occur early enough in the process that the
  results can be incorporated into the product

6
No user testing on this ballot

• may determine the outcome of the US election
• 19000 discarded ballots because holes 2 and 3
  were both punched
• statistically, 4000 votes for Buchanan that were
  probably intended for Gore

7
Unit testing

• testing on an individual module, class, unit
• based on the module specification
• normally performed by the developer responsible
• tests should include
• data flow across module interface
• local data structures and access to global data
  structures
• selective coverage of execution paths
• error handling code
• boundary tests
• first and last elements, first and last
  iterations, etc.

8
Building Unit Tests

• Pressman says
• After source level code has been developed,
  reviewed, and verified for correspondence to
  component-level design, unit test case design
  begins.
• Beck says
• Any program feature without an automated test
  simply doesnt exist.
• Unit tests frequently require
• stubs code that simulates other modules that are
  not yet available
• drivers code that sets up the test, produces
  input data, tests output data for completeness
• Stubs and drivers are overhead using them is an
  economic decision

9
Smoke Testing

• What is a smoke test?
• A series of tests designed to expose errors that
  will keep the build from properly performing its
  function
• The intent should be to uncover show stopper
  errors that have the highest likelihood of
  throwing the project behind schedule
• The smoke test should exercise the entire system
  from end to end (does not have to be exhaustive
  but should expose major problems)
• The smoke test should be thorough enough that if
  the build passes, you can assume it is stable
  enough to be tested more thoroughly
• By analogy to testing electrical circuits plug
  it in and see if it smokes.

10
Integration Testing

• may be performed
• top down
• test the top module using stubs to represent
  lower modules
• as lower modules become available, stubs are
  removed
• lower modules may be added depth-first or
  breadth-first
• bottom up
• test the bottom modules using drivers to
  represent higher modules
• collect modules from the bottom up into larger
  and larger clusters
• in some combination (e.g., sandwich
  integration)
• eventually culminates in a completely assembled
  system
• should be started as early in the process as
  possible (big bang integration is a disaster)

11
Regression Testing

• any time changes are made at any level, all
  previous testing must be considered invalid
• can do regression testing at unit, integration,
  and system level
• this means tests must be re-run to ensure the
  software still passes
• re-running previous tests is regression testing
• to be practical, requires automated test suites
• particularly problematic for user interface
  software

12
Stress Testing

• most systems have requirements for performance
  under load, e.g.,
• 100 hits per second
• 500 ambulances dispatched per day
• all data processed using 70 processor capacity
  while operating in flight mode with all sensors
  live
• systems with load requirements must be tested
  under load
• simulated load scenarios must be designed and
  supported
• frequently requires significant test code and
  equipment to adequately support

13
Acceptance or Validation Testing

• complete when the software performs in a manner
  that can reasonably be expected by the customer
• who gets to decide whats reasonable?
• validation tests demonstrate conformity with
  requirements
• failures are recorded on deficiency lists and
  must ultimately be addressed
• normally must include a configuration review
• where requirements are unclear or the customer
  base is broad, may use alpha and beta testing
• exposes incomplete software to the customer
• customers may be willing to use and report faults
  in exchange for early access to capabilities

14
System Testing

• software is only one element of the system. Also
  includes, people, data, procedures, documentation
  and hardware
• ultimately, the entire system must be tested for
  effectiveness
• particular concerns include
• recovery testing
• recovering from system failures
• security testing
• verifying that protection mechanisms will protect
  against improper penetration

15
Review

• inspect first!
• user testing
• unit testing
• smoke testing
• integration testing
• regression testing
• stress testing
• acceptance or validation testing
• system testing

16
Next ClassProcess Metrics