Software Testing and Reliability Software Test Process

1
Software Testing and ReliabilitySoftware Test
Process

• Aditya P. Mathur
• Purdue University
• August 12-16
• _at_ Guidant Corporation
• Minneapolis/St Paul, MN

Last update August 9, 2002
2
Learning Objectives

• What is software life cycle?

• How to mesh various types of testing into the
  development cycle?

• What documents are generated for and as a result
  of testing?

• What test-metrics needs to be collected and why?

3
References

• Software Testing in the Real World, Edward Kit,
  Addison Wesley, 1995.

4
Life Cycle

• Software Development Cycle is a sequence of steps
  in which a software system is developed. We shall
  refer to this sequence as, simply, Life Cycle.

• There are many models of Life Cycle.

• The waterfall model

• Iterative refinement model

• Incremental development model

• Spiral model

• Unified Process model

5
Testing and the Life Cycle

• It is widely believed that testing is necessary
  in each phase of the life cycle regardless of
  which model is adopted.

• The key questions with which most process
  managers struggle are

• How should the test process be meshed with the
  development cycle?

• What tools and techniques should be used in each
  phase?

• What data should be collected to evaluate the
  effectiveness of the test process?

6
Meshing the Test Process with the Life Cycle
Run regression Tests
Test the requirements
Test for acceptance
Test the design
Test the modules
Test the subsystems
Test the system
7
Test Tools and Techniques Requirements Phase

• Inspection and walkthroughs Ensure completeness,
  unambiguity, precision, consistency, testability,
  and traceability. Rational Rose and Analyst
  Studio, I-Logix Rhapsody

• Simulation Use this to improve the understanding
  of the system to be built where possible, work
  with the customer.

• Generate test procedures, plans, and test cases
  Develop procedures, plans and test cases to be
  used for testing each requirement. Identify what
  tools will be used in each phase, who will use
  these, and what data will be collected. Determine
  product release criteria.

8
Test Tools and Techniques Design Phase

• Inspection and walkthroughs Rational Rose

• Simulation Check for the correctness of
  scenarios. I-Logix Rhapsody

• Refine test procedures and test cases Use the
  artifacts created during design to generate
  additional test cases and refine or correct the
  existing ones.

• Risk analysis Use the application architecture
  to perform risk analysis. Rational ROOM

9
Test Tools and Techniques Coding Phase

• Inspection and walkthroughs Ensure that the code
  matches the design. If not then at leant one of
  the two needs to be modified to achieve
  consistency. Rational Rose

• Perform unit testing Test each module
  thoroughly. CleanScape TestWise Rational
  Purify and Visual Test AMC CodeTest

10
Test Tools and Techniques Integration Phase

• Perform integration testing Test each subsystem
  thoroughly. CleanScape TestWise Rational
  Purify and Visual Test Applied Microsystems
  CodeTest Windriver Sniff

11
Test Tools and Techniques System Integration

• Perform system testing Test the system to ensure
  correct functionality with respect to use cases,
  do performance, stress, and robustness tests.
  CMU Ballista Telcordia AETG Applied
  Microsystems CodeTest Windriver Diab RTA suite
  and Sniff

12
Test Tools and Techniques System Changes

• Perform regression testing Test the modified
  system to ensure that existing functions perform
  as intended. Note that system testing for the
  added features ought to be performed.
  CleanScape Testwise for test minimization

13
Test Tools and Techniques In-target system

• Perform system testing Test the application for
  all functionality, its performance, and
  robustness while the application runs embedded in
  the target platform. Advanced Microsystems
  CodeTest Home-grown test generators Rational
  Purify

14
Test Tools and Techniques Usability testing

• Perform usability testing Have a sample of the
  potential users test the application and the
  documents that accompany it. The goal is to
  ensure that (a) the documents are understandable,
  consistent, and correct and (b) the GUI satisfies
  the requirements of the potential users.

Usability testing can begin when the GUI is ready
for use, i.e. developed and tested for
correctness, but the remainder of the application
code might not be.
15
Test Tools and Techniques Security testing

• Perform security and safety testing Security and
  safety testing must be done when an application
  is accessible from the outside and might
  compromise the users privacy and safety,

The objective of security and safety testing is
to demonstrate that privacy and safety cannot be
compromised when the assumptions regarding the
use of the application are met.

• Security and safety testing is done with respect
  to the requirements for security and safety.
  Traditional testing techniques are useful but
  might have to be augmented with techniques
  specific to the application environment.

16
Responsibilities

• Planning (High level) Test team

• Unit tests Developer. (Who checks the developers
  work?)

• Integration tests Developer and/or test team
  depending on the complexity of the sub-system to
  be tested.

• System level tests Test team. These tests will
  generally require communication with the
  developers. This must be a part of the detail
  plan built at the beginning of the development
  cycle.

• Acceptance and usability tests Test team in
  collaboration with the end user (or a sample of
  the end user set).

17
Release Criteria

• Units Code coverage based. Example all
  statements, decisions, and conditions must be
  covered.

• Integration tests Code coverage and function
  based. Low level code coverage might not be
  usable if the integration produces a large
  component.

• System level tests System reliability measured
  in terms of the number and type of failures
  detected.

• Acceptance and usability tests Determined by the
  user primarily based on correct functionality,
  robustness, performance, and usability.

18
Standards for Test Documents

• SQAP Software Quality Assurance Plan,
  IEEE/ANSI, 1989 Std 730-1989.

• SVVP Software Verification and Validation Plan,
  IEEE/ANSI, 1986 Std 1012-1986 one per SQAP.

• VTP Verification Test Plan, one per verification
  activity.

• MVP Master Validation Test Plan, IEEE/ANSI, 1983
  Std 829-1983 one per SVVP.

• DTP Detailed Validation Test Plan, IEEE/ANSI,
  1983 Std 829-1983 one or more per activity.

19
Standards for Test Documents (contd.)

• TDS Test Design Specification, IEEE/ANSI, 1983
  Std 829-1983, one per DTP.

• TCS Test Case Specification IEEE/ANSI, 1983 Std
  829-1983 one or more per TDS/TPS.

• TPS Test Procedure Specification, IEEE/ANSI,
  1983 Std 829-1983 one or more per TDS.

• TC Test Case, one per TCS.

20
Structure of Standards for Test Documents
21
Defect Tracking and Classification

• Defect tracking and classification Applied
  Innovation management Bug/Defect Tracking
  Expert defectx defectX

• Defect estimation Use inspection data for defect
  estimation design metrics past data on similar
  projects.

• Defects open vs. closed

• Cross-project and cross version comparison of the
  number of defects.

• Cost of repairing a defect time and manpower
  resources needed.

22
Why collect data?

• Assists with improving the test process
  correlations between other process related
  metrics, such as code coverage, and defects can
  be established.

• Helps improve product quality.

• Useful in setting up a reward system (no penalty
  please!).

23
Summary

• Software development cycle

• Test process and the software development cycle

• Assignment of responsibilities

• Standards

• Metrics