Software Testing 101

1
Software Testing 101

• By Rick Clements
• cle_at_cypress.com

2
Overview

• Terminology
• Requirements
• Configuration control
• Test plan
• Test cases
• Test procedures
• Bug tracking ship decision
• After the testing

3
QA vs. QC
Terminology

• Quality assurance (QA) - The management of
  quality. The encouragement of best practices.
  QA is about preventing defects.
• Quality control (QC) - Validation and
  verification of the product. This includes
  reviews, inspections and testing.

4
Test Documents
Terminology

• Test plan (or validation plan) - The plan
  describing the general approach, the people
  required, equipment required and schedule for
  testing and other validation activities
• Test cases - Specific data used to test the
  product.
• Test procedures - The procedures to follow in
  testing the product. These may be manual or
  automated.

5
Types of Tests
Terminology

• Black box testing - Testing by looking at the
  requirements to develop test cases. System level
  testing is often black box testing.
• White box testing - Testing by looking at the
  structure of the program to develop test cases.
  White box testing often occurs in unit testing.

6
Levels of Testing
Terminology

• Unit testing - The process of running small parts
  of the software. The design teams often handle
  unit testing.
• Integration testing or Interface testing - The
  testing of pre-tested modules and hardware to
  determine that they work together.
• System testing - The testing of the entire
  product to see that it meets its requirements.
  This occurs after the product has been
  integrated.

7
Software Development Process
Software Implementation Debug
Software Design
Define Requirements
Software Test
Test Implementation Debug
Test Design
Software Release
8
Requirements

• Why are they important?
• How are they documented?
• Whats important?
• What if you dont have any requirements?

9
Why Are Requirements Important?
Requirements

• How do the designers know what to build?
• How can you test that it was built correctly?
• When you disagree, whos right?

10
Documenting Requirements
Requirements

• Data sheet
• Requirements specification
• Functional specification
• Data base
• Traceability matrix

11
Whats Important
Requirements

• They exist
• They are unambiguous and testable
• Cover all of the customers (not just final
  customer)
• They are under configuration control

12
No Documented Requirements
Requirements

• Ask the project manager
• Ask the marketing representative
• Has anything been sent to the customer?
• Ask a domain expert
• What are the designers building?
• Write them down

13
Configuration Control

• Why is it a testing issue?
• What to track
• Build Version Number

14
Why Is It A Testing Issue?
Configuration Control

• Ship the version that was tested
• A single test system failing
• Modules accidentally reverting to older version
• Re-create software and tests

15
What To Track
Configuration Control

• Requirements
• Software
• Hardware
• Tests

16
Version Build Number
Configuration Control

• Simple to generate
• Unique for each build or change
• Readily visible and validated for correctness

17
Test Plan

• What will be tested and not tested?
• What approach will be taken?
• What resources are needed?
• Which people are needed?
• Schedule

18
Test Cases

• Test boundary conditions
• System interfaces
• Where have other errors been found?
• What will the users do most often?
• What is the most serious if it fails?
• Usability
• What is unique to your product's environment?

19
Boundary Conditions
Test Cases

• Values at and away from the boundaries
• Valid and invalid values
• Both numeric values and strings
• Minimum-1, minimum, maximum, maximum1, a good
  middle value

20
Where Have Errors Been Found?
Test Cases

• Errors tend to cluster
• Can a common error be fixed?
• Would a code review be useful?

21
Usability
Test Cases

• Often over looked
• First to see software outside of design
• The interface makes sense if you know the design
• Need to know your users

22
Interfaces
Test Cases

• User interface
• Interfaces between software modules
• Interfaces to other programs
• Hardware / software interfaces

23
What Will Users Do Most Often?
Test Cases

• Frequently encountered errors impact the user
  more
• Test heavily used areas more heavily
• Less used areas cant be ignored

24
What Failures Are Most Serious?
Test Cases

• Areas data could be lost
• Errors with a large financial impact
• Errors that could injure someone

25
Unique to Web Applications
Test Cases

• Versions of browsers
• Different operating systems
• Server capacity
• Multiple servers - one fails?

26
Unique to GUI Based Application
Test Cases

• System configuration changes
• Size of the screen
• Number of colors available
• Default font type and size

27
Unique to Database Applications
Test Cases

• Compatible with existing data
• Testing on a copy of real data
• Server capacity

28
Unique to Embedded Applications
Test Cases

• Can multiple simultaneous stimulus be handled?
• Are hardware failures handled correctly?
• Running out of supplies
• Components breaking or degrading
• Communications errors between components
• Can temperature changes cause the system to
  behave differently?

29
Bug Tracking Ship Decision

• Bug states
• Bug Information
• Is the software ready?

30
Bug Tracking Ship Decision
31
Bug Tracking Ship Decision
32
(No Transcript)
33
After the Testing

• Known problems and solutions
• Defects not fixed
• Shorten customer service leaning curve
• Test report
• Tuned to the audience
• Introduction conclusion
• Major defects
• What was run on which versions
• What tests couldnt be run