Test Management

1
Test Management
2
Definition of VV

• Verification - is the product correct
• Validation - is it the correct product

3
Background Info

• Code Testing ? Code Walkthrough
• Definition of Testing
• Software Testing is a formal process carried out
  by a specialized team in which a software unit,
  several integrated units or an entire software
  package are examined by running the programs on a
  computer.
• Testing is the single biggest SQA task.
• on average, 24 of the development budget is
  testing
• Objectives of Testing
• reveal errors
• assurance of acceptable quality
• compile record of software errors

Software Quality Assurance by Galin pages 178-182
4
Laws of Testing

• The best person to test your code is someone
  else.
• A good test is one that finds an error.
• Testing can not prove the absence of errors.
• Complete test coverage is impossible, so
  concentrate on problem areas.
• It cost a lot less to remove bugs early.

5
Testing Stages

• Unit Testing
• modules of code
• Integration Testing
• design
• Validation Testing
• requirements
• System Testing
• system engineering

6
Reality Check

• Why not just run the whole thing and see if it
  gives us the right answer or if it crashes?

7
Unit Testing

• White Box Testing
• testing a module of code based on the source code
• Black Box Testing
• testing a module based on its description and/or
  the requirements specification
• Also called "functional" and "behavioral" testing
• Proof of Correctness
• mathematically-based analysis of the
  requirements, similar to theorem proving

8
White Box Testing Fundamentals

• White Box testing is much more expensive than
  Black Box testing.
• White Box is most appropriate when we must assure
  that the calculations are correct.
• Covering every possible path through a module is
  usually not practical.
• 10 if-then statements might require 1024 test
  cases
• instead, base the number of tests on the
  complexity of the module (e.g. Basis Path Testing)

9
Types of Code Coverage

• Function coverage
• Has each function in the program been executed?
• Entry/exit coverage
• Has every possible call and return of the
  function been executed?
• Statement coverage
• Has each line of the source code been executed?
• Condition coverage
• Has each evaluation point (such as a true/false
  decision) been executed?
• Path coverage
• Has every possible route through a given part of
  the code been executed?

More Complex
10
Example Coverage

• int example1 (int value, boolean cond1, boolean
  cond2)
• if ( cond1 )
• value
• if ( cond2 )
• value --
• return value
• Total Statement Coverage with one case - True
  True.
• Total Path Coverage with four paths - TT TF FT
  FF.
• But, total path coverage is usually impractical,
  so Basis Path Testing is usually better.

11
Simple Black Box Testing

• Create Test Cases for
• Easy-to-compute data
• Typical data
• Boundary / extreme data
• Bogus data

12
Random Testing

• Method
• just randomly generate test data
• Advantage
• very easy to generate lots of test cases
• Disadvantage
• you have to get lucky that a test case will find
  an error

13
Integration Testing

• Each of the following modules, shown below in the
  application's control flow chart, has finally
  passed unit testing standards. How do you plan
  to conduct integration testing?

A
E
D
C
B
G
F
H
I
K
J
M
L
14
User Interface Testing
15
System Testing

• Recovery Testing
• Security Testing
• use software to analyze source code for stack
  buffer overflow attacks, etc
• Performance Testing
• Stress Testing
• Alpha Testing / Beta Testing

Pressman 5th Edition
16
And the Moral of the Story is...

• Use Testing Tools

17
Types of Testing Tools

• Regression Testing
• retest to find side-effects
• Database Testing
• test the queries
• build tests based on tables and fields
• load test the database
• Load Testing
• simulate a lot of users - stress test
• Functional Testing
• record a user's actions
• try lots of random user actions
• build test cases based on Use Case Scenarios
• Testing Web Applications
• Unit Testing

18
Cost of Quality

• Not all Quality Assurance tasks cost the same
• creating and using design standards
• creating and using coding standards
• conducting formal technical design reviews
• peer reviews of code
• black box testing
• white box testing
• white box testing with high path coverage
• Not all QA tasks have the same return

higher costs
19
Cost of Bug Removal

• Tracking error discovery costs is fairly easy.

do remove_bugs() until ( removal_cost gt
error_recovery_costs )
Number of Bugs Found per Day
Day Number
20
Cost of Quality

• What level of Quality should we shoot for?

SQA Failure

Cost of SQA
Cost of Failure
Level of Software Quality
Optimal Quality Level
21
And the glue that holds it all together

• The Test Plan
• who
• what
• when
• where
• how

22
Test Plan Considerations

• What are the critical or most complex modules?
• make sure they get integration tested first
• probably deserve white-box attention
• Where have you had problems in the past?
• Third-Party delivered components?
• What training is required?
• conducting formal reviews
• use of testing tools
• defect report logging

23
IEEE 829 - Standard forSoftware Test
Documentation

• Recommends 8 types of testing documents
• Test Plan
• next slide
• Test Design Specification
• expected results, pass criteria, ...
• Test Case Specification
• test data for use in running the tests
• Test Procedure Specification
• how to run each test
• Test Item Transmittal Report
• reporting on when components have progressed from
  one stage of testing to the next
• Test Log
• Test Incident Report
• for any test that failed, the actual versus
  expected result
• Test Summary Report
• management report

http//en.wikipedia.org/wiki/IEEE_829
24
Test Plan Contents (IEEE 829 format)

• Test Plan Identifier
• References
• Introduction
• Test Items
• see next slide
• Software Risk Issues
• Features to be Tested
• Features not to be Tested
• Approach
• Item Pass/Fail Criteria
• Suspension Criteria and Resumption Requirements

1. Test Deliverables
2. Remaining Test Tasks
3. Environmental Needs
4. Staffing and Training Needs
5. Responsibilities
6. Schedule
7. Planning Risks and Contingencies
8. Approvals
9. Glossary

http//en.wikipedia.org/wiki/Test_Plan
25
4. Test Items

• Requirements Specification
• Design
• Modules / Source Code Units
• User/Operator Material
• the user interface
• User Guide
• Operations Guide
• Features
• response time, data accuracy, security, etc
• System Validation
• alpha and beta testing

somewhat based on IEEE 829
26
Wrap Up Question

• And so, with all these techniques and tools at
  our disposal, why does the product that we
  delivery to the customer still contain so many
  bugs?