Introduction to Testing

1
Introduction to Testing

• CS2335
• Fall 2005

2
Agenda

• Lecture Intro to Testing
• VV
• Vocabulary
• Types/Stages of Testing
• Writing Test Plans

3
Software Quality Assurance

• Process (Analysis, Design, Code, Test)
• Reviews/Inspections
• Multi-tiered Testing Strategy
• Configuration Control (Code/Docs)
• Standards Compliance
• Measurement/Reporting

4
Software Quality Factors
Portability Reusability Interoperabilty
Maintainability Flexibility Testability
Correctness, Reliablity, Efficiency,
Integrity Usability
5
Two Key Concepts

• Validation Are we building the right thing?
  Conformance to customer requirements and quality
  attributes.
• Verification Are we building the thing right?
  Process conformance, quality activities.

6
Where does testing fit in?
PHASE O Testing Debugging
PHASE 1 Testing shows the software works
PHASE 2 Testing shows the software doesnt work
PHASE 3 Testing doesnt prove anything, it
reduces risk of unacceptable
software delivery.
PHASE 4 Testing is not an act, a mental
discipline of quality.
Boris Biezer
7
The Testing Quote
Testing cannot show the absence of defects, it
can only show that software defects are present.
We cannot test quality into a product, we have to
design it in.
8
Test Objectives

• Tests intended to find errors
• Good test cases have high p for finding a yet
  undiscovered error
• Successful tests cause program failure, i.e. find
  an undiscovered error.
• Minimal set of test cases needs to be developed
  because exhaustive testing not possible

9
What we have here

• Developers do everything they can to make the
  software work.
• Testers do everything they can to make the
  software fail.

10
Testing Paradox
Every method you use to prevent or find bugs
leaves a residue of subtler bugs against which
those methods are ineffective.
Cost
Defects Found
Time
11
Testing Phases

• Programmer
• Unit
• Integration
• Component
• System
• Usability
• Regression
• Validation (Acceptance)
• Alpha/Beta (FUT/OT)

12
Vocabulary

• Test Case a set of data/events provided for a
  specific test.
• Test Suite a set of test cases.
• Test Driver a program written to test a unit
  module
• Test Stub a module written to test a higher
  level component
• Test Harness a test environment to run the stubs
  and drivers and check the results.

13
Mock Objects

• In OO world, mock objects are replacing stubs and
  drivers
• AKA Endo-testing
• www.mockobjects.com
• Creates a fake object that emulates the
  functionality of a real object so that testing
  is easier.
• For example MockServer, MockDB

14
Types of Unit Testing

• Structural (Whitebox)
• Functional (Blackbox)
• Statistical (Random)
• Mutation
• Object Oriented (State-based)
• Insert latest PhD thesis topic here

15
Return of the Vocabulary

• Coverage a measure of the amount of testing to
  be achieved. Normally spoken of in standard
  criteria based upon test technique for a subset
  of test cases.
• Adequacy a measure of the overall efficiency and
  effectiveness of the test suite.

16
Writing a test plan (Simple Version)

• Introduction
• Purpose
• Program Description
• Test Environment
• Test Case Descriptions
• Coverage Criteria
• Details

4. Summary
17
Sample Test Plan

• Introduction
• a. Purpose This test plan describes
  procedures for testing the Zap-O-Matic 3000
  version 1.0. It specifies
• unit, component and system-level testing
  requirements.
• b. Program Description The Zap-O-Matic
  3000 is a program that wipes a hard drive clean
  of all information. It meets NSA requirements
  for B1 security.

18
Test Plan contd

• Test Environment The tests in this plan were
  carried
• out on the following configurations
• a. Pentium 4 PC with 512MB RAM and 10 GB
  free disk
• space, Windows XP with SP2, Java JRE
  1.4.2_05.
• b. Pentium 4 PC with 1GB RAM and 3 GB free
  disk
• space, LINUX Fedora Core 3 all patches
  applied,
• JAVA JRE 1.3.1_09.

19
Test Plan contd

• Test Case Descriptions
• a. Coverage Criteria For unit level tests, we
  attempted to achieve branch coverage for all
  whitebox tests, and all boundary conditions for
  blackbox tests. For system-level testing, we
  covered all input equivalence partitions.

20
Test Plan contd
b. Details Frodo Baggins (Developer 1),
Foo.java. Method getData(int, int, string)
Test 1 Purpose check that when HDD signal
is true, the GUI displays a RED light in main
monitor panel. Test Case
5,4,HIGH Expected RED
light in main monitor panel
Actual YELLOW light in main monitor
panel
Priority HIGH, must be fixed before release,
assigned to Frodo
for action.
21
Test Plan (contd)
Arnold Ziffel (Developer 2), System test,
Paragraph 15a, Test server login Test 23
Purpose Check that server rejects invalid
login. Test Case a. Bobbo
(valid user), grtda1 (invalid password).
b. Freda (invalid
user), grtda1 (valid password). Expected
a,b both rejected Actual a
accepted, b rejected Priority
Critical, unsecure operation allowed
22
Test Plan contd

• Summary
• Unit Level 100 tests executed, 95 passed
• Failures 3 critical, 2
  urgent
• System Level 250 tests executed, 150 passed
• Failures 10 critical, 35
  urgent, 50 routine,
• 5 minor
• All critical and urgent failures are
  assigned for action.

23
Stubbing Functions
int main (int argc, char argv) if (argclt4)
usage() double x f(argv1,argv2)
void usage() return double f(char
arg1, char arg2) return 3.0
24
Writing Drivers
int main( int argc, char argv) int i
for (i0 ilt10i) printf(SQR of i is
i, i, square(i))
25
Why does testing go bad?

• Schedule compression/ Waterfall
• Writing harnesses, suites,stubs and drivers
  doesnt relate directly to LOC.
• Different philosophys
• Overconfidence in silver bullets