CPSC 333 SENG 311: Foundations Principles of SE

1
CPSC 333 / SENG 311 Foundations / Principles of
SE

• Software Testing

2
Agenda

• Testing definition and introduction
• Test case design
• Blackbox testing
• Whitebox testing
• Path coverage testing
• Loop testing
• Equivalence partitioning
• Boundary value analysis

3
What is testing?
Testing is the process of executing a program
with the intent of finding errors. Glen
Myers
4
Limits of software testing

• Good testing will find bugs
• Good testing is based on requirements
• BUT Testing can only prove the presence of
  bugsnever their absence

5
Exhaustive testing
There are 1014 possible paths! If we execute one
test per millisecond, it would take 3.170 years
to test this program!!
6
Testability how easily a program can be tested

• Qualities of a testable program
• operability - it operates cleanly
• observability
• the results are easy to see
• distinct output is generated for each input
• incorrect output is easily identified
• controllability
• processing can be controlled
• tests can be automated reproduced
• decomposability - software modules can be tested
  independently
• simplicity - no complex architecture and logic
• stability - few changes are requested during
  testing
• understandability - program is easy to understand

7
Who tests the software
Understands the system, but will test
gently, and is driven by delivery
Must learn about the system, but will attempt to
break it, and is driven by quality
8
Test cases

• Describe how to test a system/module/function
• Description must identify
• system state before executing the test
• function to be tested
• parameter values for the test
• expected outcome of the test

9
Test case examples

• system state before executing the test
• ResourcePool is not empty
• function to be tested
• removeAgent(anAgent)
• parameter values for the test
• anAgent is in NOT the ResourcePool
• expected outcome of the test
• AgentNotFoundException is thrown

• system state before executing the test
• vehicle not assigned to customer
• function/method to be tested
• rentVehicle(vehicleId)
• parameter values for the test
• vehicleId of the vehicle to be rented
• expected outcome of the test
• vehicle is allocated or reserved for customer

10
Test case design
Bugs lurk in corners and congregate
at boundaries Boris Beizer
OBJECTIVE to uncover errors CRITERIA in a
complete manner CONSTRAINT with a minimum of
effort and time
11
Adequate set of test cases

• Test the interface of the components
• all public methods
• Is one test case per method enough?
• How many do we need?

12
Black box testing
requirements
output
input
events
13
Black-box testing

• Ensure that external user interface is
  functioning properly.
• give correct inputs, check for expected outputs
• create certain events, and check that system
  responds appropriately
• ensure that all user requirements are satisfied
  by systems user interface and externally
  observed behavior
• black-box tester NOT looking at guts of systems
  implementation

14
White box testing
our goal is to ensure that all statements and
conditions have been executed at least once...
15
Why bother with white box testing?

• Black box testing
• Requirements fulfilled
• Interfaces available and working
• Question Why white box testing?

16
Why cover?

• logic errors and incorrect assumptions are
  inversely proportional to a paths execution
  probability
• we often believe that a path is not likely to be
  executed in fact, reality is often counter
  intuitive
• typographical errors are random its likely that
  untested paths will contain some

17
Exhaustive testing
There are 5201014 (approx.) possible paths!
If-then-else
18
Selective testing
a selected path
19
Path coverage testing

• 1) Derive a logical complexity measure
• 2) Use it to define a basis set of execution
  paths
• First, we compute the cyclomatic complexity
• Number of edges number of nodes 1
• (Compound decisions must be split into multiple
  nodes)
• In this case, CC(G) 4
• CC(G) provides a lower bound of tests that must
  be executed to guarantee coverage of all program
  statements

20
Cyclomatic complexity
A number of industry studies have indicated that
the higher CC(G), the higher the probability of
errors.
modules
CC(G)
modules in this range are more error prone
21
Path coverage set

• Path coverage set set of paths that will
  execute all statements and all conditions in a
  program at least once
• Goal Define test cases for basis set
• Path coverage set is not unique

22
Path coverage testing
Next, we derive the independent paths Since
CC(G) 4, there are four paths Path
1 1,2,3,6,7,8 Path 2 1,2,3,5,7,8 Path
3 1,2,4,7,8 Path 4 1,2,4,7,2,47,8 Finally, we
derive test cases to exercise these paths.
23
Troubles with cyclomatic complexity for test
coverage
Example from Sommerville

• case A is
• when One gt i 1
• when Two gt i 2
• when Three gt i 3
• when Four gt i 4
• when Five gt i 5
• end case

Str array(1..5) of STRING (One, Two,
Three, Four, Five) i 1 loop exit
when Str(i) A i i 1 end loop

• CC 5 and 1 respectively, but same functionality
  gt should be tested the same

24
Documenting test cases

• Name
• Number
• Description of system state before running the
  test case
• Values for the inputs
• Expected outputs
• Short description (if needed)

25
Loops

• Cornerstone of every program
• Loops can lead to non-terminating programs

26
Loop testing
simple loop
nested loops
concatenated loops
unstructured loops
27
Loop testing simple loops

• Minimum conditions - simple loops
• 1. skip the loop entirely
• 2. only one pass through the loop
• 3. two passes through the loop
• 4. m passes through the loop m lt n
• 5. (n-1), n, and (n1) passes through the loop
• where n is the maximum number of allowable passes

28
Nested loops

• Just extending simple loop testing number of
  tests explodes
• Reduce the number of tests
• start at the innermost loop set all other loops
  to minimum values
• conduct simple loop test add out of range or
  excluded values
• work outwards while keeping inner nested loops to
  typical values
• continue until all loops have been tested

29
Black box testing
requirements
output
input
events
30
Categories of errors in black box testing

• incorrect or missing functions
• interface errors
• errors in data structures or external database
  access
• performance errors
• initialization and termination errors

31
Equivalence partitioning
mouse picks on menu
output format requests
user queries
responses to prompts
numerical data
command key input
Partitioning is based on input conditions
32
Equivalence classes (1)

• Valid data
• user supplied commands
• responses to system prompts
• file names
• computational dataphysical parametersbounding
  valuesinitiation values
• output data formatting commands
• responses to error messages
• graphical data (e.g., mouse picks)

33
Equivalence classes (2)

• Invalid data
• data outside bounds of the program
• physically impossible data
• proper value supplied in wrong place

34
Defining equivalence classes

• Input condition is a range one valid and two
  invalid classes are defined
• Input condition requires specific value one
  valid and two invalid classes are defined
• Input condition is boolean one valid and one
  invalid class are defined

35
Boundary value analysis
output domain
36
Boundary value testing

• Range a..b
• test cases a, b, just below a, just above b
• Number of values
• test cases max, min, just below min, just above
  max
• Output bounds should be checked
• Boundaries of data structures shall be checked
  (e.g. arrays)

37
Automating software testing

• Manual software testing is time consuming
• Software testing has to be repeated after every
  change (regression testing)
• Write test drivers that can run automatically and
  produce a test report

38
Were Available!

• Questions?
• if you have any questions about contents of this
  lecture or other course-related issues, please
  come by during our office hours, or send us email
• Dr. Joshua MTW, 12-1pm, ICT 548
• joshuar_at_cpsc.ucalgary.ca
• Dr. Walker WF, 1-2pm, ICT 546
• rwalker_at_cpsc.ucalgary.ca