Testing in the Small (aka Unit Testing, Class Testing)

1
Testing in the Small (aka Unit Testing, Class
Testing)
1209
2
Unit Testing

• Focus on the smallest units of code
• Functions
• Methods
• Subroutines
• Frequently done (at least partially) during code
  development by code developers
• Often the target of testing frameworks such as
  JUnit

3
Each component is

• Tested in isolation from the rest of the system
• Tested in a controlled environment
• Uses appropriately chosen input data
• Uses component-level design description as guide

4
During Unit Tests

• Data transformations across the module are tested
• Data structures are tested to ensure data
  integrity

5
Unit Test Procedures
Driver
Module to be tested
Results
Test cases
Stub
Stub
6
Two fundamental approaches

• Black box
• Based on specification
• Inner structure of test object is not considered
• White box
• Based on specification
• Inner structure of test object is the basis of
  test case selection
• Effectiveness of black box is similar to white
  box, but the mistakes found are different.
  (Hetzel 1976, Myers 1978)

7
Black Box Unit Testing
1209
8
What is black-box testing?

• Unit (code, module) seen as a black box
• No access to the internal or logical structure
• Determine if given input produces expected
  output.

Output
Input
9
Black Box Testing

• Test set is derived from requirements
• Goal is to cover the input space
• Lots of approaches to describing input space
• Equivalence Classes
• Boundary Value Analysis
• Decision Tables
• State Transitions
• Use Cases
• . . .

10
Advantage and Disadvantage

• Advantage it does not require access to the
  internal logic of a component
• Disadvantage in most real-world applications,
  impossible to test all possible inputs
• Need to define an efficient strategy to limit
  number of test cases

11
General Black Box Process

• Analyze requirements
• Select valid and invalid inputs
• Determine expected outputs
• Construct tests
• Run tests
• Compare actual outputs to expected outputs

12
Equivalence classes Basic Strategy

• Partition the input into equivalence classes
• This is the tricky part
• Its an equivalence class if
• Every test using one element of the class tests
  the same thing that any other element of the
  class tests
• If an error is found with one element, it should
  be found with any element
• If an error is not found with some element, it is
  not found by any element
• Test a subset from each class

13
Basic strategy

• Example fact(n)
• if nlt0, or ngt200 error
• if 0ltnlt20, exact value
• if 20ltnlt200, approximate value within .1
• What classes can you see?

G
14
Basic strategy

• Example fact(n)
• if nlt0, or ngt200 error
• if 0ltnlt20, exact value
• if 20ltnlt200, approximate value within .1
• Obvious classes are nlt0, 0ltnlt20, 20ltnlt200, and
  200ltn.
• Might be some subclasses here, also, but this is
  a good start.

15
Simple Example

• Suppose you are building an airline reservation
  system. A traveler can be a child, an adult, or a
  senior.
• The price depends on the type of traveler.
• The seat reservation does not depend on the type
  of traveler.
• How many test cases can you identify for the
  reservation component and the billing component?

G
16
Finding equivalence classes

• Identify restrictions for inputs and outputs in
  the specification

17
Finding equivalence classes

• Identify restrictions for inputs and outputs in
  the specification
• If there is a continuous numerical domain, create
  one valid and two or three invalid classes
  (above, below, and NaN)

18
Finding equivalence classes

• Identify restrictions for inputs and outputs in
  the specification
• If there is a continuous numerical domain, create
  one valid and two or three invalid classes
  (above, below, and NaN)
• If a number of values is required, create one
  valid and two invalid classes (one invalid, two
  invalid)

19
Finding equivalence classes

• Identify restrictions for inputs and outputs in
  the specification
• If there is a continuous numerical domain, create
  one valid and two or three invalid classes
  (above, below, and NaN)
• If a number of values is required, create one
  valid and two invalid classes
• If a set of values is specified where each may be
  treated differently, create a class for each
  element of the set and one more for elements
  outside the set

20
Finding equivalence classes

• Identify restrictions for inputs and outputs in
  the specification
• If there is a continuous numerical domain, create
  one valid and two or three invalid classes
  (above, below, and NaN)
• If a number of values is required, create one
  valid and two invalid classes
• If a set of values is specified where each may be
  treated differently, create a class for each
  element of the set and one more for elements
  outside the set
• If there is a condition, create two classes, one
  satisfying and one not satisfying the condition

21
Boundary Values

• Programs that fail at interior elements of a
  class usually fail at the boundaries too.
• Test the boundaries.
• if it should work for 1-99, test 0, 1, 99, 100.
• if it works for A-Z, try _at_, A, Z, , a, and z
• The hard part is identifying boundaries

22
Hints

• If a domain is a restricted set, check the
  boundaries. e.g., D1,10, test 0, 1, 10, 11
• It may be possible to test the boundaries of
  outputs, also.
• For ordered sets, check the first and last
  elements
• For complex data structures, the empty list, full
  lists, the zero array, and the null pointer
  should be tested
• Extremely large data sets should be tested
• Check for off-by-one errors

23
More Hints

• Some boundaries are not obvious and may depend on
  the implementation (use gray box testing if
  needed)
• Numeric limits (e.g., test 255 and 256 for 8-bit
  values)
• Implementation limits (e.g., max array size)

24
Boundary Value in class

• Determine the boundary values for US Postal
  Service ZIP codes
• Determine the boundary values for a 15- character
  last name entry.

G
25
Decision Tables

• Construct a table (to help organize the testing)
• Identify each rule or condition in the system
  that depends on some input
• For each input to one of these rules, list the
  combinations of inputs and the expected results

26
Decision Table Example
Theater ticket prices are discounted for senior
citizens and students.
Test Case C1 Student C2 Senior Result Discount?
1 T T T
2 T F T
3 F T T
4 F F F
27
Pairwise Testing
28
Problem 1From Lee Copeland, A Practitioners
Guide to Software Test Design, Artech House
Publishers, 2004.

• A web site must operate correctly with different
  browsers IE 5, IE 6, and IE 7 Mozilla 1.1
  Opera 7 FireFox 2, 3, and 4, and Chrome.
• It must work using RealPlayer, MediaPlayer, or no
  plugin.
• It needs to run under Windows ME, NT, 2000, XP,
  and Vista, 7.0, and 8.0
• It needs to accept pages from IIS, Apache and
  WebLogic running on Windows NT, 2000, and Linux
  servers
• How many different configurations are there?

G
29
Problem 1

• A web site must operate correctly with different
  browsers IE 5, IE 6, and IE 7 Mozilla 1.1
  Opera 7 FireFox 2, 3, and 4, and Chrome.
• It must work using RealPlayer, MediaPlayer, or no
  plugin.
• It needs to run under Windows ME, NT, 2000, XP,
  Vista, 7.0, and 8.0
• It needs to accept pages from IIS, Apache and
  WebLogic running on Windows NT, 2000, and Linux
  servers
• How many different configurations are there?
• 9 browsers x 3 plugins x 7 OS x 3 web servers x 3
  server OSs 1701 combinations

30
Problem 2
From Lee Copeland, A Practitioners Guide to
Software Test Design, Artech House Publishers,
2004.

• A bank is ready to test a data processing system
• Customer types
• Gold (i.e., normal)
• Platinum (i.e., important)
• Business
• Non profits
• Account types
• Checking
• Savings
• Mortgages
• Consumer loans
• Commercial loans
• States (with different rules) CA, NV, UT, ID,
  AZ, NM
• How many different configurations are there?

31
When given a large number of combinations
(options improve )

• Give up and dont test

32
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business

33
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business
• Choose one or two cases

34
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business
• Choose one or two cases
• Choose a few that the programmers already ran

35
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business
• Choose one or two cases
• Choose a few that the programmers already ran
• Choose the tests that are easy to create

36
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business
• Choose one or two cases
• Choose a few that the programmers already ran
• Choose the tests that are easy to create
• List all combinations and choose first few

37
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business
• Choose one or two cases
• Choose a few that the programmers already ran
• Choose the tests that are easy to create
• List all combinations and choose first few
• List all combinations and randomly choose some

38
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business
• Choose one or two cases
• Choose a few that the programmers already ran
• Choose the tests that are easy to create
• List all combinations and choose first few
• List all combinations and randomly choose some
• Magically choose a small subset with high
  probability of revealing defects

39
Orthogonal Arrays

• A 2-D array with the property
• All pairwise combinations occur in every pair of
  columns
• Example consider 3 variables (columns) with
  A,B, 1,2, and (a,ß)

1 2 3
1 1 A a
2 1 B ß
3 2 A ß
4 2 B a
40
Orthogonal Array Example
1 2 3
1 1 A a
2 1 B ß
3 2 A ß
4 2 B a
Look at each pair of columns (1 and 2),( 1 and
3), and (2 and 3) Does each of the 4 pairings
appear in each? (Yes, of course!)
41
Pairwise Testing Algorithm

1. Identify variables
2. Determine choices for each variable
3. Locate an orthogonal array
4. Map test cases to the orthogonal array
5. Construct tests

42
Example using Bank

• Identify variables
• Customers (4)
• Account types (5)
• States (6)
• Determine choices for each variable
• Customer types (Gold, Platinum, Business, Non
  Profit)
• Account types (Checking, Savings, Mortgages,
  Consumer loans, Commercial loans
• States (CA, NV, UT, ID, AZ, NM)
• Locate an orthogonal array
• Map test cases to the orthogonal array
• Construct tests

43
Locate an orthogonal array (look up on web)
1 1 1
1 2 2
1 3 3
1 4 4
2 1 2
2 2 1
2 3 4
2 4 3
3 1 3
3 2 4
3 3 1
3 4 2
4 1 4
4 2 3
4 3 2
4 4 1
5 5 1
5 1 2
5 2 3
5 3 4
6 5 2
6 1 1
6 2 4
6 3 3
1 5 3
2 5 4
3 5 1
4 5 2
5 4 1
6 4 2
44
Map Test Cases (30)
CA Check Gold
CA Save Plat
CA Mort Busi
CA Cons NonP
NV Check Plat
NV Save Gold
NV Mort NonP
NV Cons Busi
UT Check Busi
UT Save NonP
UT Mort Gold
UT Cons Plat
ID Check NonP
ID Save Busi
ID Mort Plat
ID Cons Gold
AZ Comm Gold
AZ Check Plat
AZ Save Busi
AZ Mort NonP
NM Comm Plat
NM Check Gold
NM Save NonP
NM Mort Busi
CA Comm Busi
NV Comm NonP
UT Comm Gold
ID Comm Plat
AZ Cons Gold
NM Cons Plat
45
Pairwise Summary

• When the combination is large, test all pairs,
  not all combinations
• Studies indicate that most defects are single or
  double-mode effects
• Can be found in pairing
• Few defects require more than two modes
• Orthogonal arrays have the pairwise property and
  can be found or generated

46
State Transition Testing
47
State Transition Testing

• Build STD of system or component
• Cover the STD
• Visit each state
• Trigger each event
• Exercises each transition
• Exercise each path
• Might not be possible

48
Example
Payment made
Ticket Printed
Paid
Res Made
Ticketed
Cancel
Cancel/refund
Cust Order/start timer
Time Expires
Cancelled Cust
Cancelled Non Pay
Ticket Delivered
Cancel/refund
Used
Customer makes reservation and has limited time
to pay. May cancel at any time.
Groups How many tests to visit each state?
49
Example Each State
Payment made
Ticket Printed
Paid
Res Made
Ticketed
Cancel
Cancel/refund
Cust Order/start timer
Time Expires
Cancelled Cust
Cancelled Non Pay
Ticket Delivered
Cancel/refund
Used
3 tests needed From start to final From
start to cancelled non-pay From start to Res
Made to Cancelled Cust
50
Example Each Event
Payment made
Ticket Printed
Paid
Res Made
Ticketed
Cancel
Cancel/refund
Cust Order/start timer
Time Expires
Cancelled Cust
Cancelled Non Pay
Ticket Delivered
Cancel/refund
Used
3 tests needed From start to final From
start to cancelled non-pay From start to Res
Made to Cancelled Cust
51
Example Each Transition
Payment made
Ticket Printed
Paid
Res Made
Ticketed
Cancel
Cancel/refund
Cust Order/start timer
Time Expires
Cancelled Cust
Cancelled Non Pay
Ticket Delivered
Cancel/refund
Used
5 tests needed
52
Use Case Testing

• Use the use cases to specify test cases
• Use case specifies both normal, alternate, and
  exceptional operations
• Use cases may not have sufficient detail
• Beizer estimates that 30-40 of effort in testing
  transactions is generating the test data
• Budget for this!

53
White-Box Testing

• Pfleeger, S. Software Engineering Theory and
  Practice 2nd Edition. Prentice Hall, 2001.
• Ghezzi, C. et al., Fundamentals of Software
  Engineering. Prentice Hall, 2002.
• Pressman, R., Software Engineering A
  Practitioners Approach, Mc Graw Hill, 2005.
• Hutchinson, Marnie, Software Testing
  Fundamentals, Wiley, 2003.

0310
54
White Box Testing

• Also known as
• Glass box testing
• Structural testing
• Test set is derived from structure of code
• Code structure represented as a Control Flow
  Graph
• Goal is to cover the CFG

55
Control Flow Graphs

• Programs are made of three kinds of statements
• Sequence (i.e., series of statements)
• Condition (i.e., if statement)
• Iteration (i.e., while, for, repeat statements)

56
Control Flow Graphs

• Programs are made of three kinds of statements
• Sequence (i.e., series of statements)
• Condition (i.e., if statement)
• Iteration (i.e., while, for, repeat statements)
• CFG visual representation of flow of control.
• Node represents a sequence of statements with
  single entry and single exit
• Edge represents transfer of control from one node
  to another

57
Control Flow Graph (CFG)
n1
n1
n1
n3
Join
Join
Sequence
If-then-else
Iterative
If-then
58
Control Flow Graph (CFG)
n1
n1
n1
n3
Join
Join
Sequence
If-then-else
Iterative
If-then
When drawing CFG, ensure that there is one exit
include the join node if needed
59
Example 1 CFG
1. read (result) 2. read (x,k) 3. while
result lt 0 then 4. ptr ? false 5. if x gt k
then 6. ptr ? true 7. x ? x 1 8.
result ? result 1 9. print result
Draw CFG
60
Example 1 CFG
1
2
1. read (result) 2. read (x,k) 3. while
result lt 0 then 4. ptr ? false 5. if x gt k
then 6. ptr ? true 7. x ? x 1 8.
result ? result 1 9. print result
3
9
4
5
6
7
8
61
Example 1 CFG
1
1,2
2
3
3
9
9
4
4,5
1. read (result) 2. read (x,k) 3. while
result lt 0 then 4. ptr ? false 5. if x gt k
then 6. ptr ? true 7. x ? x 1 8.
result ? result 1 9. print result
5
6
6
Join
7
7,8
8
62
In Class Write CFGs

• Example 2
• if (a lt b) then
• while(a lt n)
• 3. a ? a 1
• 4. else
• 5. while(b lt n)
• 6. b ? b 1

• Example 3
• 1. read (a,b)
• 2. if (a ? 0 b ? 0) then
• 3. c ? a b
• if c gt 10 then
• c ? max
• else c ? min
• 7. else print Done

• Example 4
• 1. read (a,b)
• 2. if (a ? 0 b ? 0) then
• 3. c ? a b
• 4. while( c lt 100)
• 5. c ? a b
• 6. c ? a b

63
Write a CFG

• Example 2
• 1. if (a lt b) then
• 2. while (a lt n)
• 3. a ? a 1
• 4. else
• 5. while (b lt n)
• 6. b ? b 1

64
Answers

• Example 3
• 1. read (a,b)
• 2. if (a ? 0 b ? 0) then
• 3. c ? a b
• 4. if c gt 10 then
• 5. c ? max
• else c ? min
• 7. else print Done

65
Answers

• Example 4
• 1. read (a,b)
• 2. if (a ? 0 b ? 0) then
• 3. c ? a b
• 4. while( c lt 100)
• 5. c ? a b
• 6. c ? a b

66
Coverage

• Statement
• Branch
• Condition
• Path
• Def-use
• Others

67
Statement Coverage

• Every statement gets executed at least once
• Every node in the CFG gets visited at least once

68
Examples number of paths neededfor Statement
Coverage? (from Hutchinson, Software Testing
Fundamentals, Wiley, 2003)
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
5
5
69
Examples number of paths neededfor Statement
Coverage? (from Hutchinson, Software Testing
Fundamentals, Wiley, 2003)
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
4
1
5
5
70
Branch coverage

• Every decision is made true and false
• Every edge in a CFG of the program gets traversed
  at least once
• Also known as
• Decision coverage
• All edge coverage
• Basis path coverage
• Branch is finer than statement
• C1 is finer than C2 if
• ? T1?C1 ?T2?C2 ? T2 ? T1

71
Examples number of paths neededfor Branch
Coverage? (from Hutchinson, Software Testing
Fundamentals, Wiley, 2003)
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
5
5
72
Examples number of paths neededfor Branch
Coverage? (from Hutchinson, Software Testing
Fundamentals, Wiley, 2003)
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
5
2
5
5
73
Condition coverage

• Every complex condition is made true and false by
  every possible combination
• E.G., (x and y)
• x true, y true
• xfalse, ytrue
• x true, y false
• x false, y false
• There are lots of different types of condition
  coverage Condition, multiple condition,
  condition/decision, modified condition/decision
  (MCDC), Im only covering the simplest.
• Condition coverage is not finer than branch
  coverage
• There are pathological cases where you can
  achieve Condition and not Branch
• Under most circumstances, achieving Condition
  achieves Branch

74
Condition Coverage CFGs

• One way to determine the number of paths is to
  break the compound conditional into atomic
  conditionals
• Suppose you were writing the CFG for the assembly
  language implementation of the control construct
• If (A AND B) then
• C
• Endif
• (short circuit eval) (no short circuit eval)
• LD A LD A in general, lots of
• BZ endif LAND B code for A and B
• LD B BZ endif
• BZ endif JSR C
• JSR C endif nop
• endif nop

75
AND Condition
1

• 1. read (a,b)
• 2. if (a 0 b 0) then
• 3. c ? a b
• 4. else c ? a b
• paths
• 1, 2A,2B,3, J
• 1, 2A, 2B, 4, J
• 1, 2A, 4, J

2A
2B
4
3
Join
76
OR Condition
1

• 1. read (a,b)
• 2. if (a 0 b 0) then
• 3. c ? a b
• 4. while( c lt 100)
• 5. c ? a b
• Paths
• 1, 2A, 3, 4, 5, 4 J
• 1, 2A, 3, 4, J
• 1, 2A, 2B, 3, 4, 5, 4, J
• 1,2A, 2B, J

2A
3
2B
4
5
Join
77
Path

• A path is a sequence of statements
• A path is sequence of branches
• A path is a sequence of edges

78
Examples number of paths neededfor Path
Coverage? (from Hutchinson, Software Testing
Fundamentals, Wiley, 2003)
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
5
5
79
Examples number of paths neededfor Path
Coverage? (from Hutchinson, Software Testing
Fundamentals, Wiley, 2003)
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
5
16
5
5
80
Path Coverage-1

• Every distinct path through code is executed at
  least once
• Example
• 1. read (x)
• 2. read (z)
• 3. if x ? 0 then begin
• 4. y ? x z
• 5. x ? z end
• 6. else print Invalid
• 7. if y gt 1 then
• 8. print y
• 9. else print Invalid
• Test Paths
• 1, 2, 3, 4, 5, J1, 7, 8, J2
• 1, 2, 3, 4, 5, J1, 7, 9, J2
• 1, 2, 3, 6, J1, 7, 8, J2,
• 1, 2, 3, 6, J1, 7, 9, J2

1,2,3
4,5
6
Join1
7
8
9
Join2
81
Counting Paths

• It is not feasible to calculate the total number
  of paths

82
Linearly independent paths

• It is feasible to calculate the number of
  linearly independent paths
• The number of linearly independent paths is the
  number of end-to-end paths required to touch
  every path segment at least once
• A linearly independent path introduces at least
  one new set of process statements or a new
  condition

83
Cyclomatic Complexity

• Software metric for the logical complexity of a
  program.
• Defines the number of independent paths in the
  basis set of a program
• Provides the upper bound for the number of tests
  that must be conducted to ensure that all
  statements been have executed at least once
• For Edges (E) and Nodes (N)
• V(G) E N 2

84
Examples Complexity of CFGs
3,4
3,4
1
1
5
5
6
2
Join
Join
3
N3 E2 E-N2 2-32 1V(G)
N4 E4 E-N2 4-42 2V(G)
N3 E3 E-N2 3-32 2V(G)
N1 E0 E-N2 0-12 1V(G)
85
Example
1
2,3
6
4,5
8
7
9
10
11
86
In Class Compute the cyclomatic complexity
1,2
3
9
4,5
6
Join
7,8
87
Example 1 CFG
1,2
N7 E8 E-N2 8-72 3V(G)
3
9
4,5
6
Join
7,8
88
Example 2
N6 E8 E-N2 8-62 4V(G)
89
Answers
N7 E8 E-N2 8-72 3V(G)
90
Answers
N6 E7 E-N2 7-62 3V(G)
91
Independent Path Coverage

• Basis set does not yield minimal test set
• Example
• 1. read (x)
• 2. read (z)
• 3. if x ? 0 then begin
• 4. y ? x z
• 5. x ? z end
• 6. else print Invalid
• 7. if y gt 1 then
• 8. print y
• 9. else print Invalid
• Cyclomatic complexity 3
• Test Paths
• 1, 2, 3, 4, 5, J1, 7, 8, J2
• 1, 2, 3, 4, 5, J1, 7, 9, J2
• 1, 2, 3, 6, J1, 7, 8, J2,

1,2,3
4,5
6
Join1
7
8
9
Join2
92
Def-Use Coverage

• Def-use coverage every path from every
  definition of every variable to every use of that
  definition is exercised in some test.
• Example
• 1. read (x)
• 2. read (z)
• 3. if x ? 0 then begin
• 4. y ? x z
• 5. x ? z end
• 6. else print Invalid
• 7. if y gt 1 then
• 8. print y
• 9. else print Invalid

Def x, z Use x
1,2,3
4,5
Def y, x Use x, z
6
Use none
Join
7
Use y
8
Use none
9
Use y

• Test Path 1, 2, 3, 4, 5, 7, 8, J

Join
93
Strength of Coverage
Statement
Arrows point from weaker to stronger
coverage. Stronger coverage requires more test
cases.
Branch
Def-Use
Condition
Path
94
What paths dont tell you

• Timing errors
• Unanticipated error conditions
• User interface inconsistency (or anything else)
• Configuration errors
• Capacity errors