Black-Box Testing Techniques II

1
Black-Box Testing Techniques II
Software Testing and Verification Lecture 5

• Prepared by
• Stephen M. Thebaut, Ph.D.
• University of Florida

2
Cause-Effect Analysis

• Cause-Effect Analysis is a combinatorial approach
  that can be viewed as a logical extension of
  partition testing.
• It extends the idea of partitioning a
  multi-dimensional input space by providing a
  systematic means for generating test case
  templates to cover different combinations of
  input Causes resulting in output Effects.

3
Causes and Effects

• A CAUSE may be thought of as a distinct input
  condition, or an equivalence class of input
  conditions.
• An EFFECT may be thought of as a distinct output
  condition or change in program state.

4
Causes and Effects

• Causes and Effects are represented as Boolean
  variables.
• The logical relationships among them CAN (but
  need not) be represented as one or more Boolean
  graphs.

? ?
Causes
Effects
V ?
5
C-E Analysis Process Steps

• Identify Causes and Effects
• The most critical and usually the most difficult
  step
• Choose an appropriate level of abstraction.
• Divide and conquer as necessary.
• Effects may or may not be mutually exclusive.

6
C-E Analysis Process Steps (contd)

• Deduce Logical Relationships and Constraints
• Relationships take the form of conditionals and
  utilize the logical operators AND, OR, and NOT.
• Constraints describe relationships among Causes
  that allow for the identification of impossible
  combinations.

7
C-E Analysis Process Steps (contd)

• Deduce Logical Relationships and Constraints
  (contd)
• Boolean graphs provide a convenient and
  economical way to visualize relationships and
  constraints.

8
C-E Analysis Process Steps (contd)

• Identify an appropriate Test Case Selection
  Strategy
• Determines the number and nature of
  Cause-combinations to be considered.
• Strategies can be designed to meet a variety of
  coverage requirements/ cost constraints.

9
C-E Analysis Process Steps (contd)

• Construct a Test Case Coverage Matrix
• Typically involves tracing through the
  Cause-Effect relationships to identify
  combinations of Causes resulting in each Effect
  according to the selection strategy chosen.
• This can be extremely tedious...

10
Question

• To what extent do you think CASE support might
  be applicable to each step in the process? For
  which steps do you think it might be most
  important?

11
Illustration of Step 1 (Identify Causes and
Effects)

• The first input is a yes/no response to the
  question Do you reside within the city? The
  second input is gross pay for the year in
  question.
• A non-resident will pay 1 of the gross pay in
  city tax.
• Residents pay on the following scale
• - If gross pay is no more than 30,000, the tax
  is 1.
• - If gross pay is more than 30,000, but no more
  than
• 50,000, the tax is 5.
• - If gross pay is more than 50,000, the tax is
  15.

12
Guidelines for identifying Causes and Effects

• Underline words or phrases in the specification
  that correspond to input/output conditions or
  changes in state.
• List each Cause and Effect.
• Assign a unique number to each (use different
  number ranges to differentiate Causes from
  Effects).

13
Guidelines for identifying Causes and Effects
(contd)

• The first input is a yes/no response to the
  question Do you reside within the city? The
  second input is gross pay for the year in
  question.
• A non-resident will pay 1 of the gross pay in
  city tax.
• Residents pay on the following scale
• - If gross pay is no more than 30,000, the tax
  is 1.
• - If gross pay is more than 30,000, but no more
  than
• 50,000, the tax is 5.
• - If gross pay is more than 50,000, the tax is
  15.

14
Illustration of Step 1 (contd)

• Ignoring, again, the unspecified responses to
  invalid inputs, we have
• Causes Effects
• (1) Non-Resident (11) 1 tax
• (2) Resident (12) 5 tax
• (3) 0 ? Gross Pay ? 30K (13) 15 tax
• (4) 30K ? Gross Pay ? 50K
• (5) Gross Pay ? 50K

15
Illustration of Step 2 (Deduce Logical
Relationships and Constraints)

• The first input is a yes/no response to the
  question Do you reside within the city? The
  second input is gross pay for the year in
  question.
• A non-resident will pay 1 of the gross pay in
  city tax.
• Residents pay on the following scale
• - If gross pay is no more than 30,000, the tax
  is 1.
• - If gross pay is more than 30,000, but no more
  than
• 50,000, the tax is 5.
• - If gross pay is more than 50,000, the tax is
  15.

16
What are the constraints?

• Causes Effects
• (1) Non-Resident (11) 1 tax
• (2) Resident (12) 5 tax
• (3) 0 ? Gross Pay ? 30K (13) 15 tax
• (4) 30K ? Gross Pay ? 50K
• (5) Gross Pay ? 50K

17
Constraints deducible from spec, problem domain
knowledge, etc.

• (1) ? (2) V (1) ? (2) (i.e., one, and only
  one of (1) and (2) must be true.)
• (3) ? (4) ? (5) V (3) ? (4) ? (5) V
• (3) ? (4) ? (5)
• (11) ? (12) ? (13) V (11) ? (12) ?
• (13) V (11) ? (12) ? (13)

18
What are the logical relationships?

• The first input is a yes/no response to the
  question Do you reside within the city? The
  second input is gross pay for the year in
  question.
• A non-resident will pay 1 of the gross pay in
  city tax.
• Residents pay on the following scale
• - If gross pay is no more than 30,000, the tax
  is 1.
• - If gross pay is more than 30,000, but no more
  than
• 50,000, the tax is 5.
• - If gross pay is more than 50,000, the tax is
  15.

19
Conditionals deducible from specification and
constraints

• From the specification we have
• (1) gt (11)
• (2) ? (3) gt (11)
• (2) ? (4) gt (12)
• (2) ? (5) gt (13)

20
Conditionals deducible from specification and
constraints (contd)

• Which, in light of the identified constraints,
  simplify to
• (1) V (3) gt (11)
• (2) ? (4) gt (12)
• (2) ? (5) gt (13)

21
Boolean Graph Representation

• Non-Res (1)
• (11) 1 tax
• 0,30K (3)
• (30K,50K (4)
• (12) 5 tax
• Res (2)
• (13) 15 tax
• gt50K (5)

V ?
O
O
? ?
O
? ?
22
Cause/Effect Constraints
Exclusive
Inclusive
E
at most one
I
at least one
One Only One
Requires
A
A gt B
O
one and only one
B
23
Illustration of Step 3 (Identify Test Case
Selection Strategy)

• Simple (but extreme) strategies
• All Feasible Combinations of Cause Values
  (AFCCV)
• All Effects (AE)
• For the relationships depicted in our graph, how
  many test cases would be required to achieve
  AFCCV coverage? AE coverage?

24
AFCCV and AE Coverage

• Non-Res (1)
• (11) 1 tax
• 0,30K (3)
• (30K,50K (4)
• (12) 5 tax
• Res (2)
• (13) 15 tax
• gt50K (5)

V ?
O
O
? ?
O
? ?
25
AFCCV and AE Coverage (contd)

• AFCCV
• There are 25 32 possible value combinations for
  all 5 Causes.
• For Causes (1) and (2), there are 2 feasible
  value combination pairs (due to the one and only
  one constraint) TF and FT.
• (contd)

26
AFCCV and AE Coverage (contd)

• AFCCV (contd)
• Similarly, for Causes (3), (4), and (5), there
  are 3 feasible value combination triples TFF,
  FTF, and FFT.
• Thus, there are 2 X 3 6 feasible combinations
  of values for all 5 Causes, requiring a total of
  6 test cases.

27
AFCCV and AE Coverage (contd)

• AE
• There are 3 mutually exclusive Effects.
• Thus, a different combination of Cause values is
  required for each Effect to evaluate to True.
• Therefore, 3 test cases are required.
• (contd)

28
AFCCV and AE Coverage (contd)

• AE (contd)
• In general, when there are N Effects, N or fewer
  test cases are required for AE Coverage.
• When the N Effects are mutually exclusive, all N
  test cases are required.

29
AFCCV and AE Coverage (contd)

• Note that AE is analogous to partitioning an
  input space based solely on the specified
  outputs.
• AFCCV is analogous to associating a separate
  equivalence class with every (feasible)
  combination of the individual input classes
  (i.e., the brute-force approach).
• Question do these strategies depend on the
  Cause-Effect relationships?

30
Another Test Case Selection Strategy

• REPEAT
• Select the next (initially, the first) Effect.
• Tracing back through the graph (right to left),
  find all feasible combinations of connected Cause
  values that result in the Effect being True.
• For each new such combination found
• Determine values of all other Effects, and
• Enter values for each Cause and Effect in a new
  column of the test case coverage matrix.
• UNTIL each Effect has been selected.

31
What Should We Call this Strategy?

• How about All Feasible Combinations of
  Connected Cause Values that Result in Each Effect
  being True (AFCCCVREET)?

32
Illustration of Step 4 (Construct a Test Case
Coverage Matrix)

• For Strategy 3, this involves tracing through
  the Cause-Effect relationships to identify all
  feasible combinations of connected Causes
  resulting in each Effect being true

33
Applying Strategy 3

• Non-Res (1)
• (11) 1 tax
• 0,30K (3)
• (30K,50K (4)
• (12) 5 tax
• Res (2)
• (13) 15 tax
• gt50K (5)

V ?
O
O
? ?
O
? ?
34
Applying Strategy 3

• Cause Value Combinations for Effect 11
• (1) V (3) ? 1, 3 or
• 1, ?3 or
• ?1, 3
• Cause Value Combinations for Effect 12
• Cause Value Combinations for Effect 13

35
Coverage Matrix
TEST CASES TEST CASES TEST CASES TEST CASES TEST CASES TEST CASES
CAUSES 1 2 3 4 5
Non-Resident (1) T T F
Resident (2) F F T
0 ? Gross Pay ? 30K (3) T F T
30K ? Gross Pay ? 50 (4) F ? F
Gross Pay ? 50K (5) F ? F
EFFECTS
1 tax (11) T T T
5 tax (12) F F F
15 tax (13) F F F
? dont care, subject to Cause constraint B
36
Applying Strategy 3

• Non-Res (1)
• (11) 1 tax
• 0,30K (3)
• (30K,50K (4)
• (12) 5 tax
• Res (2)
• (13) 15 tax
• gt50K (5)

V ?
O
O
? ?
O
? ?
37
Applying Strategy 3

• Cause Value Combinations for Effect 11
• (1) V (3) ? 1, 3 or
• 1, ?3 or
• ?1, 3
• Cause Value Combinations for Effect 12
• (4) ? (2) ? 2, 4
• Cause Value Combinations for Effect 13

38
Coverage Matrix (contd)
TEST CASES TEST CASES TEST CASES TEST CASES TEST CASES TEST CASES
CAUSES 1 2 3 4 5
Non-Resident (1) T T F F
Resident (2) F F T T
0 ? Gross Pay ? 30K (3) T F T F
30K ? Gross Pay ? 50K (4) F ? F T
Gross Pay ? 50K (5) F ? F F
EFFECTS
1 tax (11) T T T F
5 tax (12) F F F T
15 tax (13) F F F F
? dont care, subject to Cause constraint B
39
Applying Strategy 3

• Non-Res (1)
• (11) 1 tax
• 0,30K (3)
• (30K,50K (4)
• (12) 5 tax
• Res (2)
• (13) 15 tax
• gt50K (5)

V ?
O
O
? ?
O
? ?
40
Applying Strategy 3

• Cause Value Combinations for Effect 11
• (1) V (3) ? 1, 3 or
• 1, ?3 or
• ?1, 3
• Cause Value Combinations for Effect 12
• (4) ? (2) ? 2, 4
• Cause Value Combinations for Effect 13
• (5) ? (2) ? 2, 5

41
Coverage Matrix (contd)
TEST CASES TEST CASES TEST CASES TEST CASES TEST CASES TEST CASES
CAUSES 1 2 3 4 5
Non-Resident (1) T T F F F
Resident (2) F F T T T
0 ? Gross Pay ? 30K (3) T F T F F
30K ? Gross Pay ? 50K (4) F ? F T F
Gross Pay ? 50K (5) F ? F F T
EFFECTS
1 tax (11) T T T F F
5 tax (12) F F F T F
15 tax (13) F F F F T
? dont care, subject to Cause constraint B
42
Complete Coverage Matrix
TEST CASES TEST CASES TEST CASES TEST CASES TEST CASES TEST CASES
CAUSES 1 2 3 4 5
Non-Resident (1) T T F F F
Resident (2) F F T T T
0 ? Gross Pay ? 30K (3) T F T F F
30K ? Gross Pay ? 50K (4) F ? F T F
Gross Pay ? 50K (5) F ? F F T
EFFECTS
1 tax (11) T T T F F
5 tax (12) F F F T F
15 tax (13) F F F F T
? dont care, subject to Cause constraint B
43
Comparing Strategy 3 to AFCCV and AE

• How does Strategy 3 differ from AFCCV?
• For each Effect, only the connected Causes are
  considered.
• It is less conservative
• Does not ensure that every feasible combination
  of Cause values will be covered.
• And thus does not ensure that every feasible
  combination of Effect values will be covered.
  (Relevant when Effects are not mutually
  exclusive.)

44
Comparing Strategy 3 to AFCCV and AE

• How does it differ from AE?
• It is more conservative (ALL feasible
  combinations of connected Cause values must be
  covered for each Effect).

45
Coming up in Black-Box Testing Techniques III

• We step-through another (somewhat more complex)
  example of Cause-Effect Analysis,
• Describe a test case design technique for
  exploring boundary conditions, and
• Consider a test case design strategy based on
  intuition and experience.

46
Black-Box Testing Techniques II
Software Testing and Verification Lecture 5

• Prepared by
• Stephen M. Thebaut, Ph.D.
• University of Florida