Testing

1
Testing

• Software Engineering in PracticeCS420

2
Review What is Testing?

• IEEE
• The process of operating a system or component
  under specified condition, observing or recording
  the results, and making an evaluation of some
  aspect of the system or component.
• Specified Condition Test Cases

3
Review What is testing?

• Rick Craig and Stefan Jaskiel
• Testing is a concurrent lifecycle process of
  engineering, using and maintaining testware in
  order to measure and improve the quality of the
  software being tested
• Includes planning, analysis, and design that lead
  to test cases

4
Review What is testing?

• Testing is the process of comparing what is
  with what ought to be

5
Review Test Cases

• Need to be designed
• Systematic approach
• Documentation Planning
• Goal and purpose

6
Review Test Case

• Inputs
• Expected Outputs
• Order of Execution
• (and Testing Criteria) how do I know when Im
  done testing???

7
Review Types of Testing

• Black Box
• Reqs and spec, no knowledge of internal structure
• White Box
• Based on internal paths, structure
• Ex1 - definition use pairs

8
Pairwise Testing

• A Web Site must operate with various
• Browsers Netscape, IE, Opera,
• Plug-ins RealPlayer, Media Player, or none
• Operating Systems 95, 98, ME, NT, 2000, XP
• Web Servers Apache, Weblogic, IIS
• OS Sever Linux, NT, 2000
• -Large Number of combinations 1200

9
Worst Case

• Dont test at all
• Test all combinations as miss release date
• Choose one or two and hope for best
• Choose test you have already run
• Choose test that are easy to create
• Make a list of combos and choose first few
• Make a list of all the combinations choose a
  random subset

10
Best Case

• Choose a specially selected, fairly small subset
  that find a great many defects, more that you
  would expect from a small subset.
• But how?

11
Combinatorial Testing Problems

• Developers/Testers frequently face situations
  where testing combinations of inputs to a system
  is important.
• Examples of systems that require combinatorial
  testing include transaction-based systems,
  form-based systems, APIs, others.

12
Combinatorial Testing Problem
Combinatorial testing problems generally follow a
simple input-process-output model the state of
the system is not the focus of combinatorial
testing.
13
Combinatorial Testing

• Testers begin the process of combinatorial
  testing by selecting test data values for each
  system input.
• The number of combinations of the selected test
  data values is very large phenomenon referred to
  as the combinatorial explosion.

14
OPEN TICKET MAINTENANCE TICKET DATA
Tkt Id ____ Entity for Tkt
__________________________ Referred to
Entity __________ Work Type
___________ Loc/Fic _______ Loc/Unit
________ Title Loc
_________________________
Related Entities ________________________
_______ Ticket Create Time _________
Date ______ History Log Id
__________________ Date __/__/__ Rptd
Code _______ Rpt Description
________________ Report Type ____ Direction
______ CLFI
____________
Do you want to enter
comments? ____
With 3 values per variable, there are 320 ? 3.5
109 inputs to test
I.S. Dunietz, W.K. Ehrlich, B.D. Szablak, C.L.
Mallows, and A. Iannino, Applying Design of
Experiments to Software Testing, Proceedings of
the Nineteenth International Conference on
Software Engineering, pp. 205-215, ACM Press, May
1997.
15
Combinatorics - Combinations

• The number of ways of picking k unordered
  outcomes from n possibilities.

Example
16
Combinatorics - Permutations

• The number of ways of obtaining an ordered
  subset of k elements from a set of n elements is
  given by

The number of permutations on a set of n elements
is given by n!   
17
Combinatorial Test Reduction

• Testing all combinations of selected test data
  values is usually impossible!
• Not to test any combinations of input may
  increase the risk that a software fault escapes
  into a fielded system.
• Testers look for ways to select an effective and
  economical subset of test cases.

18
Pairwise Testing
Dont test all combinations. Test all pairs. -
Bernie Berger, STAREast 2003 International
Conference on Software Testing Pairwise testing
is a popular, frequently written about
technique. Covered in Books by Kaner, Bach,
Pettichord Copeland McGregor Sykes Splaine
Jakiel. More than 50 conference and journal
papers on the topic.
19
Pairwise Definition

• Pairwise testing an approach to combinatorial
  testing that executes a pairwise test data set.
• Pairwise test data set - A set of test cases that
  covers all combinations of the selected test data
  values for every pair of a systems input
  variables.
• Pairwise testing is also known as 2-way testing
  other approaches include 3-way, 4-way, , n-way.

20
All-pairs testing or pairwise testing

• is a combinatorial testing method that, for each
  pair of input parameters to a system tests all
  possible discrete combinations of those
  parameters. This can be done much faster than an
  exhaustive search of all combinations The number
  of tests is typically O(nm), where n and m are
  the number of possibilities for each of the two
  parameters with the most choices.
• The reasoning behind all-pairs testing the
  simplest bugs in a program are generally
  triggered by a single input parameter. The next
  simplest category of bugs consists of those
  dependent on interactions between pairs of
  parameters, which can be caught with all-pairs
  testing. Bugs involving interactions between
  three or more parameters are progressively less
  common, whilst at the same time being
  progressively more expensive to find by
  exhaustive testing, which has as its limit the
  exhaustive testing of all possible inputs.

21
Pairwise Example
A1, 2 BNW, SE CTDK, BDM
System T
22
Pairwise Example
A1, 2 BNW, SE CTDK, BDM
A, B, C-------1,NW,-1,SE,-2,NW,-2,SE,-
A, B, C-------1,-,TDK1,-,BDM2,-,TDK2,-,BDM
A, B, C--------,NW,TDK-,NW,BDM-,SE,TDK-,SE,BD
M
Dont care condition indicated with by a dash ( -
)
23
Pairwise Reductions
24
Motivation for Using Pairwise Testing

• Generates small test sets, relative to the
  combinatorial test data set.Size of the
  pairwise test set is usually 1-20 of the
  combinatorial test data set
• Many combinations of selected test data values
  tested.

25
Motivation for Using Pairwise Testing
This method finds all double-mode faults that are
two parameters conflicting with each other.
G.T. Daich, New Spreadsheet Tool Helps
Determine Minimal Set of Test Parameter
Combinations, CrossTalk, August 2003, pp.
26-30. An analysis of field data at Bellcore
indicated that most field faults are caused by
interactions of 1 or 2 fields. G.C. Patton,
DAT (Defect Analsysis Team) 1986-1990 Overview,
Internal Bellcore Technical Memo, 1991.
26
Motivation for Using Pairwise Testing
Wallace and Kuhn determined that 98 of the
reported software defects in recalled medical
devices could have been detected by testing all
pairs of parameter settings. D.R. Wallace, D.R.
Kuhn, Failure Modes in Medical Device Software
An Analysis of 15 Years of Recall Data, Intl.
Journal of Reliability, Quality, and Safety
Engineering, vol. 8, no. 4.
27
Creating Pairwise Test Sets

• Generation using Orthogonal Arrays
• Algorithmic generation

28
Orthogonal Array

• Mathematical curiosity discovered by monks in the
  1400s
• Promoted by Dr. Genichi Taguchi in the 1950s for
  use in statistical test design in manufacturing,
  agriculture, medicine, etc.
• Orthogonal Array Two dimensional array of
  numbers in which any two columns in the array
  receive an even distribution of all pairwise
  combinations of values in the array.

29
    ORTHOGONAL ARRAY
30
Orthogonal Array

• Orthogonal arrays do not exist for all setting of
  input and values
• Sloanes catalog of orthogonal arrays.www.researc
  h.att.com/njas/oadir
• rdExpert commercial pairwise testing tool based
  on orthogonal arrays.www.phadkeassociates.com

31
Algorithmic Generation

• Couldnt I just write a program to do this?
• AETG Commercial n-way test generations
  tool.aetgweb.argreenhouse.com
• Allpairs Free perl script from James
  Bachwww.satisfice.com
• tvguwm Free java source codewww.sourceforge.com
  

32
How Well Does Pairwise Testing Work?
The all-pairs and all-values tests detected 88
of the correctness and convergence bugs, but
only half of the interface and engine bugs. B.
Smith, M. S. Feather, and N. Muscettola,
"Challenges and Methods in Testing the Remote
Agent Planner," in Proc. 5th Int.nl Conf. on
Artificial Intelligence Planning and Scheduling
(AIPS 2000), 2000, pp. 254-263.
33
Pairwise vs. Random Selection
We would like to know that our testing techniques
work at least as well as random testing. Make a
list of all the combinations and choose a random
subset.
34

35
(No Transcript)
36
How Many Pairs does a Random Set Cover?
37
Pairwise Weakness
Fault detection capability of a combinatorial
test set is determined by 1) The selected test
data values. 2) The combinations of these values
that are tested. 3) The combinatorial
characteristics of the system under test. The
pairwise technique addresses point 2 only!
38
Pairwise Conclusion

• Pairwise testing is the execution of a test data
  set that covers all combinations of the selected
  test data values for every pair of a systems
  input variables.
• It is used on combinatorial testing problems that
  appropriate for pairwise testing (i.e., program
  outputs influenced by at most two inputs).

39
Pairwise Conclusion

• The strength of pairwise testing is that it
  systematically generates small test sets that
  combine many test data values.
• The weakness of the pairwise technique is that is
  does not take into account the combinatorial
  characteristics of the system under test.

40
Testing with Use-Cases

• Use-Cases do not specify input
• Must select inputs
• Try equivalence class
• Boundary value techniques
• Create at least one test case for every extension
• Violate pre-conditions
• Try different ordering of steps
• Post-Conditions can be used as an Oracle