Tutorial 1, Q1 solution

1
Tutorial 1, Q1 solution

• The five scales
• Nominal scale serves only for classification. The
  order of the classes is unimportant as is the
  difference between them. A renaming of classes is
  a valid transformation of scales. Example blood
  types.
• The Ordinal Scale is for classification where the
  order is important but the difference between
  classes is arbitrary. Any monotonic
  transformation is meaningful. Example top ten
  whatevers.
• Interval scale order important and differences
  between classes meaningful. Any affine
  transformation (S' kS c) is meaningful, as in
  the conversion from C to F. Eg. temperature in
  Celsius (F (9/5)C 32).
• Ratio scale as interval scale but also has
  meaningful zero. Any Linear transformation (S'
  kS) is meaningful as in conversion of currencies.
  
• The Absolute scale is just used for counting.
  Meaningful zero and differences between values.
  Only the identity transformation is valid. Eg.
  Lines of Code (given a particular definition of
  LoC).

2
Tutorial 1, Q1 solution

• 100 C is the boiling point of water" is a
  meaningful. Degrees C is an interval scale since
  each degree is equal but the zero is arbitrary.
  Transformation to Fahrenheit scale is example of
  an affine transformation which is valid for
  interval scales.
• Today is twice as hot as yesterday" is unlikely
  to be meaningful as weather temperature is
  measured in C or F which are not ratio scales
  (which would be required for twice" to be
  meaningful). Differences are meaningful however,
  so if today is 10 degrees C, and yesterday was 5
  degrees C, then it would be meaningful to say
  that today is 5 degrees hotter than yesterday.
• The FT index is an example of an indirect measure
  (calculated from the value of lots of different
  shares). The calculation is made such that the
  resulting value is on a ratio scale. The zero is
  meaningful (the shares have value) and the
  differences are equal.
• Depending on its precise definition, Line of Code
  is probably an absolute scale so the sentence is
  meaningful. Transformations which convert between
  lines of code measures for different languages
  may be empirically substantiated but are not
  valid for an absolute scale.
• Elapsed time is a ratio scale made by subtracting
  two points in time. Points in time are measured
  on interval scale (eg. 3/10/2002 101130).
  Transformations of elapsed time such as 1 month
  20 working days are valid for elapsed time (ratio
  scale). But (31 Jan 1997) 31 (1st Jan 1997)
  is not valid since date is an interval scale.
• Cost is a ratio scale. Assuming we are talking
  about something like cost of maintenance per
  year, then the sentence is meaningful.
• Complexity however defined is likely to be at
  least an ordinal scale so the sentence is OK.

3
Tutorial 1 Q2 Solution

• Check the notes for week two. There are lots of
  problems with the Lines of Code LoC measure
• Do blank lines count?
• What about comments?
• Compiler directives?
• Declarations?
• Procedure headings?
• Imports, exports?
• What about side effects in expressions you can
  get lots of effects in a single expression, for
  example in Java you can write
• X i,j,k,--p,tt4
• This assigns to X, but has also 5 side effects.

4
Control Flowgraphs
Loop
Atomic
Sequence
Conditional
Start or Stop node
5
Tutorial 1, Q3 solution

• Draw the CFG

• IF(...)THEN x1 ELSE x2 FI y50 IF
  (...) THEN z1 ELSE DO WHILE (...) zz1 OD
  FI y0

6
Tutorial 1, Q3 solution

• To calculate number of nodes
• M(S(C(A,A),S(A,S(C(A,L(A)),A))))
• M(C(A,A))M(S(A,S(C(A,L(A)),A)))-1
• M(A) M(A) M(A) M(S(C(A,L(A)),A)) -1 -1
• 2 2 2 M(C(A,L(A))) M(A) -1 -1 -1
• 2 2 2 M(A) M(L(A)) 2 -1 -1 -1
• 2 2 2 2 M(A) 1 2 -1 -1 -1
• 2 2 2 2 2 1 2 -1 -1 -1
• 10
• For number of edges
• M(S(C(A,A),S(A,S(C(A,L(A)),A))))
• M(C(A,A)) M(S(A,S(C(A,L(A)),A)))
• M(A) M(A) 2 M(A) M(S(C(A,L(A)),A))
• 1 1 2 1 M(C(A,L(A))) M(A)
• 1 1 2 1 M(A) M(L(A)) 2 1
• 1 1 2 1 1 M(A) 2 2 1
• 1 1 2 1 1 1 2 2 1
• 12

10
12
4
4
7
8
2
1
1
2
2
1
6
7
5
6
2
3
1
3
2
1
2
1
7
Another (simpler) way

• Sandip Hirani noticed a simpler way in the
  tutorial (well done Sandip)

10
12
4
4

• Number of Nodes
• FA() 2
• Fs(m1,m2) m1 m2 - 1
• FC(m1,m2) m1 m2
• FL(m1) m1 1

7
8
2
1
1
2
2
1
6
7
5
6
2
3
1
3
6 As 3 Ss 2 Cs 1 L
6 2 3 -1 2 0 1 1
2
1
2
1
8
Control Flowgraph Based Testing

• Can use CFGs to guide selection of tests for
  white box or structural testing
• White box testing is typically used for unit
  testing

9
What is a test?

• Testing (in general)
• For specification S and program P
• A test case is an input-output pair (i,S(i))
• A test is a comparison S(i)P(i)
• A test suite is a set of test cases which in some
  sense cover the possibilities
• The static measures above form basis of test
  strategies.

10
Oracle problem

• For input I, what should the output be?
• The tester must know
• but human-based testing is slow and expensive
• If we want to automate then an oracle must tell
  us
• We can use a model (IBM do this, for example)
• We can look for known wrong behaviour
• We can use capture/replay for regression testing
  (lots of tools offer this)

11
When is it Tested Enough?

• When should we stop testing?
• This is a very hard question to answer
• We can only talk about when we have covered
  certain aspects of the software
• In white box testing, these aspects are
  structures in the source code
• We seek to cover (or exercise) them
• We define a coverage criterion
• also known as eth test adequacy criterion

12
Test Coverage

• Coverage is usually given in terms of percentage
  of the chosen structurs covered at least once
• Examples
• Statement coverage - all nodes in CFG
• Branch coverage - all edges in CFG
• Eg 40 statement coverage means 40 of nodes in
  CFG are exercised by test suite

13
Example Test Strategies

• A testing strategy can be based on coverage of
  one or more of
• Statements
• Branches
• Simple paths (No edge more than once)
• Visit each loop (Simple paths plus loops once)
• All paths (Not feasible in general)
• Note No strategy can guarantee adequate testing
  (except exhaustive testing, which is infeasible)

14
Statement and branch coverage

• Correspond to nodes and edges of CFG
• For our language they are equivalent
• If we allowed simple conditional (without else)
  then branch coverage is strictly stronger than
  statement coverage

15
Number of Simple Paths

• A simple path is one in which no edge is
  traversed more than once
• nsp(A) 1
• nsp(S(P1,P2)) nsp(P1) x nsp(P2)
• nsp(C(P1,P2)) nsp(P1) nsp(P2)
• nsp(L(P1)) nsp(P1)1

16
So in the overall CFG, how many simple paths?
nsp(S(P1,P2)) nsp(P1) x nsp(P2)
17
Number of Simple Paths example
18
Visit Each Loop

• Each loop is visited once (for each path in its
  body) and zero times
• vl(A) 1
• vl(S(P1,P2)) vl(P1) x vl(P2)
• vl(C(P1,P2)) vl(P1) vl(P2)
• vl(L(P1)) vl(P1) 1
• Same as number of simple paths

19
All paths

• In general unbounded number of tests
• But if set the maximum number of iterations, k,
  of each loop, Li
• Then
• ap(A) 1
• ap(S(P1,P2)) ap(P1) x ap(P2)
• ap(C(P1,P2)) ap(P1) ap(P2)
• ap(Li(P1)) S (ap(P1))i
• i 0,,k

20
How do we find test sets?

• Given a test strategy it is not easy to find test
  cases that exercise the required paths
• Even for Statement Coverage some parts of the
  code may be unreachable
• A single path can achieve Branch Coverage for
  while(...) do some complex program but
  unlikely to be possible in practice

21
Unreachable nodes

• Consider this code fragment
• if agtb then
• if bgtc then
• if agtc then S1
• else S2
• What can you say about S1 and S2?

22
A glimpse ahead

• next week
• we will learn about evolutionary testing
• Here we will see how we an automate the
  generation of test cases to cover these adequacy
  criteria

23
Data Flow Based Testing

• The flow of data in a program is vital to the
  computation it performs
• Analysis of the definition and use of data in
  execution paths
• Annotate CFG with mode of use of a particular
  variable
• Choose test cases which exercise particular kinds
  of usage

24
Kinds of use of variables

• Use the following abbreviation for status
• d - defined (written, created or initialised)
• k - killed (undefined or released)
• u - used for
• uc - used in calculation
• up - used in predicate

We shall be concerned primarily with d and u
25
Example

• For the variable x in
• IF(...) THEN
• x1 ELSE
• x2 FI
• y50
• IF(x1)THEN
• x2 ELSE
• DO ... OD
• FI
• xx-1

Notice that we distinguish between predicate and
computational uses
26
Possible Data Flows

• dd - harmless but suspicious
• dk - harmless but suspicious
• du - normal usage
• kd not OK (very suspicious)
• kk - suspicious
• ku - a bug
• ud OK possibly suspicious (if first u)
• uk - normal
• uu - normal

27
Some Dataflow Test Strategies

• ADUP - All du paths
• AU - All uses
• APUC - All predicate use/some computation
• ACUP - All computation use/some predicate
• AD - All definitions (single use)
• APU,ACU - All predicate,computation uses

28
Some Dataflow Test Strategies

• ADUP - All du paths
• AU - All uses
• APUC - All predicate use/some computation
• ACUP - All computation use/some predicate
• AD - All definitions (single use)
• APU,ACU - All predicate,computation uses

29
Du paths

• A path from node x to node y is definition clear
  for a variable v iff for all nodes apart from x
  and y on the path, there is no assignment to v.
• A du-path from node x to node y for a variable v
  is a simple path from x to y which is definition
  clear for v and which assigns to v at x and uses
  v at y
• Definition clear means we dont redefine the
  variable along the way
• Recall Simple means no edge is traversed more
  than once

30
Recall Our Example
For just the variable x in IF(...) THEN
x1 ELSE x2 FI y50
IF(x1)THEN x2 ELSE DO ... OD
FI xx-1
31
Recall Our Example
For just the variable x in IF(...) THEN
x1 ELSE x2 FI y50
IF(x1)THEN x2 ELSE DO ... OD
FI xx-1
32
A special case?

• Can there be a du path from a node to itself?
• Consider xx1
• Suppose that this is the only node in the body of
  a loop.

33
Note this special case

• There can be a du path from a node to itself
• Such a node would, at least, have to define and
  use the same variable and would have to occur in
  a loop

34
All du Paths ADUP

• For every variable and
• for every definition, d, of that variable and
• for every use, u, of d and
• for every du-path from d to u
• there is test which exercises that path.

35
Recall Our Example
For just the variable x in IF(...) THEN
x1 ELSE x2 FI y50
IF(x1)THEN x2 ELSE DO ... OD
FI xx-1
36
All du Paths - example
1)
2)
3)
4)
5)
6)
7)
37
All du Paths - example
Notice that each du pair may have more than one
du path
1)
2)
3)
4)
5)
6)
7)
38
All uses strategy - AU

• Same as all du paths except we only require at
  least one path from each definition to each use

For every variable and for every definition, d,
of that variable and for every use, u, of d
and for at least one du-path from d to u
there is test which exercises that path.
39
AU - example
1)
2)
3)
4)
5)
6)
7)
40
AU - example
Notice we are allowed a choice here
1)
2)
3)
4)
5)
6)
7)
41
AU - example
Indeed, at least one means we could have all
1)
2)
3)
4)
5)
6)
7)
42
Meeting the adequacy criterion

• We often seek a minimal set of test data which
  meets the adequacy criterion
• Notice that there may be more than one minimal
  set
• as in the previous example for AU testing
• Choosing a minimal set makes less work for the
  tester
• But then the unit is less tested as a result
• But we can be more precise about just what
  testing has achieved

43
Relaxing the criterion

• These criteria are called adequacy criteria
• The AU criterion relaxes the All du paths
  criterion
• The next two criteria a going to be different
  relaxations of the AU criterion, so in our
  example, we will start with the set of paths we
  had from AU
• That is

44
AU - example
1)
2)
3)
4)
6)
45
All predicate/some computational use APUC

• As AU except
• only paths to predicate uses
• plus paths to computational uses if any
  definitional use not covered otherwise
• In our example this makes no difference
• ACUP - same but vice versa

46
APUC - example
1)
2)
3)
4)
6)
47
APUC - example
1)
2)
3)
4)
6)
48
All computational/some predicate use ACUP

• As AU except
• only paths to computational uses
• plus paths to predicate uses if any definitional
  use not covered otherwise
• In our example this does make a difference

49
ACUP - example
This definition to computational use path
contains a path from definition to predicate use
1)
2)
3)
4)
6)
50
ACUP - example
So we can ignore test case 1)
1)
2)
3)
4)
6)
51
ACUP - example
Similarly, test case 6) covers test case 2)
1)
2)
3)
4)
6)
52
(No Transcript)
53
APU - example
Now we dont need 3), 4) and 6)
1)
2)
3)
4)
6)
54
(No Transcript)
55
ACU - example
For AU we needed 1) and 2) However, for ACU will
still need all three of 3), 4) and 6)
1)
2)
3)
4)
6)
56
All Definitions - AD

• All definitions covered by some use
• 1), 2) and 3) would suffice
• Also 3), 4) and 6) would suffice

57
AD - example
Which is better? 1), 2) and 3) or 3), 4) and 6) ?
1)
2)
3)
4)
6)
58
Relative power of test strategies
Statement
Provided all branches from predicate are
exercised
59
In industry at the time of writing

• Statement and branch coverage are used widely
• Typically we seek, at least, branch coverage
• For example avionics and other standards require
  it
• Tools exist to measure coverage of all kinds
  described here
• Testers (and developers) use these techniques at
  the unit level
• But they are not the only techniques, merely one
• The tools do not generally automate the
  generation of test cases
• This is where the current bleeding edge lies
• Automation is the key

60
Tutorial 2, Q1

• Consider the following simple program (in no
  particular language) to find the smallest prime
  factor of a number
• smallest(int p) / precondition p gt 1 /
• int q 2
• while(p mod q gt 0 AND q lt sqrt(p) )
• q q1
• if (p mod q 0)
• then
• print(q,''is the smallest factor'')
• else
• print(p, is prime'')

• Draw the control for this program and (if it
  helps) the structural tree-form of this program.
  Treat the pre condition as an assignment, treat
  prints as simple nodes (like assignments)
• Calculate
• number of nodes
• number of edges
• number of simple paths
• all paths with k2

61
Tutorial 2, Q2

• Consider the following simple program (in no
  particular language) to find the smallest prime
  factor of a number
• smallest(int p) / precondition p gt 1 /
• int q 2
• while(p mod q gt 0 AND q lt sqrt(p) )
• q q1
• if (p mod q 0)
• then
• print(q,''is the smallest factor'')
• else
• print(p, is prime'')

• Draw the control for this program and annotate
  the nodes to indicate the definitions/uses of
  variables p and q.
• Label the nodes and give All DU Paths for p and
  q.
• Which of these paths are required for AU, APU,
  ACU, and AD coverage.
• Give a test suite which gives 100 coverage for
  All DU Paths.
• Explain whether or not the all predicate uses"
  test set is sufficient for branch coverage of
  this program?

62
References Phyllis Frankl and Elaine Weyuker. An
Applicable Family of Data Flow Testing
CriteriaIEEE Transactions on Software
Engineering, 14(10), pp. 1483-1498, October 1988.
This is the seminal paper that introduced the
idea of data flow testing. The BCS Special
Interest Group on testing has produced an
excellent document, which describes testing
standards. The document is now a British standard
(BS7925-2), for which you have to pay a fee.
However, the draft of the documents, is available
for free download from http//www.testingstandards
.co.uk/ The BCS special interest group on
testing is another useful source of information
http//www.sigist.org.uk/