Testing and Debugging

1
Testing and Debugging

• Lecture 8,
• Programmeringsteknik del A

2
Whats the Difference?
Testing means trying out a program which is
believed to work, in an effort to gain confidence
that it is correct. If no errors are revealed by
thorough testing, then, probably, relatively few
errors remain. Debugging means observing a
program which is known not to work, in an effort
to localise the error. When a bug is found and
fixed by debugging, testing can be resumed to see
if the program now works. This lecture describes
recently developed tools to help with each
activity.
3
Debugging
median xs isort xs !! (length xs div 2) isort
foldr insert insert x x insert x
(yys) xlty xyys xgty yxys
Heres a program with a bug
A test reveals median doesnt work.
Mediangt median 8,4,6,10,2,7,3,5,9,1 2 Mediangt
isort 8,4,6,10,2,7,3,5,9,1 1,8,4,6,10,2,7,3,5,9

We start trying functions median calls. isort
doesnt work either.
4
Debugging Tools
The Manual Approach We choose cases to try, and
manually explore the behaviour of the program, by
calling functions with various (hopefully
revealing) arguments, and inspecting their
outputs. The Automated Approach We connect a
debugger to the program, which lets us observe
internal values, giving us more information to
help us diagnose the bug.
5
The Haskell Object Observation Debugger
Provides a function which collects observations
of its second argument, tagged with the String,
and returns the argument unchanged. Think of it
as like connecting an oscilloscope to the
program the programs behaviour is unchanged,
but we see more. (You need to import the library
which defines observe in order to use it
add import Observe at the start of your program).
observe String -gt a -gt a
6
What Do Observations Look Like?
Mediangt sum observe "nn" (nn) n lt-
1..4 30 gtgtgtgtgtgtgt Observations ltltltltltlt nn 1
4 9 16
We add a probe to the program
The values observed are displayed, titled with
the name of the observation.
7
Observing a List
Mediangt sum (observe "squares" nn n lt-
1..4) 30 gtgtgtgtgtgtgt Observations
ltltltltltlt squares (1 4 9 16 )
Observing the entire list lets us see the order
of values also.
Now there is just one observation, the list
itself. Lists are always observed in cons form.
8
Observing a Pipeline
Mediangt (sum . observe "squares" . map (\x-gtxx))
1..4 30 gtgtgtgtgtgtgt Observations ltltltltltlt squares
(1 4 9 16 )
We can add observers to pipelines --
long compositions of functions -- to see
the values flowing between them.
9
Observing Counting Occurrences
countOccurrences map (\ws -gt (head ws,
length ws)) . observe "after groupby" .
groupBy () . observe "after sort" . sort
. observe "after words . words
Add observations after each stage.
10
Observing Counting Occurrences
Maingt countOccurrences "hello clouds hello
sky" ("clouds",1),("hello",2),("sky",1) gtgtgtgtgtgtgt
Observations ltltltltltlt after groupby (("clouds"
) ("hello" "hello" ) ("sky" )
) after sort ("clouds" "hello" "hello"
"sky" ) after words ("hello" "clouds"
"hello" "sky" )
11
Observing Consumers
An observation tells us not only what value
flowed past the observer -- it also tells us how
that value was used! Maingt take 3 (observe "xs"
1..10) 1,2,3 gtgtgtgtgtgtgt Observations
ltltltltltlt xs (1 2 3 _)
The _ is a dont care value -- certainly some
list appeared here, but it was never used!
12
Observing Length
Maingt length (observe "xs" (words "hello
clouds")) 2 gtgtgtgtgtgtgt Observations ltltltltltlt xs (_
_ )
The length function did not need to inspect the
values of the elements, so they were not observed!
13
Observing Functions
We can even observe functions themselves! Maingt
observe "sum" sum 1..5 15 gtgtgtgtgtgtgt Observations
ltltltltltlt sum \ (1 2 3 4 5 ) -gt
15
observe sum sum is a function, which is applied
to 1..5
We see arguments and results, for the calls
which actually were made!
14
Observing foldr
Recall that foldr () 0 1..4 1 (2 (3
(4 0))) Lets check this, by observing the
addition function. Maingt foldr (observe "" ())
0 1..4 10 gtgtgtgtgtgtgt Observations ltltltltltlt \
4 0 -gt 4 , \ 3 4 -gt 7 , \ 2 7 -gt 9 , \ 1
9 -gt 10
15
Observing foldl
We can do the same thing to observe foldl, which
behaves as foldl () 0 1..4 (((0 1) 2)
3) 4 Maingt foldl (observe "" ()) 0
1..4 10 gtgtgtgtgtgtgt Observations ltltltltltlt \ 0
1 -gt 1 , \ 1 2 -gt 3 , \ 3 3 -gt 6 , \ 6 4
-gt 10
16
How Many Elements Does takeWhile Check?
takeWhile isAlpha hello clouds hello sky
hello takeWhile isAlpha selects the alphabetic
characters from the front of the list. How many
times does takeWhile call isAlpha?
17
How Many Elements Does takeWhile Check?
Maingt takeWhile (observe "isAlpha" isAlpha)
"hello clouds hello sky" "hello" gtgtgtgtgtgtgt
Observations ltltltltltlt isAlpha \ ' ' -gt False
, \ 'o' -gt True , \ 'l' -gt True , \ 'l'
-gt True , \ 'e' -gt True , \ 'h' -gt True
takeWhile calls isAlpha six times -- the last
call tells us its time to stop.
18
Observing Recursion
fac 0 1 fac n ngt0 n fac (n-1)
Maingt observe "fac" fac 6 720 gtgtgtgtgtgtgt
Observations ltltltltltlt fac \ 6 -gt 720
We observe this use of the function.
We did not observe the recursive calls!
19
Observing Recursion
fac observe "fac" fac' fac' 0 1 fac' n ngt0
n fac (n-1)
Maingt fac 6 720 gtgtgtgtgtgtgt Observations
ltltltltltlt fac \ 6 -gt 720 , \ 5 -gt 120 , \
4 -gt 24 , \ 3 -gt 6 , \ 2 -gt 2 , \ 1 -gt
1 , \ 0 -gt 1
We observe all calls of the fac function.
20
Debugging median
median xs observe "isort xs" (isort xs) !!
(length xs div 2) Maingt median
4,2,3,5,1 2 gtgtgtgtgtgtgt Observations ltltltltltlt isort
xs (1 4 2 3 5 )
Wrong answer the median is 3
Wrong (unsorted) result from isort
21
Debugging isort
isort Ord a gt a -gt a isort foldr
(observe "insert" insert) Maingt median
4,2,3,5,1 2 gtgtgtgtgtgtgt Observations
ltltltltltlt insert \ 1 -gt 1 , \ 5 (1
) -gt 1 5 , \ 3 (1 5 ) -gt 1 3
5 , \ 2 (1 3 5 ) -gt 1 2 3
5 , \ 4 (1 2 3 5 ) -gt 1 4 2
3 5
All well, except for this case
22
Debugging insert
insert x x insert x (yys) xlty
observe "xlty" (xyys) xgty observe "xgty"
(yxys) Maingt median 4,2,3,5,1 2 gtgtgtgtgtgtgt
Observations ltltltltltlt xgty (1 5 ) (1 3
5 ) (1 2 3 5 ) (1 4 2 3
5 )
Observe the results from each case
Only the second case was used!
23
The Bug!
I forgot the recursive call insert x
x insert x (yys) xlty xyys xgty
yinsert x ys Maingt median 4,2,3,5,1 3
Bug fixed!
The right answer
24
Summary

• The observe function provides us with a wealth of
  information about how programs are evaluated,
  with only small changes to the programs
  themselves.
• That information can help us understand how
  programs work (foldr, foldrl, takeWhile etc.)
• It can also help us see where bugs are.

25
Testing
Testing means trying out a program which is
believed to work, in an effort to gain confidence
that it is correct. Testing accounts for more
than half the development effort on a large
project (Ive heard all from 50-80). Fixing a
bug in one place often causes a failure somewhere
else -- so the entire system must be retested
after each change. At Ericsson, this can take
three months!
26
Hacking vs Systematic Testing
Hacking Systematic testing

• Try some examples until the software seems to
  work.
• Record test cases, so that tests can be repeated
  after a modification (regression testing).
• Document what has been tested.
• Establish criteria for when a test is successful
  -- requires a specification.
• Automate testing as far as possible, so you can
  test extensively and often.

27
QuickCheck A Tool for Testing Haskell Programs

• Based on formulating properties, which
• can be tested repeatedly and automatically
• document what has been tested
• define what is a successful outcome
• are a good starting point for proofs of
  correctness
• Properties are tested by selecting test cases at
  random!

28
Random Testing?
Is random testing sensible? Surely carefully
chosen test cases are more effective?
By taking 20 more points in a random test,
any advantage a partition test might have had is
wiped out. D. Hamlet

• QuickCheck can generate 100 random test cases in
  less time than it takes you to think of one!
• Random testing finds common (i.e. important!)
  errors effectively.

29
A Simple QuickCheck Property
prop_Sort Int -gt Bool prop_Sort xs ordered
(sort xs)
Check that the result of sort is ordered.
Random values for xs are generated.
Maingt quickCheck prop_Sort OK, passed 100 tests.
The tests were passed.
30
Some QuickCheck Details
We must import the QuickCheck library.
import QuickCheck prop_Sort Int -gt
Bool prop_Sort xs ordered (sort xs)
The type of a property must not be polymorphic.
We give properties names beginning with prop_
so we can easily find and test all the properties
in a module.
Maingt quickCheck prop_Sort OK, passed 100 tests.
quickCheck is an (overloaded) higher
order function!
31
A Property of insert
prop_Insert Int -gt Int -gt Bool prop_Insert x
xs ordered (insert x xs) Maingt quickCheck
prop_Insert Falsifiable, after 4
tests -2 5,-2,-5
Whoops! This list isnt ordered!
32
A Corrected Property of insert
Result is no longer a simple Bool.
prop_Insert Int -gt Int -gt Property prop_Inser
t x xs ordered xs gt ordered (insert x
xs) Maingt quickCheck prop_Insert OK, passed
100 tests.
Read it as implies if xs is ordered, then so
is (insert x xs).
Discards test cases which are not ordered.
33
Using QuickCheck to Develop Fast Queue Operations

• What were going to do
• Explain what a queue is, and give slow
  implementations of the queue operations, to act
  as a specification.
• Explain the idea behind the fast implementation.
• Formulate properties that say the fast
  implementation is correct.
• Test them with QuickCheck.

34
What is a Queue?
Join at the back
Leave from the front

• Examples
• Files to print
• Processes to run
• Tasks to perform

35
What is a Queue?

• A queue contains a sequence of values. We can add
  elements at the back, and remove elements from
  the front.
• Well implement the following operations
• empty Queue a -- an empty queue
• isEmpty Queue a -gt Bool -- tests if a queue
  is empty
• add a -gt Queue a -gt Queue a -- adds an element
  at the back
• front Queue a -gt a -- the element at the
  front
• remove Queue a -gt Queue a -- removes an
  element from the front

36
The Specification Slow but Simple
type Queue a a empty isEmpty q
qempty add x q qx front (xq) x remove
(xq) q
Addition takes time depending on the number of
items in the queue!
37
The Idea Store the Front and Back Separately
b
c
d
e
f
g
h
i
a
j
Old
Fast to remove
Slow to add
Fast to remove
Periodically move the back to the front.
b
c
d
e
a
New
i
h
g
f
j
Fast to add
38
The Fast Implementation
type Queue a (a,a) flipQ (,b) (reverse
b,) flipQ (xf,b) (xf,b) emptyQ
(,) isEmptyQ q qemptyQ addQ x (f,b)
(f,xb) removeQ (xf,b) flipQ (f,b) frontQ
(xf,b) x
Make sure the front is never empty when the back
is not.
39
Relating the Two Implementations
What list does a double-ended queue represent?
retrieve Queue a -gt a retrieve (f, b) f
reverse b

• What does it mean to be correct?
• retrieve emptyQ empty
• isEmptyQ q isEmpty (retrieve q)
• retrieve (addQ x q) add x (retrieve q)
• retrieve (removeQ q) remove (retrieve q) and
  so on.

40
Using Retrieve Guarantees Consistent Results
Example frontQ (removeQ (addQ 1 (addQ 2
emptyQ))) front (retrieve (removeQ (addQ 1
(addQ 2 emptyQ)))) front (remove (retrieve
(addQ 1 (addQ 2 emptyQ)))) front (remove
(add 1 (retrieve (addQ 2 emptyQ)))) front
(remove (add 1 (add 2 (retrieve emptyQ))))
front (remove (add 1 (add 2 empty)))
41
QuickChecking Properties
prop_Remove Queue Int -gt Bool prop_Remove q
retrieve (removeQ q) remove (retrieve
q) Maingt quickCheck prop_Remove 4 Program
error removeQ (,sized_v1740
(instArbitrary_v1
Removing from an empty queue!
42
Correcting the Property
prop_Remove Queue Int -gt Property prop_Remove
q not (isEmptyQ q) gt retrieve
(removeQ q) remove (retrieve q) Maingt
quickCheck prop_Remove 0 Program error removeQ
(,Arbitrary_arbitrary instArbitrary
How can this be?
43
Making Assumptions Explicit
We assumed that the front of a queue will never
be empty if the back contains elements! Lets
make that explicit goodQ Queue a -gt
Bool goodQ (,) True goodQ (xf,b)
True goodQ (,xb) False prop_Remove q
not (isEmptyQ q) goodQ q gt retrieve
(removeQ q) remove (retrieve q)
NOW IT WORKS!
44
How Do We Know Only Good Queues Arise?
Queues are built by add and remove
addQ x (f,b) (f,xb) removeQ (xf,b) flipQ
(f,b)
New properties prop_AddGood x q goodQ q gt
goodQ (addQ x q) prop_RemoveGood q not
(isEmptyQ q) goodQ q gt goodQ (removeQ q)
45
Whoops!
Maingt quickCheck prop_AddGood Falsifiable, after
0 tests 2 (,)
addQ x (f,b) (f,xb) removeQ (xf,b) flipQ
(f,b)
See the bug?
46
Whoops!
Maingt quickCheck prop_AddGood Falsifiable, after
0 tests 2 (,)
addQ x (f,b) flipQ (f,xb) removeQ (xf,b)
flipQ (f,b)
47
Looking Back

• Formulating properties let us define precisely
  how the fast queue operations should behave.
• Using QuickCheck found a bug, and revealed hidden
  assumptions which are now explicitly stated.
• The property definitions remain in the program,
  documenting exactly what testing found to hold,
  and providing a ready made test-bed for any
  future versions of the Queue library.
• We were forced to reason much more carefully
  about the programs correctness, and can have
  much greater confidence that it really works.

48
Summary

• Testing is a major part of any serious software
  development.
• Testing should be systematic, documented, and
  repeatable.
• Automated tools can help a lot.
• QuickCheck is a state-of-the-art testing tool for
  Haskell.

49
The remaining slides discuss an important
subtlety when using QuickCheck
50
Testing the Buggy insert
prop_Insert Int -gt Int -gt Property prop_Inser
t x xs ordered xs gt ordered (insert x
xs) Maingt quickCheck prop_Insert Falsifiable,
after 51 tests 5 -3,4
Why so many tests?
Yields -3,5,4
51
Observing Test Data
Collect values during testing.
prop_Insert Int -gt Int -gt Property prop_Inser
t x xs ordered xs gt collect (length
xs) (ordered (insert x xs)) Maingt quickCheck
prop_Insert OK, passed 100 tests. 43 0. 37
1. 11 2. 8 3. 1 4.
Random lists which happen to be ordered are
likely to be short!
Distribution of length xs.
52
A Better Property
prop_Insert Int -gt Property prop_Insert x
forAll orderedList (\xs -gt collect (length
xs) (ordered (insert x xs))) Maingt quickCheck
prop_Insert2 OK, passed 100 tests. 22 2. 15
0. 14 1. 8 6. 8 5.
Read this as ?xs?orderedList.
8 4. 8 3. 4 8. 3 9. 3 11.
2 12. 2 10. 1 7. 1 30. 1 13.
53
What is forAll?
A generator for test data of type a.
A higher order function! forAll (Show a,
Testable b) gt Gen a -gt (a -gt b) -gt Property
A function, which given a generated a, produces
a testable result.
forAll orderedList (\xs -gt collect (length xs)
(ordered (insert x xs)))
54
What is orderedList?
A test data generator orderedList Gen Int
A generator for a, behaves like IO a a command
producing a.
Some primitive generators arbitrary Arbitrary
a gt Gen a oneof Gen a -gt Gen a frequency
(Int,Gen a) -gt Gen a
55
Defining orderedList
We can use the do syntax to write generators,
like IO, but we cannot mix Gen and IO!
orderedList Gen Int orderedList do n
lt- arbitrary listFrom n where
listFrom n frequency
(1,return ), (4,do m
lt- arbitrary ns lt-
listFrom (nabs m) return (nns))
Choose an n, and make a list of elements gt n.
20 of the time, just stop.
80 of the time, generate another list ns of
elements gtn, and return nns.
Choose a number gt n.