Debugging

1
Debugging

• CS130
• Lecture 11

2
Administrivia

• HW3 was due today
• HW solutions will be online by tonight
• In class, open book, closed computer exam 5/18
• 90 minutes
• Question 1 Short answers
• Questions 2-5 Longer (but not too long!)

3
Guest Lecture

• 5/20 Dr. Ani Nahapetian from UCLA Center for
  Wireless Health
• Software Engineering for Safety Critical Medical
  Devices
• This is an extremely important emerging area
• Attendance is mandatory!

4
Debugging

• Purpose of testing is to find failure
• Failure symptom - situation that gives wrong
  result
• Produces bug reports, tests that fail
• Purpose of debugging is to find and correct the
  fault
• Fault cause - mistake in the code
• Produces changes to the code

5
Psychology of Debugging

• Sensitive, emotional subject
• Denial, finger pointing, lame excuses
• Debugging is problem solving and attack it as
  such
• Fix the problem, even if it is not your fault
• Debugging mindset
• Easy to panic approaching deadline or nervous
  boss
• Important rule for debugging next slide

6

• Dont Panic!

7
Why is Debugging So Hard?

• Symptom and cause can be remote
• Spatially and/or temporally remote
• Symptom might be intermittent
• E.g., timing problems, or hardware interaction
• Bad feature interactions
• May be hard to reproduce the erroneous run
• Requires unusual skills
• Combines brain teasers with the annoying
  recognition that you made a mistake
• Large variance in debugging skills

8
Debugging Strategies

• Make the bug reproducible
• Create a simple test that reproduces the bug
• Any change to the test changes the error behavior
• Tracing
• Add print statements and read the output
• Backtracking
• Work backwards from where symptom appears
• Cause elimination
• Hypothesize causes, then create tests to validate
  
• Rubber Ducking
• Explain it to someone else
• Works because you explicitly say things that you
  have taken for granted

9
Tracing

• Watch execution of program
• Determine when it goes wrong
• Print statements
• Single stepping
• Trace package
• Breakpoints
• Assertions / watchpoints

10
Backtracking

• Determine a failure
• Find fault by backing up to first failure
• Repeatedly run program with new print statements
• What could have caused this? Put in print
  statements to test each possibility.
• Remove unneeded print statements
• Works well when you have reproducibility

11
Backtracking

• Reverse execution 1
• Research
• Record each change to variables
• Back up execution by undoing change
• Reverse execution 2
• Essentially, checkpoint/replay
• Save checkpoints during computation
• That is, save entire state of computation
• To travel to time t
• Return to last checkpoint before time t
• Rerun computation up to time t

12
Cause elimination

• What could cause this bug?
• List all possibilities
• Check each one
• By examining program
• By performing experiments (tracing)
• Requires knowledge of system and the errors that
  people make
• Do not assume
• Do not assume that bug exists in OS, compiler, or
  library
• Easy to look into your code
• May be you are calling the library incorrectly

13
Debugging expertise

• Good debuggers know the model of the system
• in their head
• can imagine how the system executes
• can perform mental experiments
• know their limits

14
Debugging expertise

• Good debuggers know the kinds of errors people
  make
• C, C null pointer and other pointer errors
• C, C losing memory, using memory twice
• not closing a file
• not initializing an object correctly
• sending message to wrong object

15
GDB

• The Gnu DeBugger
• A very good debugger
• Widely used
• Runs on everything
• Also, a classic implementation
• Mostly standard debugger technology
• Similar debuggers exist for many languages

16
Debugger Architecture

• Three major pieces
• User interface
• Symbol piece
• Mapping from source to object code
• Execution piece
• Manipulating, running object code

17
Concepts

• Debuggers use compiler terminology
• Some cs132 background helpful here
• Source code
• The program you write
• Object code
• The compiled program

• Symbols
• Variable names, procedure names

18
User Interface

• Not much to say, except that its classic Unix
• gdb
• (gdb) file foo
• Reading symbols from foo done
• (gdb) break main
• Breakpoint 1 at 0x40107e file foo.c line 10
• (gdb) run
• Starting program /home/ksen/foo.exe
• Breakpoint 1, main() at foo.c10
• 10 printf(Hello world!)
• DDD Better user interface to gdb

19
Features

• Breakpoints
• Single stepping
• Host/Target

20
Breakpoints

• The fundamental debugging primitive
• How does it work?
• Via an object code rewriting hack
• To stop at line 10, write an invalid opcode at
  line 10
• Trap resulting fault, recover, and switch to the
  UI

21
Single Stepping

• To single step
• Set breakpoint at next instruction
• Resume execution
• Trap exception, clear breakpoint, repeat
• Or
• Use software interpreter
• Interpret instructions to next source statement

22
Other Features

• Other features based on breakpoints
• Skip over function call
• Put breakpoint at the return site
• Break on nth execution of a statement
• Break when an expression becomes true !
• Break when an expression changes value !
• What do you do when you hit a breakpoint
• Print the call stack
• Inspect data values
• Etc.

23
Just-in-Time Debugging

• Start the debugger when the program fails an
  assertion
• You do not have to reproduce the run
• Microsoft Visual Studio does this
• Can simulate this with gdbs ability to attach to
  a process
• On assertion failure, you trigger an external
  process that starts gdb

24
Debugging Conclusions

• Debugging is hard
• Can increase dramatically effectiveness with the
  right tools
• Next Look at two interesting research directions
• Delta Debugging
• Co-operative Bug Isolation

25
Delta Debugging

• Minimize error causing input

26
How do we go from this
lttd alignleft valigntopgtltSELECT NAME"op sys"
MULTIPLE SIZE7gtltOPTION VALUE"All"gtAllltOPTION
VALUE"Windows 3.1"gtWindows 3.1ltOPTION
VALUE"Windows 95"gtWindows 95ltOPTION
VALUE"Windows 98"gtWindows 98ltOPTION
VALUE"Windows ME"gtWindows MEltOPTION
VALUE"Windows 2000"gtWindows 2000ltOPTION
VALUE"Windows NT"gtWindows NTltOPTION VALUE"Mac
System 7"gtMac System 7ltOPTION VALUE"Mac System
7.5"gtMac System 7.5ltOPTION VALUE"Mac System
7.6.1"gtMac System 7.6.1ltOPTION VALUE"Mac System
8.0"gtMac System 8.0ltOPTION VALUE"Mac System
8.5"gtMac System 8.5ltOPTION VALUE"Mac System
8.6"gtMac System 8.6ltOPTION VALUE"Mac System
9.x"gtMac System 9.xltOPTION VALUE"MacOS X"gtMacOS
XltOPTION VALUE"Linux"gtLinuxltOPTION
VALUE"BSDI"gtBSDIltOPTION VALUE"FreeBSD"gtFreeBSDltO
PTION VALUE"NetBSD"gtNetBSDltOPTION
VALUE"OpenBSD"gtOpenBSDltOPTION VALUE"AIX"gtAIXltOPT
ION VALUE"BeOS"gtBeOSltOPTION VALUE"HP-UX"gtHP-UXltO
PTION VALUE"IRIX"gtIRIXltOPTION VALUE"Neutrino"gtNe
utrinoltOPTION VALUE"OpenVMS"gtOpenVMSltOPTION
VALUE"OS/2"gtOS/2ltOPTION VALUE"OSF/1"gtOSF/1ltOPTIO
N VALUE"Solaris"gtSolarisltOPTION
VALUE"SunOS"gtSunOSltOPTION VALUE"other"gtotherlt/SE
LECTgtlt/tdgtlttd alignleft valigntopgtltSELECT
NAME"priority" MULTIPLE SIZE7gt ltOPTION
VALUE"--"gt--ltOPTION VALUE"P1"gtP1ltOPTION
VALUE"P2"gtP2ltOPTION VALUE"P3"gtP3ltOPTION
VALUE"P4"gtP4ltOPTION VALUE"P5"gtP5lt/SELECTgtlt/tdgt
lttd alignleft valigntopgtltSELECT NAME"bug
severity" MULTIPLE SIZE7gtltOPTION
VALUE"blocker"gtblockerltOPTION VALUE"critical"gtcr
iticalltOPTION VALUE"major"gtmajorltOPTION
VALUE"normal"gtnormalltOPTION VALUE"minor"gtminorltO
PTION VALUE"trivial"gttrivialltOPTION
VALUE"enhancement"gtenhancementlt/SELECTgtlt/trgtlt/t
ablegt
File
Print
Segmentation Fault
27
into this
ltSELECTgt
File
Print
Segmentation Fault
28
Simplification

• Once one has reproduced a problem, one must find
  out whats relevant
• Does the problem really depend on 10,000 lines of
  input?
• Does the failure really require this exact
  schedule?
• Do we need this sequence of calls?

29
Why Simplify?

• Ease of communication. A simplified test case is
  easier to communicate/conceptualize
• Easier debugging. Smaller test cases result in
  smaller states and shorter executions
• Identify duplicates. Simplified test cases
  subsume several duplicates
• More general

30
Motivation

• The Mozilla open-source web browser project
  receives several dozens bug reports a day.
• Each bug report has to be simplified
• Eliminate all details irrelevant to producing the
  failure
• To facilitate debugging
• To make sure it does not replicate a similar bug
  report
• In July 1999, Bugzilla listed more than 370 open
  bug reports for Mozilla.
• These were not even simplified
• Mozilla engineers were overwhelmed with work
• They created the Mozilla BugAThon a call for
  volunteers to process bug reports

31
Your Solution

• How do you solve these problems?
• Binary search
• Cut the test case in half
• Iterate
• Brilliant idea Andreas Zeller and co-workers
  Why not automate this?

32
Binary Search

• Proceed by binary search. Throw away half the
  input and see if the output is still wrong.
• If not, go back to the previous state and discard
  the other half of the input.

33
Binary Search

• Proceed by binary search. Throw away half the
  input and see if the output is still wrong.
• If not, go back to the previous state and discard
  the other half of the input.

?
34
Binary Search

• Proceed by binary search. Throw away half the
  input and see if the output is still wrong.
• If not, go back to the previous state and discard
  the other half of the input.

35
Binary Search

• Proceed by binary search. Throw away half the
  input and see if the output is still wrong.
• If not, go back to the previous state and discard
  the other half of the input.

?
36
Binary Search

• Proceed by binary search. Throw away half the
  input and see if the output is still wrong.
• If not, go back to the previous state and discard
  the other half of the input.

37
Binary Search

• Proceed by binary search. Throw away half the
  input and see if the output is still wrong.
• If not, go back to the previous state and discard
  the other half of the input.

?
38
Binary Search

• Proceed by binary search. Throw away half the
  input and see if the output is still wrong.
• If not, go back to the previous state and discard
  the other half of the input.

39
Binary Search

• Proceed by binary search. Throw away half the
  input and see if the output is still wrong.
• If not, go back to the previous state and discard
  the other half of the input.

?
40
Binary Search

• Proceed by binary search. Throw away half the
  input and see if the output is still wrong.
• If not, go back to the previous state and discard
  the other half of the input.

41
Binary Search

• Proceed by binary search. Throw away half the
  input and see if the output is still wrong.
• If not, go back to the previous state and discard
  the other half of the input.

?
42
Binary Search

• Proceed by binary search. Throw away half the
  input and see if the output is still wrong.
• If not, go back to the previous state and discard
  the other half of the input.

?
43
Binary Search

• Proceed by binary search. Throw away half the
  input and see if the output is still wrong.
• If not, go back to the previous state and discard
  the other half of the input.

?
44
Binary Search

• Proceed by binary search. Throw away half the
  input and see if the output is still wrong.
• If not, go back to the previous state and discard
  the other half of the input.

Simplified input
45
Complex Input
lttd alignleft valigntopgtltSELECT NAME"op sys"
MULTIPLE SIZE7gtltOPTION VALUE"All"gtAllltOPTION
VALUE"Windows 3.1"gtWindows 3.1ltOPTION
VALUE"Windows 95"gtWindows 95ltOPTION
VALUE"Windows 98"gtWindows 98ltOPTION
VALUE"Windows ME"gtWindows MEltOPTION
VALUE"Windows 2000"gtWindows 2000ltOPTION
VALUE"Windows NT"gtWindows NTltOPTION VALUE"Mac
System 7"gtMac System 7ltOPTION VALUE"Mac System
7.5"gtMac System 7.5ltOPTION VALUE"Mac System
7.6.1"gtMac System 7.6.1ltOPTION VALUE"Mac System
8.0"gtMac System 8.0ltOPTION VALUE"Mac System
8.5"gtMac System 8.5ltOPTION VALUE"Mac System
8.6"gtMac System 8.6ltOPTION VALUE"Mac System
9.x"gtMac System 9.xltOPTION VALUE"MacOS X"gtMacOS
XltOPTION VALUE"Linux"gtLinuxltOPTION
VALUE"BSDI"gtBSDIltOPTION VALUE"FreeBSD"gtFreeBSDltO
PTION VALUE"NetBSD"gtNetBSDltOPTION
VALUE"OpenBSD"gtOpenBSDltOPTION VALUE"AIX"gtAIXltOPT
ION VALUE"BeOS"gtBeOSltOPTION VALUE"HP-UX"gtHP-UXltO
PTION VALUE"IRIX"gtIRIXltOPTION VALUE"Neutrino"gtNe
utrinoltOPTION VALUE"OpenVMS"gtOpenVMSltOPTION
VALUE"OS/2"gtOS/2ltOPTION VALUE"OSF/1"gtOSF/1ltOPTIO
N VALUE"Solaris"gtSolarisltOPTION
VALUE"SunOS"gtSunOSltOPTION VALUE"other"gtotherlt/SE
LECTgtlt/tdgtlttd alignleft valigntopgtltSELECT
NAME"priority" MULTIPLE SIZE7gt ltOPTION
VALUE"--"gt--ltOPTION VALUE"P1"gtP1ltOPTION
VALUE"P2"gtP2ltOPTION VALUE"P3"gtP3ltOPTION
VALUE"P4"gtP4ltOPTION VALUE"P5"gtP5lt/SELECTgtlt/tdgt
lttd alignleft valigntopgtltSELECT NAME"bug
severity" MULTIPLE SIZE7gtltOPTION
VALUE"blocker"gtblockerltOPTION VALUE"critical"gtcr
iticalltOPTION VALUE"major"gtmajorltOPTION
VALUE"normal"gtnormalltOPTION VALUE"minor"gtminorltO
PTION VALUE"trivial"gttrivialltOPTION
VALUE"enhancement"gtenhancementlt/SELECTgtlt/trgtlt/t
ablegt
File
Print
Segmentation Fault
46
Simplified Input

• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• Simplified from 896 lines to one single line
• Required 12 tests only

47
Binary Search

• ltSELECT NAME"priority" MULTIPLE SIZE7gt

48
Binary Search
?

• ltSELECT NAME"priority" MULTIPLE SIZE7gt

49
Binary Search
?

• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt

?
50
Binary Search
?

• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt

?
?
51
Binary Search
?

• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• What do we do if both halves pass?

?
?
52
Change Granularity
53
General DD Algorithm

• Basic idea
• Start with few large changes first
• If all alternatives pass or are unresolved, apply
  more smaller changes.

?1
?2
?2
?1
?1
?2
?3
?4
?5
?6
?7
?8
?1
?2
?3
?4
?5
?6
?7
?8
54
Binary Search
?

• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt

?
?
55
Binary Search
?

• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt

?
?
?
56
Binary Search
?

• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt

?
?
?
?
57
Binary Search
?

• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt

?
?
?
?
?
58
Binary Search
?

• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt

?
?
?
?
?
?
59
Inputs and Failures

• Let E be the set of possible inputs
• rP ?? E corresponds to an input that passes
• rF ?? E corresponds to an input that fails

60
Binary Search
?

• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• Example of rF and rP ?

?
?
?
?
?
?
61
Binary Search
?

• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• ltSELECT NAME"priority" MULTIPLE SIZE7gt
• Example of rF and rP ?
• rF ltSELECT NALE SIZE7gt
• rP ltSELECT NAME"priori

?
?
?
?
?
?
62
Changes

• We can go from one input to another by changes
• A change ? is a mapping ? E ? E which takes one
  input and changes it to another input
• r1 ltSELECT NAty" MULTIPLE SIZE7gt
• ? insert ME"priori at appropriate position
• What is ?(r1)?

63
Changes

• We can go from one input to another by changes
• A change ? is a mapping ? E ? E which takes one
  input and changes it to another input
• r1 ltSELECT NAty" MULTIPLE SIZE7gt
• ? insert ME"priori at appropriate position
• What is ?(r1)?
• ltSELECT NAME"priority" MULTIPLE SIZE7gt

64
Decomposing Changes

• A change ? can be decomposed to a number of
  elementary changes ?1, ?2, ..., ?n where ? ?1 o
  ?2 o ... o ?n
• where (?i o ?j)(r) ?i(?j(r))
• For example, deleting a part of the input file
  can be decomposed to deleting characters one be
  one from the input file
• another way to say it by composing deleting of
  single characters we can get a change that
  deletes part of the input file
• ? insert ME"priori
• ? ?1 o ?2 o ?3 o ?4 o ?5 o ?6 o ?7 o ?8 o ?9 o
  ?10
• ?1 insert M
• ?2 insert E

65
Summary

• We have an input with failure rF
• We have an input without failure rP
• We have a set of changes cF ?1, ?2, ..., ?n
  such that
• rF (?1 o ?2 o ... o ?n )(rP) where rF is a run
  with failure
• Each subset c of cF is a test case

66
Testing Test Cases

• Given a test case c, we would like to know if the
  input generated by changing rP by the changes in
  c is an input that causes a failure
• We define a function
• test Powerset(cF) ? P, F, ?
• such that, given c?1, ?2, ..., ?m ? cF
• test(c) F iff (?1 o ?2 o ... o ?m )(rP) is a
  failing input

67
Minimizing Test Cases

• Now the question is Can we find the minimal test
  case c such that test(c) F?
• A test case c ? cF is called the global minimum
  of cF if
• for all c ? cF , c lt c ? test(c) ? F
• Global minimum is the smallest set of changes
  which will make the program fail

68
Minimizing Test Cases

• Now the question is Can we find the minimal test
  case c such that test(c) F?
• A test case c ? cF is called the global minimum
  of cF if
• for all c ? cF , c lt c ? test(c) ? F
• Global minimum is the smallest set of changes
  which will make the program fail
• Finding the global minimum may require us to
  perform exponential number of tests

69
Search for 1-minimal Input

• Different problem formulation
• Find a set of changes that cause the failure, but
  removing any change causes the failure to go away
• This is 1-minimality

70
Minimizing Test Cases

• A test case c ? cF is called a local minimum of
  cF if
• for all c ? c , test(c) ? F
• A test case c ? cF is n-minimal if
• for all c ? c , c ? c ? n ? test(c) ? F
• A test case is 1-minimal if
• for all ?i ? c , test(c ?i) ? F

71
Naïve Algorithm

• To find a 1-minimal subset of C, simply
• Remove one element ? from C
• If c ? X, recurse with smaller set
• If C ? ¹ X, C is 1-minimal

72
Analysis

• In the worst case,
• We remove one element from the set per iteration
• After trying every other element
• Work is potentially
• N (N-1) (N-2)
• This is O(N2)

73
Work Smarter, Not Harder

• We can often do better
• Silly to start out removing 1 element at a time
• Try dividing change set in 2 initially
• Increase of subsets if we cant make progress
• If we get lucky, search will converge quickly

74
Minimization Algorithm

• The delta debugging algorithm finds a 1-minimal
  test case
• It partitions the set cF to ?1, ?2, ... ?n
• ?1, ?2, ... ?n are pairwise disjoint
• cF ?1 ? ?2 ? ... ? ?n
• Define the complement of ?i as ?i cF ? ?i
• Start with n 2
• Tests each test case defined by the partition and
  their complements
• Reduce the test case if a smaller failure
  inducing set is found
• otherwise refine the partition, i.e. n n2

75
Each Step of the Minimization Algorithm

• Let n 2
• (A) Start with ? as test set
• (B) Test each ?1, ?2, ... ?n and each ?1, ?2,
  ..., ?n
• (C) There are four possible outcomes
• Some ?i causes failure
• Go to step (A) with ? ?i and n 2
• Some ?i causes failure
• Go to step (A) with ? ?i and n n ? 1
• No test causes failure
• Increase granularity Go to (A) with ? ? and n
  2n
• The granularity can no longer be increased
• Done, found the 1-minimal subset

76
Delta Debugging
77
Analysis

• Worst case is still quadratic
• Subdivide until each set is of size 1
• Reduced to the naïve algorithm
• Good news
• For single failure, converges in log N
• Binary search again

78
The GNU C Compiler
define SIZE 20 double mult(double z, int n)
int i, j i 0 for (j 0 j lt n j)
i i j 1 zi zi (z0
1.0) return zn void copy(double to,
double from, int count) int n (count 7)
/ 8 switch (count 8) do case 0 to
from case 7 to from case
6 to from case 5 to from
case 4 to from case 3 to
from case 2 to from case 1
to from while (--n gt 0) return
mult(to, 2) int main(int argc, char
argv) double xSIZE, ySIZE double
px x while (px lt x SIZE) px (px
x) (SIZE 1.0) return copy(y, x, SIZE)

• This program (bug.c) causes GCC 2.95.2 to crash
  when optimization is enabled
• We would like to minimize this program in order
  to file a bug report
• In the case of GCC, a passing program run is the
  empty input
• For the sake of simplicity, we model change as
  the insertion of a single character
• r? is running GCC with an empty input
• r? means running GCC with bug.c
• each change di inserts the ith character of bug.c

79
The GNU C Compiler

• The test procedure would
• create the appropriate subset of bug.c
• feed it to GCC
• return ? iff GCC had crashed, and ? otherwise

77
755
377
188
80
The GNU C Compiler

• The minimized code is
• The test case is 1-minimal
• No single character can be removed without
  removing the failure
• Even every superfluous whitespace has been
  removed
• The function name has shrunk from mult to a
  single t
• This program actually has a semantic error
  (infinite loop), but GCC still isn't supposed to
  crash
• So where could the bug be?
• We already know it is related to optimization
• If we remove the O option to turn off
  optimization, the failure disappears

t(double z,int n)int i,jfor()iij1ziz
i(z00)return zn
81
The GNU C Compiler

• The GCC documentation lists 31 options to control
  optimization on Linux
• It turns out that applying all of these options
  causes the failure to disappear
• Some option(s) prevent the failure

ffloat-store fno-default-inline fno-defer-pop
fforce-mem fforce-addr fomit-frame-pointer fno-
inline finline-functions fkeep-inline-functions
fkeep-static-consts fno-function-cse ffast-math
fstrength-reduce fthread-jumps fcse-follow-jum
ps fcse-skip-blocks frerun-cse-after-loop freru
n-loop-opt fgcse fexpensive-optimizations fsche
dule-insns fschedule-insns2 ffunction-sections
fdata-sections fcaller-saves funroll-loops funr
oll-all-loops fmove-all-movables freduce-all-giv
s fno-peephole fstrict-aliasing
82
The GNU C Compiler

• We can use test case minimization in order to
  find the preventing option(s)
• Each di stands for removing a GCC option
• Having all di applied means to run GCC with no
  option (failing)
• Having no di applied means to run GCC with all
  options (passing)
• After seven tests, the single option -ffast-math
  is found which prevents the failure
• Unfortunately, it is a bad candidate for a
  workaround because it may alter the semantics of
  the program
• Thus, we remove -ffast-math from the list of
  options and make another run
• Again after seven tests, it turn out that
  -fforce-addr also prevents the failure
• Further examination shows that no other option
  prevents the failure

83
The GNU C Compiler

• So, this is what we can send to the GCC
  maintainers
• The minimal test case
• The failure only occurs with optimization
• -ffast-math and -fforce-addr prevent the failure

84
Case Studies

• Reducing a Mozilla bug to
• Print the html file containing ltSELECTgt
• Minimizing fuzz input
• Fuzzing Take a program, send it randomly
  generated input and see it crashes
• Used delta debugging to minimize fuzz input

85
Summary

• Delta Debugging is a technique, not a tool
• Bad News
• Probably must be reimplemented for each
  significant system
• To exploit knowledge of changes
• Good News
• Relatively simple algorithm, significant payoff
• Its worth reimplementing

86
Co-operative Bug Isolation

• Problem No matter how we try, deployed code will
  probably have bugs
• Insight User community for software is much much
  larger than test team
• Idea Ben Liblit and others Use information
  from on-the-field bugs to debug software
• Collect feedback from post-deployment crashes and
  mine for bug causes

87
Example?
88
Bug Isolation Architecture
Predicates
ShippingApplication
ProgramSource
Sampler
Compiler
StatisticalDebugging
Counts J/L
Top bugs withlikely causes
89
Model of Behavior

• Each behavior is expressed asa predicate P on
  program stateat a particular program point.
• Count how often P observed true andP
  observed using sparse but fair random samples of
  complete behavior.

90
Returned Values Are Interesting

• n fprintf()

• Did you know that fprintf() returns a value?
• Do you know what the return value means?
• Do you remember to check it?

91
Returned Value Predicate Counts

• n fprintf()
• // return value lt 0 ?// return value 0 ?//
  return value gt 0 ?

• Syntax yields instrumentation site
• Site yields predicates on program behavior
• Exactly one predicate true per visit to site

92
Many Other Behaviors of Interest

• Assert statements
• Perhaps automatically introduced
• Unusual floating point values
• Did you know there are nine kinds?
• Coverage of modules, functions, basic blocks,
• Reference counts negative, zero, positive,
  invalid
• Kinds of pointer stack, heap, null,
• Temporal relationships x before/after y
• More ideas? Toss them all into the mix!

93
Summarization and Reporting

• Observation stream ? observation count
• How often is each predicate observed true?
• Removes time dimension, for good or ill
• Bump exactly one counter per observation
• Feedback report is
• Vector of predicate counters
• Success/failure outcome label
• Still quite a lot to measure
• What about performance?

94
Fair Sampling Transformation
Predicates
ShippingApplication
ProgramSource
Sampler
Compiler
StatisticalDebugging
Counts J/L
Top bugs withlikely causes
95
Sampling the Bernoulli Way

• Decide to examine or ignore each site
• Randomly
• Independently
• Dynamically
• Cannot toss coin at each site too slow
• Cannot toss periodically based on a counter

96
Amortized Coin Tossing

• Randomized global countdown
• Small countdown ? upcoming sample
• Selected from geometric distribution
• Inter-arrival time for biased coin toss
• How many tails before next head?
• Mean sampling rate is tunable parameter

97
Weighing Acyclic Regions

• Break CFG into acyclic regions
• Each region has
• Finite number of paths
• Finite max number of instrumentation sites
• Compute max weight in bottom-up pass

4
1
3
2
1
1
2
1
98
Weighing Acyclic Regions

• Clone acyclic regions
• Fast variant
• Slow variant
• Choose at run time
• Retain only decrements on fast path, and even
  these can be optimized

99
What Does This Give Us?

• Absolutely certain of what we do see
• Subset of dynamic behavior
• Success/failure label for entire run
• Uncertain of what we dont see
• Given enough runs, samples reality
• Common events seen most often
• Rare events seen at proportionate rate

100
Playing the Numbers Game
Predicates
ShippingApplication
ProgramSource
Sampler
Compiler
StatisticalDebugging
Counts J/L
Top bugs withlikely causes
101
Find Causes of Bugs

• Gather information about many predicates
• 298,482 predicates in bc
• 857,384 predicates in Rhythmbox
• Vast majority not related to any particular bug ?
• How do we find the useful bug predictors?
• Data is incomplete, noisy, irreproducible,

102
Some Definitions
103
Fun With Multi-Valued Logic

• Identify unlucky sites on the doomed path
• Background risk of failure for reaching this
  site, regardless of predicate truth/falsehood

104
Isolate the Predictive Value of P

• Does P being true increase the chance of failure
  over the background rate?
• Formal correspondence to likelihood ratio testing

105
Simulated Iterative Bug Fixing

• Rank all predicates under consideration
• Select the top-ranked predicate P
• Add P to bug predictor list
• Discard P and all runs where P was true
• Simulates fixing the bug predicted by P
• Reduces rank of similar predicates
• Repeat until out of failures or predicates

106
Lessons Learned

• Can learn a lot from actual executions
• Users are running buggy code anyway
• We should capture some of that information
• Great potential in hybrid approaches
• Dynamic reality-driven debugging
• Statistical best-effort with uncertainty
• Static use program structure to fill in the gaps

107
Vision for Statistical Debugging

• Bug triage that directly reflects reality
• Learn the most, most quickly, about the bugs that
  happen most often
• Variability is a benefit rather than a problem
• Results grow stronger over time
• Find bugs while you sleep!

108
Acknowledgments

• Alex Aiken, Ben Liblit