Debugging Temporal Specifications with Concept Analysis

1
Debugging Temporal Specifications with Concept
Analysis

• Ras Bodik UC BerkeleyGlenn Ammons IBM
  ResearchDavid Mandelin UC BerkeleyJim
  Larus Microsoft Research

2
Whats a temporal specification?

• compare with testing

verification
program
program
modelchecker
does the property hold?
tester
is the outputcorrect?
property
test case

• sample temporal properties
• allocated memory is freed Xmalloc(_) free(X)
• locks are released lock(X) unlock(X)

3
Coverage of testing, model checking
4
So where do specs come from?
program
? ? ? ? ? ?
modelchecker
spec
unlock(X)
5
Specification mining

• Key idea
• mine specs by observing how programs use the API
• Two problems
• 1. What constraints to include in the spec?
• Ffopen() read(F,_) fclose(F) or?
  Ffopen() fclose(F)
• 2. How to remove buggy behavior?
• in practice, programs are full of latent bugs

6
The specification mining process

• lock(X).
• lock(X) lock(X) unlock(X).
• lock(X) unlock(X) unlock(X).
• lock(X) lock(X).
• lock(X) unlock(X).

7
Example
trace
two scenarios
specification
h(3, 5) c(10) a(4, 5) d(4, 7) b(0, 5) f(10) h(8,
11) e(7) f(50) d(15, 1) c(7) a(9, 11) b(6,
7) d(9, 14) f(20) e(7)
?
?
8
Scenario extraction

• h(3, 5)
• c(10)
• a(4, 5)
• d(4, 7)
• b(0, 5)
• f(10)
• h(8, 11)
• e(7)
• f(50)
• d(15, 1)
• c(7)
• a(9, 11)
• b(6, 7)
• d(9, 14)
• f(20)
• e(7)

h(_, X) a(Y, X) b(_, X) d(Y, Z) e(Z)
9
Classifying scenarios
?
?
?
10
Concept analysis mammals
11
Concept analysis mammals
cats gibbons dogs dolphins humans whales

12
Concept analysis mammals
hairy
cats gibbons dogs
13
Concept analysis mammals
4-legged hairy
cats dogs
14
Concept analysis scenarios
15
Concept analysis scenarios
scen. 0 scen. 1 scen. 2 scen. 3 scen. 4 scen. 5

16
Concept analysis scenarios
scen. 0 scen. 1 scen. 2
Takes transition 1
17
Concept analysis scenarios
scen. 0 scen. 2
Takes transition 0 Takes transition 1
18
Experimental results summary

• Case study
• X11 API (contains hundreds of procedures)
• Training set
• 90 traces from 72 programs using X11
• Specifications mined
• 17 (both new and known informally)
• Bugs found with these specs
• 199 (ranging from races to performance bugs)

19
Example specification
English rule Pass XSetSelectionOwner() the
timestamp from the last event. Mined spec
(event.time X) XNextEvent() XFilterEvent(even
t.time X) XtDispatchEvent(event.time
X) (event.time X) XCheckWindowEvent() cb_XtAc
tionProc(event.time X)
XSetSelectionOwner(time X)
20
Debugging with concept analysis

• In terms of classification steps
• Classifying with the concept lattice usually was
• much faster than classifying scenarios
  individually
• within a factor of two from optimal
• In terms of wall-clock time
• Classifying with the concept lattice usually was
• about the same or slightly slower
• but achieves higher confidence
• better UI needed