Title: Debugging Temporal Specifications with Concept Analysis
1Debugging Temporal Specifications with Concept
Analysis
- Ras Bodik UC BerkeleyGlenn Ammons IBM
ResearchDavid Mandelin UC BerkeleyJim
Larus Microsoft Research
2Whats a temporal specification?
verification
program
program
modelchecker
does the property hold?
tester
is the outputcorrect?
property
test case
- sample temporal properties
- allocated memory is freed Xmalloc(_) free(X)
- locks are released lock(X) unlock(X)
3Coverage of testing, model checking
4So where do specs come from?
program
? ? ? ? ? ?
modelchecker
spec
unlock(X)
5Specification mining
- Key idea
- mine specs by observing how programs use the API
- Two problems
- 1. What constraints to include in the spec?
- Ffopen() read(F,_) fclose(F) or?
Ffopen() fclose(F) - 2. How to remove buggy behavior?
- in practice, programs are full of latent bugs
6The specification mining process
- lock(X).
- lock(X) lock(X) unlock(X).
- lock(X) unlock(X) unlock(X).
- lock(X) lock(X).
-
- lock(X) unlock(X).
-
7Example
trace
two scenarios
specification
h(3, 5) c(10) a(4, 5) d(4, 7) b(0, 5) f(10) h(8,
11) e(7) f(50) d(15, 1) c(7) a(9, 11) b(6,
7) d(9, 14) f(20) e(7)
?
?
8Scenario extraction
- h(3, 5)
- c(10)
- a(4, 5)
- d(4, 7)
- b(0, 5)
- f(10)
- h(8, 11)
- e(7)
- f(50)
- d(15, 1)
- c(7)
- a(9, 11)
- b(6, 7)
- d(9, 14)
- f(20)
- e(7)
h(_, X) a(Y, X) b(_, X) d(Y, Z) e(Z)
9Classifying scenarios
?
?
?
10Concept analysis mammals
11Concept analysis mammals
cats gibbons dogs dolphins humans whales
12Concept analysis mammals
hairy
cats gibbons dogs
13Concept analysis mammals
4-legged hairy
cats dogs
14Concept analysis scenarios
15Concept analysis scenarios
scen. 0 scen. 1 scen. 2 scen. 3 scen. 4 scen. 5
16Concept analysis scenarios
scen. 0 scen. 1 scen. 2
Takes transition 1
17Concept analysis scenarios
scen. 0 scen. 2
Takes transition 0 Takes transition 1
18Experimental results summary
- Case study
- X11 API (contains hundreds of procedures)
- Training set
- 90 traces from 72 programs using X11
- Specifications mined
- 17 (both new and known informally)
- Bugs found with these specs
- 199 (ranging from races to performance bugs)
19Example specification
English rule Pass XSetSelectionOwner() the
timestamp from the last event. Mined spec
(event.time X) XNextEvent() XFilterEvent(even
t.time X) XtDispatchEvent(event.time
X) (event.time X) XCheckWindowEvent() cb_XtAc
tionProc(event.time X)
XSetSelectionOwner(time X)
20Debugging with concept analysis
- In terms of classification steps
- Classifying with the concept lattice usually was
- much faster than classifying scenarios
individually - within a factor of two from optimal
- In terms of wall-clock time
- Classifying with the concept lattice usually was
- about the same or slightly slower
- but achieves higher confidence
- better UI needed