Extended Static Checking for Java

1
Extended Static Checkingfor Java

• Cormac FlanaganK. Rustan M. LeinoMark
  LillibridgeGreg NelsonJames B. SaxeRaymie
  Stata
• Compaq SRC

18 June 2002PLDI02, Berlin, Germany
2
Vision

• Increased programmer productivity and program
  reliability through increased rigor

Record design decisions Utilize automatic
checking Detect errors and improve
maintainability
3
Users view
ESC/Java
Warning messages
Annotated Java program
public class Bag private /_at_non_null/ int
a private int n //_at_ invariant 0 lt n n lt
a.length public Bag(/_at_non_null/ int
initialElements) n initialElements.length
a new intn System.arraycopy(initialElement
s, 0, a, 0, n) public void add(int x)
if (n a.length) int b new
int2(a.length1) System.arraycopy(a, 0, b,
0, n) a b an x n public
int extractMin() int m Integer.MAX_VALUE
int mindex 0 for (int i 0 i lt n i)
if (ai lt m) mindex i m
ai if (0 lt n) n-- amindex
an return m // The program text
continues down here, but if youre // reading
this, you probably arent paying attention to //
the talk.
Bag.java18 Array index possibly too large
4
ESC/Java distinguishing features

• Modular checking
• Annotation language captures design decisions
• Powered by automatic theorem prover
• Not decidable

5
Demo
6
Design tradeoffs

• Missed errors
• Spurious warnings
• Annotation overhead
• Performance

7
Tool architecture
Annotated Java program
Translator
Verification condition
Valid
Automatic theorem prover
Resource exhausted
Counterexample context
Post processor
Warning messages
8
Tool architecture, detail
Annotated Java program
Sugared command
Translator
Primitive command
Passive command
Verification condition
Automatic theorem prover
Counterexample context
Post processor
Warning messages
9
Tool architecture, detail
10
Annotation language
Annotated Java program

• Simple
• non_null
• Method annotations
• requires E
• modifies w
• ensures P
• exsures (T x) Q
• Object invariants
• invariant E

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
11
Annotation language
Annotated Java program

• Simple
• non_null
• Method annotations
• requires E
• modifies w
• ensures P
• exsures (T x) Q
• Object invariants
• invariant E

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
12
Annotation language
Annotated Java program

• Specification expressions
• side-effect free Java expressions
• no , no method calls
• \result, \old(E)
• ensures \result \old(x)
• gt
• (\forall T x P), (\exists T x P)
• (\forall int j 0 lt j j lt n gt aj gt 0)
• \typeof(E), \type(T), lt
• requires \typeof(x) \typeof(this)

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
13
Annotation language
Annotated Java program

• Specification expressions
• side-effect free Java expressions
• no , no method calls
• \result, \old(E)
• ensures \result \old(x)
• gt
• (\forall T x P), (\exists T x P)
• (\forall int j 0 lt j j lt n gt aj gt 0)
• \typeof(E), \type(T), lt
• requires \typeof(x) \typeof(this)

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
14
Annotation language
Annotated Java program

• Concurrency
• monitored_by lock
• /_at_ monitored_by this / long x
• \locksetlock
• requires \locksetthis
• lock0 lt lock1
• \max(\lockset)
• requires \max(\lockset) lt this

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
15
Annotation language
Annotated Java program

• Concurrency
• monitored_by E
• /_at_ monitored_by this / long x
• \locksetlock
• requires \locksetthis
• lock0 lt lock1
• \max(\lockset)
• requires \max(\lockset) lt this

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
16
Annotation language
Annotated Java program

• Ghost variables
• ghost public T x
• ghost public int objectState
• ghost public \TYPE elementType
• set x E
• set objectState Open
• set elementType \type(T)

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
17
Annotation language
Annotated Java program

• Ghost variables
• ghost public T x
• ghost public int objectState
• ghost public \TYPE elementType
• set x E
• set objectState Open
• set elementType \type(T)

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
18
Annotation language
Annotated Java program

• Miscellaneous
• assert E
• assume E
• assume x gt 0 // because x yy
• nowarn
• x aj //_at_ nowarn
• axiom E
• axiom (\forall int x x gtgt 2 gt 0)

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
19
Sugared commands
Annotated Java program

• S,T assert E
• assume E
• x E
• raise
• S T
• S ! T
• S T
• loop inv E S ? T end
• call x t.m(E)

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
20
Sugared commands
Annotated Java program

• x t.f.g
• assert t ! null
• tmp select(f, t)
• assert tmp ! null
• x select(g, tmp)
• if (x lt 0) x -x /_at_ assert x gt 0 /
• ( assume x lt 0 x -x
• assume !(x lt 0)
• )
• assert x gt 0

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
21
Sugared commands
Annotated Java program

• x t.f.g
• assert lblneg(Null_at_58.9, t ! null)
• tmp select(f, t))
• assert lblneg(Null_at_58.11, tmp ! null)
• x select(g, tmp)
• if (x lt 0) x -x /_at_ assert x gt 0 /
• ( assume x lt 0 assume lblpos(Then2807,
  true) x -x
• assume !(x lt 0) assume lblpos(Else2807,
  true)
• )
• assert x gt 0

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
22
Primitive commands
Annotated Java program

• S,T assert E
• assume E
• x E
• raise
• S T
• S ! T
• S T
• loop inv E S ? T end
• call x t.m(E)

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
23
Primitive commands
Annotated Java program

• //_at_ requires Pre modifies w ensures
  Postvoid m(U u)
• call x t.m(E)
• var u in
• u E
• assert Pre
• var w0 in
• w0 w
• havoc w
• assume Post
• end
• end

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
24
Passive commands
Annotated Java program

• S,T assert E
• assume E
• x E

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
25
Passive commands
Annotated Java program

• if (x lt 0) x -x /_at_ assert x gt 0 /
• ( assume x0 lt 0 assume x1 -x0 assume x2
  x1
• assume !(x0 lt 0) assume x2 x0
• )
• assert x2 gt 0

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
26
Weakest preconditions
Annotated Java program

• wp(assert E, Q) E Q
• wp(assume E, Q) E gt Q
• wp(ST, Q) wp(S, wp(T,Q))
• wp(S T, Q) wp(S, Q) wp(T, Q)
• wp(S, Q) wp(S, true) wlp(S, Q)
• wlp(S, Q) wlp(S, false) Q

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
27
Verification condition
Annotated Java program

• Universal background predicate
• (FORALL (t) (lt t t))
• Type-specific background predicate
• (lt T_T T_java.lang.Object)
• Verification condition BPUniv BPT gt
  VCmethod

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
28
Verification condition
(BG_PUSH (AND (lt T_T T_java.lang.Object)
(EQ T_T (asChild T_T T_java.lang.Object))
(DISTINCT arrayType T_boolean T_char T_byte
T_short T_int T_long
T_float T_double T_.TYPE
T_T T_java.lang.Object))) (EXPLIES (LBLNEG
vc.T.abs.2.2 (IMPLIES (AND (EQ
elems_at_pre elems) (EQ elems (asElems
elems)) (lt (eClosedTime elems) alloc) (EQ
LS (asLockSet LS)) (EQ alloc_at_pre alloc))
(NOT (AND (EQ _at_true (is x2.21
T_int)) (OR (AND (OR
(AND (lt x2.21 0) (LBLPOS
trace.Then0,3.15 (EQ _at_true _at_true))
(EQ x3.17 (- 0 x2.21)) (EQ
x2.21lt1gt x3.17)) (AND (NOT
(lt x2.21 0)) (LBLPOS
trace.Else1,3.4 (EQ _at_true _at_true))
(EQ x2.21lt1gt x2.21))) (NOT (LBLNEG
Assert_at_4.8 (gt x2.21lt1gt 0)))) (AND
(OR (AND (lt x2.21 0)
(LBLPOS trace.Then0,3.15 (EQ _at_true
_at_true)) (EQ x3.17 (- 0 x2.21))
(EQ x2.21lt1gt x3.17)) (AND
(NOT (lt x2.21 0)) (LBLPOS
trace.Else1,3.4 (EQ _at_true _at_true))
(EQ x2.21lt1gt x2.21))) (LBLNEG
Assert_at_4.8 (gt x2.21lt1gt 0)) (NOT
(LBLNEG Exception_at_5.2 (EQ ecReturn
ecReturn))))))))) (AND (DISTINCT
ecReturn)))
Annotated Java program

• class T
• static int abs(int x)
• if (x lt 0) x -x
• //_at_ assert x gt 0

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
29
Theorem prover Simplify
Annotated Java program

• Nelson-Oppen cooperating decision procedures
• conguence closure
• linear arithmetic
• partial orders
• quantifiers
• Key features
• automatic no user interaction
• refutation based searches for counterexamples
• heuristics tuned for program checking
• labels
• time limit

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
30
Counterexamples and warnings
Annotated Java program

• Counterexample labels (IndexTooBig_at_26.5
  vc.Bag.add.20.2 trace.Then0,21.23)
  context (AND (NEQ tmp1!a23.23
  null) (NEQ this null) (EQ alloc_at_pre
  alloc) (EQ tmp4!n26.6 0)
  (lt alloc (vAllocTime tmp3!a26.4)) )
• Bag add(int) ...--------------------------------
  ----------------------------------------Bag.java
  26 Warning Array index possibly too large
  (IndexTooBig) an x Execution
  trace information Executed then branch in
  "Bag.java", line 21, col 23.---------------------
  --------------------------------------------------
  -

Sugared command
Primitive command
Translator
Passive command
Verification condition
Automatictheorem prover
Counterexample context
Post processor
Warning messages
31
Experience
Errors found

• Javafe (ESC/Java front end, 40kloc) 12
• Mercator (web crawler) 2
• Sparse matrix library 2
• Annotation wizard 2
• Performance plotting program 1
• External user experience
• Experience with automatically inferred annotations

32
Performance

• 50 of all methods lt 0.5 s
• 80 of all methods lt 1 s
• time limit 300 s
• total time for Javafe (40kloc) 65 min.

33
Experience annotations

• Capture common design decisions
• Suggested immediately by warnings
• Overhead 4-10 of source code
• 1 annotation per field or parameter
• Most common annotations
• non_null
• container element types

34
Related work

• ESC/Modula-3
• Detlefs, Leino, Nelson, Saxe 1992-1996
• Full functional Java specification and
  verification
• JML, LOOP
• Languages and language features
• Euclid, Eiffel, B, Escher, Guava, Vault, Cqual,
• LCLint, refinement types, Types against races
• Other checking techniques
• Abstract interpretation, PREfix, SLAM, Bandera,
  Java PathFinder 2, Canvas, ESP, AST Toolkit,
  Metal,
• Tools built on ESC/Java
• Daikon, Houdini

35
Conclusions

• Theorem proving for program analysis works!
• Annotations express design decisions
• Cost effective?
• Ready for use by disciplined programming teams
  and in teaching
• Future challenges
• beyond method-modular checking
• more specification methodologies
• Download ESC/Java research.compaq.com/SRC/escSo
  urces now available, too!