Materialization in Shape Analysis with Structural Invariant Checkers

1
Materialization in Shape Analysis with Structural
Invariant Checkers

• Bor-Yuh Evan Chang
• Xavier Rival
• George C. Necula
• University of California, Berkeley
• August 27, 2007
• ITU Copenhagen

2
Whats shape analysis? Whats special?
Shape analysis tracks memory manipulation in a
flow-sensitive manner.

• Memory manipulation
• Particularly important in systems code (in C)
• Flow-sensitive
• Many important properties
• E.g., Is an object freed? Is a file open?
• Heap abstracted differently at different points
• E.g., Not based on allocation site

3
Example Typestate with shape analysis
Concrete Example
Abstraction

• cur l
• while (cur ! null)
• assert(cur is red)
• make_purple(cur)
• cur cur!next

program-specific predicate
flow-sensitive heap abstraction

• make_purple() could be
• lock()
• free()
• open()

4
Shape analysis is not yet practical
Usability Choosing the heap abstraction difficult

• Built-in high-level predicates
• - Hard to extend
• No additional user effort

Parametric in low-level, analyzer-oriented
predicates Very general and expressive - Hard
for non-expert
Parametric in high-level, developer-oriented
predicates Extensible Easier for developers
5
Shape analysis is not yet practical
Scalability Finding right level of abstraction
difficult
Ç
Ç
Ç
Ç
Ç
emp
Ç
Ç
Ç
6
Hypothesis
The developer can describe the memory in a
compact manner at an abstraction level sufficient
for the properties of interest (at least
informally).

• Good abstraction is program-specific

7
Observation
Checking code expresses a shape invariant and an
intended usage pattern.

• bool redlist(List l)
• if (l null)
• return true
• else
• return
• l!color red
• redlist(l!next)

8
Proposal
An automated shape analysis with a memory
abstraction parameterized by invariant checkers.

• Extensible
• Abstraction based on the developer-supplied
  checkers
• Targeted for Usability
• Global data structure specification, local
  invariant inference
• Targeted for Scalability
• Based on the hypothesis

9
Shape analysis is an abstract interpretation on
memory states with

• Materialization (partial concretization)
• To perform strong updates
• And widening for termination

10
Outline

• Memory abstraction
• Restrictions on checkers
• Challenge Intermediate invariants
• Materialization by forward unfolding
• Where and how
• Challenge Unfolding segments
• Materialization by backward unfolding
• Challenge Back pointers
• Deciding where to unfold generically

11
Abstract memory using checkers
Some number of points-to edges that satisfies
checker c
Graphs
values (address or null)
checker run c()

points-to relation _at_f ?
partial run ?
Example
Disjointly, !next , !next , and is a
list.
next

list
next
12
Checkers as inductive definitions
bool list(List l) if (l null) return
true else return list(l!next)
Disjointness Checker run can dereference any
object field only once
13
What can a checker do?

• In this talk, a checker
• is a pure, recursive function
• dereferences any object field only once during a
  run
• only one argument can be dereferenced (traversal
  arg)
• has only additional pointer parameters

Traversal argument
bool dll(Dll l, Dll prev) if (l null)
return true else return l!prev
prev dll(l!next)
9.
Only fields from traversal argument
Ç
? null
14
Example checker Two-level skip list
9,.
9.
Ç
Ç
?
null
skip


next
skip0(g)
15
back to the abstract domain
16
Challenge Intermediate invariants
assert(redlist(l)) cur l while (cur ! null)
make_purple(cur) cur cur!next assert(p
urplelist(l))
Prefix Segment Described by ?
Suffix Described by checkers
17
Prefix segments as partial checker runs
Abstraction
Checker Run
Doesnt quite work because we need materialization
Formula
?
c(?) c(?)
18
Outline

• Memory abstraction
• Restrictions on checkers
• Challenge Intermediate invariants
• Materialization by forward unfolding
• Where and how
• Challenge Unfolding segments
• Materialization by backward unfolding
• Challenge Back pointers
• Deciding where to unfold generically

19
Flow function Unfold and update edges
x!next x!next!next
Unfold inductive definition
Strong updates using disjointness of regions
20
Unfolding where, how, and why ok
x!next x!next!next

• Where
• Reach a traversal argument with x!next
• How and Why Ok (concretizations same)
• By definition

21
What about unfolding segments?


list
list
x
y
materialize x!next

Ç




list
list
list
next
x, y
x
y
22
Segment connector (for unfolding)
unfolded points-to
folded recursive calls
pure formula
Concrete store ¾ Val ! Val valuation º SymVal
! Val
Inductive Definitions c() Ç (Mu Mf Æ F)
Ç
c() c0(0)
¾, º ²
iff there exists an i such that c() i c0(0)
23
Basic properties of segments

• If ¾, º ² c() c0(0), then ¾, º ² c()
  c0(0)
• If ¾, º ² (c() c0(0)) c0(0), then ¾, º ²
  c() (elimination)
• , º ² c() c() (reflexivity)
• If ¾, º ² (c() c0(0)) (c0(0) c
  00(00)), then ¾, º ² c() c
  00(00) (transitivity)

24
Outline

• Memory abstraction
• Restrictions on checkers
• Challenge Intermediate invariants
• Materialization by forward unfolding
• Where and how
• Challenge Unfolding segments
• Materialization by backward unfolding
• Challenge Back pointers
• Deciding where to unfold generically

25
Challenge Back pointers
Example Removal in doubly-linked lists

• Traversal on next field to find element to
  remove
• Materialize cur!prev and remove cur

9.
Ç
? null
Need to unfold backward
26
Backwards unfolding by forwards unfolding
i
27
Outline

• Memory abstraction
• Restrictions on checkers
• Challenge Intermediate invariants
• Materialization by forward unfolding
• Where and how
• Challenge Unfolding segments
• Materialization by backward unfolding
• Challenge Back pointers
• Deciding where to unfold generically

28
Deciding where to unfold
A pointer that may materialize these fields
Where in the traversal it may be materialized

• Observations Can indicate (with types) what
  fields are materialized for a checker parameter

types f1hl1i, , fnhlni levels l
n unk
hl1i
hlni

• Levels

Level 0 Materialized in this call.
Level -1 Materialized just before this call
29
Example Doubly-linked lists
nexth0i, prevh0i, ½ nexth-1i, prevh-1i
9( nexth1i, prevh1i).
Before Traversal argument had level 0 fields
(implicitly)
Backward unfolding parameter ½ has level -1
Ç
? null
30
Example Alternative doubly-linked list
nexth0i, prevh-1i
9( nexth2i, prevh1i).
nexth1i, prevh0i, ½ nexth-1i, prevh-2i
9( nexth1i, prevh1i).
Ç
Ç
? null
? null
31
Types can be inferred automatically

• Checking

fh0i lt typeof()
typeof() 1 lt declared_typeof(¼) (where
c(¼) )
Inference using a fixed-point computation with
types initialized to
32
SummaryEnabling materialization anywhere

• Defined segments as partial checker runs directly
  (inductively)
• For forward unfolding
• Backward unfolding derived from forward unfolding
• Checker parameter types with levels
• For deciding where to unfold
• Inferable and does not affect soundness

33
SummaryGiven checkers, everything is automatic
34
Conclusion

• Invariant checkers can form the basis of a memory
  abstraction that
• Is easily extensible on a per-program basis
• Expresses developer intent
• Critical for usability
• Prerequisite for scalability
• Enabling materialization anywhere
• Inductive segments
• Pre-analysis on checkers to decide where to
  unfold robustly

35
(No Transcript)
36
Challenge Termination and precision
last l cur l!next while (cur ! null) //
cur, last if () last cur cur cur!
next
Observation Previous iterates are less unfolded
Fold into checker edges But where and how much?
37
History-guided folding
last l cur l!next while (cur ! null) if
() last cur cur cur! next

• Match edges to identify where to fold
• Apply local folding rules

l, last
cur
l,
r
last
cur
l
v
?
l
last
?
Yes
l
last
cur
38
SummaryEnabling checker-based shape analysis

• Built-in disjointness of memory regions
• As in separation logic
• Checkers read any object field only once in a run
• Generalized segment abstraction
• Based on partial checker runs
• Generalized folding into inductive predicates
• Based on iteration history (i.e., a widening
  operator)

39
Experimental results

• Verified structural invariants as given by
  checkers are preserved across data structure
  manipulation
• Limitations (in scull driver)
• Arrays not handled (rewrote as linked list), char
  arrays ignored
• Promising as far as number of disjuncts