ZING Systematic State Space Exploration of Concurrent Software

1
ZING Systematic State Space Exploration of
Concurrent Software

• Jakob Rehof
• Microsoft Research
• http//research.microsoft.com/rehof
• http//research.microsoft.com/zing
• Joint work with
• Tony Andrews (MS)
• Shaz Qadeer (MSR)
• Sriram K. Rajamani (MSR)

2
Lecture II Outline

• ZING language
• Demos
• LTM analysis
• X86 -gt ZING
• Procedure Summaries
• Conformance Theory (I)

3
Zing Language

• Zing C - some types inheritance
  concurrency modeling features
• Concurrency shared memory message-passing
• Modeling features nondeterminism sets
  symbolic execution

4
Concurrency
static activate void foo()
async-call-statement

async invocation-expression

5
Modeling
choose-expression choose( type )
choose( primary-expression ) event-statement
event(integer-expression,
integer-expression,
boolean-expression) set-declaration
set identifier type

6
Channels
channel-declaration chan identifier type

send-statement send(expression,
expression)
7
Synchronization
select-statement select select-qualifiers
join-statements select-qualifier end
first visible join-statement join-list -gt
embedded-statement timeout -gt
embedded-statement
8
Synchronization
join-list join-pattern join-list
join-pattern join-pattern
wait(boolean-expression) receive(expression,
expression) event(integer-expression,
integer-expression,
boolean-expression)
9
Lecture II Outline

• ZING language
• Demos
• LTM analysis
• X86 -gt ZING
• Procedure Summaries
• Conformance Theory (I)

10
Indigo Transaction Manager
Abstracting the LTM
LTM
DTM (20K LOC)
Volatile RM interface
Prepare Rollback Commit
CreateTx CloneTx DurableEnlist VolatileEnlist Comm
it Abort
Prepared ForceRollback EnlistmentDone
LTM (10K LOC)
Prepare Rollback Commit
LTM interface to RMs
LTM interface to Client
Durable RM interface
11
Overview of the bug

• Transactions tx1 and tx2 must be inserted in the
  same bucket of the hash table.
• Transaction tx2 is a bystander that ensures that
  tx1.next is non-null (a necessary precondition).
• Transaction tx1 is committed. The commit thread
  and the timer thread interleave (4 context
  switches at specific locations) such that
  tx1.next is set to null by the timer thread and
  subsequently dereferenced by the commit thread.

tx1
tx2
12
Lecture II Outline

• ZING language
• Demos
• LTM analysis
• X86 -gt ZING
• Procedure Summaries
• Conformance Theory (I)

13
Lecture II Outline

• ZING language
• Demos
• LTM analysis
• X86 -gt ZING
• Procedure Summaries
• Conformance Theory (I)

14
Procedure Summaries for Concurrent Programs

• Generalized CFL-Reachability Algorithm
• Qadeer, Rajamani, Rehof
• Summarizing Procedures in Concurrent Programs.
  
• POPL 2004
• Implemented in ZING
• Approx. one year sustained effort

15
Summarization for sequential programs

• Procedure summarization (Sharir-Pnueli 81,
  Reps-Horwitz-Sagiv 95) is the key to efficiency

int x void incr_by_2() x x
void main() x 0 incr_by_2()
x 0 incr_by_2()

• Bebop, ESP, Moped, MC, Prefix,

16
What is a summary in sequential programs?

• Summary of a procedure P Set of all (pre-state
  ? post-state) pairs obtained by invocations of P

x ? x 0 ? 2 1 ? 3
17
Assertion checking for sequential programs

• Boolean program with
• g number of global vars
• m max. number of local vars in any scope
• k size of the CFG of the program
• Complexity is O( k ? 2 O(gm) ), linear in the
  size of CFG
• Summarization enables termination in the presence
  of recursion

18
Assertion checking forconcurrent programs
Ramalingam 00 There is no algorithm for
assertion checking of concurrent boolean
programs, even with only two threads.
19
Our contribution

• Precise semi-algorithm for verifying properties
  of concurrent programs
• based on model checking
• procedure summarization for efficiency
• Termination for a large class of concurrent
  programs with recursion and shared variables
• Generalization of precise interprocedural
  dataflow analysis for sequential programs

20
What is a summary in concurrent programs?

• Unarticulated so far
• Naïve extension of summaries for sequential
  programs do not work

21
Attempt 1
Advantage summary computable as in a sequential
program
Disadvantage summary not usable for executions
with interference from other threads
22
Attempt 2
Advantage Captures all executions

• Disadvantage s and s must comprise full program
  state
• summaries are complicated
• do not offer much reuse

23
The theory of movers (Lipton 75)

• R right movers
• lock acquire

• L left movers
• lock release

• B both right left movers
• variable access holding lock

• N non-movers
• access unprotected variable

24
Transaction
Lipton any sequence (RB) (N?) (LB) is a
transaction
Other threads need not be scheduled in the middle
of a transaction
? Transactions may be summarized
25
If a procedure body is a single transaction,
summarize as in a sequential program
bool availableN mutex m int
getResource() int i 0 L0
acquire(m) L1 while (i lt N) L2 if
(availablei) L3 availablei
false L4 release(m) L5 return
i L6 i L7
release(m) L8 return i
Choose N 2 Summaries ? m, (a0,a1) ?
? ? i, m, (a0,a1) ? ? 0, (0, 0) ?
? ? 2, 0, (0,0) ? ? 0, (0, 1) ? ? ?
1, 0, (0,0) ? ? 0, (1, 0) ? ? ? 0, 0,
(0,0) ? ? 0, (1, 1) ? ? ? 0, 0, (0,1) ?
26
Transactional procedures

• In the Atomizer benchmarks (Flanagan-Freund 04),
  a majority of procedures are transactional

27
What if a procedure body comprises
multiple transactions?
bool availableN mutex mN int
getResource() int i 0 L0 while
(i lt N) L1 acquire(mi) L2 if
(availablei) L3 availablei
false L4 release(mi) L5
return i else L6
release(mi) L7 i
L8 return i
Choose N 2 Summaries ? pc,i,(m0,m1),(a0
,a1) ? ? ? pc,i,(m0,m1),(a0,a1) ?
? L0, 0, (0,), (0,) ? ? ? L1, 1, (0,),
(0,) ? ? L0, 0, (0,), (1,) ? ? ? L5, 0,
(0,), (0,) ? ? L1, 1, (,0), (,0) ? ? ?
L8, 2, (,0), (,0) ? ? L1, 1, (,0), (,1) ?
? ? L5, 1, (,0), (,0) ?
28

• What if a transaction
• starts in caller and ends in callee?
• starts in callee and ends in caller?

29

• What if a transaction
• starts in caller and ends in callee?
• starts in callee and ends in caller?

int x mutex m
void foo() acquire(m) x
bar() x-- release(m)
void bar() release(m)
acquire(m)
1
2

• Solution
• Split the summary into pieces
• Annotate each piece to indicate whether
• transaction continues past it

30
Two-level model checking

• Top level performs state exploration
• Bottom level performs summarization
• Top level uses summaries to explore reduced set
  of interleavings
• Maintains a stack for each thread
• Pushes a stack frame if annotated summary edge
  ends in a call
• Pops a stack frame if annotated summary edge ends
  in a return

31
Termination

• Theorem
• If all recursive functions are transactional,
  then our algorithm terminates.
• The algorithm reports an error iff there is an
  error in the program.

32
Concurrency recursion
Summaries for foo ? pc,r,m,g ? ? ?
pc,r,m,g ? ? L0,1,0,0 ? ? ? L5,1,0,1 ? ?
L0,1,0,1 ? ? ? L5,1,0,2 ?
33
Summary (!)

• Transactions enable summarization
• Identify transactions using the theory of movers
• Transaction boundaries may not coincide with
  procedure boundaries
• Two level model checking algorithm
• Top level maintains a stacks for each thread
• Bottom level maintains summaries

34
Sequential programs

• For a sequential program, the whole execution is
  a transaction
• Algorithm behaves exactly like classic
  interprocedural dataflow analysis