Tayfun Elmas, Serdar Tasiran Ko University, Istanbul, Turkey

1
VyrdMC Driving Runtime Refinement Checking Using
Model Checkers

• Tayfun Elmas, Serdar TasiranKoç University,
  Istanbul, Turkey

2
Motivation
Verifying ConcurrentData Structures

• Widely-used software systems are built on
  concurrent data structures
• File systems, databases, internet services
• Standard Java and C class libraries
• Intricate synchronization mechanisms to improve
  performance
• Prone to concurrency errors
• Concurrency errors
• Data loss/corruption
• Difficult to detect, reproduce through testing

3
Our Approach
Runtime Checking of Refinement

• Use refinement as correctness criterion
• More thorough than assertions
• More observability than pure testing
• Linearizability, atomicity
• For each execution of Impl
• there exists an equivalent atomic execution of
  Impl
• Refinement
• For each execution of Impl
• there exists an equivalent, atomic execution of
  Spec
• Refinement uses separate, more permissive Spec
• Example Allow methods to terminate exceptionally
  
• Model failure due to resource contention
• Runtime verification Check refinement using
  execution traces
• Can handle industrial-scale programs
• Intermediate between testing exhaustive
  verification

4
Outline

• Motivating example
• Refinement
• I/O-refinement
• View-refinement
• The VYRD tool
• Experience
• Conclusions

5
Multiset
I/O Refinement
6
Experience
The Boxwood Project
7
Experience
Concurrency Bug in Cache
Flush()starts
Write(handle,AB) starts
Flush() ends
Write(handle, AB)ends
8
Experience
Concurrency Bug in Cache

• Concurrent execution of Write and Flush on the
  same handle
• Write to a dirty entry not locked properly by
  Write
• Hard to catch through testing
• Cache entry is correct, permanent version wrong
• As long as Reads hit in Cache,return value
  correct
• Caught through testing only if
• Cache fills, clean entry in Cache is evicted
• No Writes to entry in the meantime
• Entry read after eviction
• Very unlikely

9
View-refinement
More Observability

• I/O-refinement may miss errors
• Our solution View-refinement
• I/O-refinement correspondence between states
  of Impl and Specat commit points
• Add new observed action to ?
• ?call, commit and return action
• Label commit actions with state information
• Catches state discrepancy right when it happens
• Early warnings for possible I/O refinement
  violations
• Abstraction function computes value of view
  variable at commit point
• Written by user
• An extra method of the component

10
View-refinement
Checking Refinement
11
Experience
Experimental Results

• Applied to industrial-scale designs
• Boxwood (30K LOC)
• Scan Filesystem
• Java Libraries
• Moderateinstrumentation effort
• I/O-refinement
• Low logging andverification overhead
• View-refinement
• More effectivein catching errors
• May take morelogging overhead

12
Observation, VyrdMC

• Weakness of runtime verification Need many
  threads, method calls to trigger concurrency
  error
• View refinement improves observability, but no
  better if bug not triggered
• Small test case (e.g. two threads, one method
  call each) sufficient, but
• must start from non-trivial initial state
• must pick the right interleaving
• The interleaving that triggers bug not known a
  priori

13
Observation, VyrdMC

• Weakness of runtime verification Poor coverage
• Need many threads, method calls to trigger
  concurrency error
• View refinement More observability, but no
  better if bug not triggered
• Small test case (e.g. two threads, one method
  call each) sufficient, but
• must start from non-trivial initial state
• must pick the right interleaving (not known a
  priori)
• Idea
• Lead design to interesting initial state first
• Run very small, multi-threaded test
• Explore all distinct thread interleavings of
  fixed testwith execution-based model checker
• Computationally feasible
• Get good checking from small test
• Controllability, coverage Execution-based model
  checker
• Observability, View refinement

14
Outline

• Motivating example
• The VYRD tool
• Experience
• Conclusions

15
The VYRD Tool
Architecture

• Instrument Impl
• Write abstraction function
• Testing thread
• Generates Impl trace
• Logs actions executed
• Verification thread
• Replays Implreplay and Spec
• using the log
• ComputesviewImpl and viewSpec
• Checks refinement for?-traceImpl and ?-traceSpec
• Online/Offline checking

16
The VYRD Tool
Test Harness
Thread 1
Thread 2
Thread 3
LookUp(3)
Insert(3)
Delete(3)

• Each thread
• issues a small number (one?) of method calls
  consecutively
• Pick arguments so that
• threads contend over same part of data structure

17
The VYRD Tool
Code Instrumentation

• Source code/bytecode instrumentation
• in order to log actions
• in the order they happen
• Commit actionsannotated by user

18
Guiding Vyrd with a Model Checker
Call InsertP(5,7)
Call InsertP(5,7)
Call InsertP(6,8)
Call InsertP(6,8)
Read A0.elt
Read A0.elt
Read A0.elt
A0.content5
A0.content5
Read A1.elt
A1.content7
A1.content7
ATOMIC
ATOMIC
A0.validtrue
A0.validtrue
A1.validtrue
A1.validtrue
A0.content6
InsertP(5,7)Returns success
A2.content8
Read A2.elt
Read A2.elt
ATOMIC
A2.content6
A0.validtrue
A2.validtrue
Read A3.elt
ATOMIC
A3.content8
InsertP(5,7)Returns success
A2.validtrue
A3.validtrue
InsertP(6,8)Returns success
InsertP(6,8)Returns success
19
ExplicitMC (like Java PathFinder)

• Important side benefit Automates, streamlines
  logging and replaying or implementation
• Makes it program independent
• Logged action Bytecode executed in VM
• Model checkers VM has hooks for calling
  appropriate methods of refinement checker, logger
  after each bytecode instruction
• Semantics of replay for VMs actions (bytecodes)
  defined, already implemented in model checkers
  VM
• Refinement checker simply uses it
• Automates most labor-intensive part of runtime
  refinement checking
• Programmer must only insert commit point
  annotations, write abstraction function

20
Handling Backtracking
Call InsertP(5,7)

• Special log entries indicate parent-child
  relationships in tree, jumps to ancestors
• Vyrd resets to initial state and replays actions
  from root to back-tracked ancestor

Call InsertP(6,8)
Read A0.elt
Read A0.elt
Read A1.elt
A0.content5
A1.content7
ATOMIC
A0.validtrue
A1.content7
ATOMIC
A1.validtrue
A0.validtrue
InsertP(5,7)Returns success
A1.validtrue
A0.content6
Read A2.elt
A2.content8
A2.content6
Read A2.elt
ATOMIC
Read A3.elt
A0.validtrue
ATOMIC
A3.content8
A2.validtrue
A2.validtrue
A3.validtrue
InsertP(5,7)Returns success
InsertP(6,8)Returns success
InsertP(6,8)Returns success
21
Stateless MC (like Verisoft)

• No state caching, just store path from root to
  node
• Terminates because state-transition diagram is a
  DAG/tree.
• Benefits
• Efficient
• Vyrd is re-started anyway
• At re-starts, Vyrd resets state,
• No need to rewind log
• Disadvantages
• Must manually implement logging, replay
• Must instrument implementation source or byte code

22
Partial order reductions

• Pure blocks, annotations
• If pure block terminates normally, no net
  modification of program state
• Example
• Lock node,
• Check if data is there,
• Release lock if it isnt
• Many industrial-scale programs make extensive use
  of pure blocks
• Enumerating all distinct interleavings of even
  two concurrent methods infeasible if purity is
  not exploited
• Solution Use non-deterministic abstraction of
  pure block
• Makes pure execution of pure block independent of
  any other action

23
Experience
VerifyingCacheChunk Manager

• Highly concurrent access to each module
• Methods for manipulating (handle,byte-array)
  pairs
• View for CacheChunkManagerset of
  (handle,byte-array) pairs
• For each handle
• If Cache hit and dirty entry Get byte-array from
  Cache
• If Cache hit and clean entry Get byte-array from
  Cache
• Cache and Chunk Manager must have same byte-array
• If Cache miss Get byte-array from Chunk Manager

24
Assume-Guarantee Reasoning for Multi-layered
Architectures

• Multi-layered architecture
• Each layer uses layer immediately below
• Layers do not call methods from componentsabove

• One-pass verification
• Run C1 C2 ... Cn all together at once
• When replaying Ci run it in conjunction with a
  separate instance of Si1 Si2 ...
  Sn
• Models the assumption that the environment of Ci
  respects its spec for this execution
• Assumption verified when Ci1 Ci2, ..., Cn
  checked for refinement violations on the same
  execution

S1
C1
S2
C2
Specifications (Atomized Versions)
Components
S3
C3
...
...
Sn
Cn
25
Future Work

• Implementing VyrdMC around Java PathFinder
• Including automatic program abstraction to
  exploit pure annotations
• Stateless exploration in Java PathFinder?