CS294-32: Dynamic Partial Order Reduction

1
CS294-32 Dynamic Partial Order Reduction

• Koushik Sen
• UC Berkeley

2
Exponential Blowup
Computation tree for a single thread
Conditional statement
Computation tree for two threads
Scheduler choice thread switch
3
Solution

• All paths in the tree are not important for
  statement reachability
• Many paths are equivalent to each other
• Prune equivalent paths gt Partial Order Reduction
• Generate inputs along with Partial Order Reduction

4
Equivalent Paths
x0, y0
x0, y0
One partial order
Same partial order Different linear order gt
Different Path gt Equivalent Path
5
Independent transitions
s

• B and R are independent transitions if
• they commute B R R B
• neither enables nor disables the other

• Example x 3 and y 2 are independent

6
Existing Approaches

• Static Partial Order Reduction
• Valmari 91, Peled 93, Godefroid 96, SPIN model
  checkerby Holzmann, Verisoft
• Limitation
• Results in a large dependent relation
• Pointers -gt Whether two pointers point to the
  same location is determined conservatively (May
  point-to)
• Results in over-approximation of the dependency
  relation
• Limited POR

7
Example static partial-order reduction
Global Vars lock m int i1,i2 int x0 int
n100 char a
Thread 2 lock(m) i2 x unlock(m) for(
i2ltn i22) ai2 r
Thread 1 lock(m) i1 x unlock(m)
for( i1ltn i12) ai1 b

• Static analysis gives
• i1, i2 are thread-local
• x is protected by m
• but ai1 and ai2 may alias
• Static POR gives O(n2) explored states and
  transitions
• but only two possible terminating states

never alias (in practice)
8
Dynamic partial-order reduction

• Static POR relies on static analysis
• to yield approximate information about run-time
  behavior
• pointers gt coarse information gt limited POR gt
  path explosion
• Dynamic POR
• while model checker executes the program,
• it sees exactly which threads access which
  locations
• use to simultaneously reduce the path space while
  model-checking

9

• Focus on Race-Detection and Flipping Algorithm
  and Concolic Testing jCUTE
• Race-Detection and Flipping Algorithm is a
  simplified form of DPOR
• Proof of correctness in the presence of inputs
  and conditionals

10
Event

• (t,l,a)
• If thread t executes the statement labeled l and
  the access type is a
• a 2 w,r,l,u,?
• An execution path ? is a sequence of events

11
Sequential Relation

• e (t,l,a) and e (t,l,a)
• e C e
• e e', or
• tt and e appears before e' in ?, or
• t ? t, t created the thread t, and e appears
  before e'' in ?, where e'' is the fork event on t
  creating the thread t, or
• there exists an event e'' in ? such that e C e''
  and e'' C e'.

12
Causal Relation (Happens-Before Relation)

• e (t,l,a) and e (t,l,a)
• e ¹ e
• e C e, or
• e appears before e' in ? and both access a shared
  memory location m and one of the accesses is
  update (write, lock acquire, release), or
• there exists an event e'' in ? such that e ¹ e''
  and e'' ¹ e'.
• ¹ is a partial order relation

13
Equivalent Paths

• Definition Two execution paths are equivalent if
  they are linearizations of the same partial order
• Proposition Exploration of one linear order of
  each partial order is sufficient for statement
  reachability

14
Race Relation

• Not so strict definition (see paper for the
  strict definition)
• e (t,l,a) and e (t,l,a)
• e l e
• e ¹ e
• Not (e C e or e C e)
• There exists no e1 such that e ¹ e1 and e1 ¹ e
• Where e1 is not equal to e or e

1. Events in race relation can be permuted by
   changing schedule
2. What happens if we have locks? (see paper)

15
DPOR (Race-detection and flipping) Example

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

16
DPOR (Race-detection and flipping) Example

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 1
17
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 1




18
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 1

x 1
t1
y 2

Race
y 3

x 4
19
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 1

x 1
Backtrack Here
t1
y 2

Race
y 3

x 4
20
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 1

x 1
Backtrack Here
t1
y 2
Race
y 3
x 4
21
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 2

x 1
t1
y 3

x 4

y 2
22
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 2
t1
x 1
t1 ,t2
Race
y 3

x 4
Race

y 2
23
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 2
t1
x 1
Cannot Backtrack Here
t1 ,t2
Race
y 3

x 4
Race

y 2
24
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 2
Backtrack Here
t1
x 1
t1 ,t2
Race
y 3

x 4
Race

y 2
25
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 2
Backtrack Here
t1
x 1
Race
y 3
x 4
Race
y 2
26
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 3
t1
y 3

x 4

x 1

y 2
27
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 3
t1
y 3
t2
x 4

Race
x 1

y 2
28
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 3
t1
y 3
Backtrack Here
t2
x 4

Race
x 1

y 2
29
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 3
t1
y 3
Backtrack Here
t2
x 4
Race
x 1
y 2
30
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 4
t1
y 3
t2
x 1

y 2

x 4
31
DPOR (Race-detection and flipping) Example
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 4
Cannot Backtrack Here
t1,t2
y 3
Cannot Backtrack Here
t1,t2
Race
x 1

y 2
Race

x 4
Done!
32
DPOR (POPL 05) Example
Postponed
Persistent

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 1


x 1
t1
t2
y 2


Race
y 3


x 4
33
DPOR (POPL 05) Example
Postponed
Persistent

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 2
t1
t2
x 1
t1 ,t2
t1 ,t2
y 3


x 4


y 2
34
DPOR (POPL 05) Example
Postponed
Persistent

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 3
t1
t2
y 3
t2
t1
x 4


x 1


y 2
35
DPOR (POPL 05) Example
Postponed
Persistent

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 4
t1,t2
t1,t2
y 3
t1,t2
t1,t2
x 1


y 2


x 4
36
DPOR Problem

• Thread t1
• 1 x 1
• 2 x 2
• Thread t2
• 1 y 1
• 2 x 3

• DPOR (both approaches) explores all 6 execution
  paths gt No reduction
• Example in the POPL 05 paper has error
• Think about it
• Need Sleep Set to obtain reduction

37
Sleep Set Example
38
DPOR (Race-detection and flipping) Example

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

39
DPOR (Race-detection and flipping) Example

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 1
40
DPOR (Race-detection and flipping) Example
Postponed
Delayed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 1








41
DPOR (Race-detection and flipping) Example
Postponed
Delayed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 1


x 1
t1

y 2


Race
y 3


x 4
42
DPOR (Race-detection and flipping) Example
Postponed
Delayed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 1


x 1
Backtrack Here
t1

y 2


Race
y 3


x 4
43
DPOR (Race-detection and flipping) Example
Postponed
Delayed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 1


x 1
Backtrack Here
t1

y 2

Race
y 3

x 4
44
DPOR (Race-detection and flipping) Example
Postponed
Delayed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 2


x 1
t1
t1
y 3

t1
x 4


y 2
45
DPOR (Race-detection and flipping) Example
Postponed
Delayed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 2
t1

x 1
t1
t1
Race
y 3

t1
X
x 4
Race


y 2
46
DPOR (Race-detection and flipping) Example
Postponed
Delayed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 2
t1

x 1
Nothing to Backtrack Here
t1
t1
Race
y 3

t1
X
x 4
Race


y 2
47
DPOR (Race-detection and flipping) Example
Delayed
Postponed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 2
Backtrack Here
t1

x 1
t1
t1
Race
y 3

t1
X
x 4
Race


y 2
48
DPOR (Race-detection and flipping) Example
Postponed
Delayed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 2
Backtrack Here
t1

x 1
t1
Race
y 3
t1
X
x 4
Race

y 2
49
DPOR (Race-detection and flipping) Example
Postponed
Delayed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 3
t1
t1
y 3

t1
x 4


x 1


y 2
50
DPOR (Race-detection and flipping) Example
Postponed
Delayed

• Thread t1
• 1 x 1
• 2 y 2
• Thread t2
• 1 y 3
• 2 x 4

Execution 3
t1
t1
y 3

t1
x 4


Race
X
x 1


y 2
51
jCUTE

• Key Observation Concolic execution is ideal for
  testing concurrent programs with complex data
  inputs
• Use symbolic execution to generate new inputs
• Use concrete execution to perform partial order
  reduction

?
52
jCUTE

• Key Observation Concolic execution is ideal for
  testing concurrent programs with complex data
  inputs
• Use symbolic execution to generate new inputs
• Use concrete execution to perform partial order
  reduction
• Explore Interesting thread schedules or total
  orders
• Where to perform context switches?
• How to perform context switches?

?
53
jCUTE

• Key Observation Concolic execution is ideal for
  testing concurrent programs with complex data
  inputs
• Use symbolic execution to generate new inputs
• Use concrete execution to perform partial order
  reduction
• Explore Interesting thread schedules or total
  orders
• Where to perform context switches?
• Detect data race and lock race
• How to perform context switches?
• Hijack the scheduler using semaphores
• Insert semaphores through instrumentation

?
54
jCUTE Example

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

55
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0

56
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0

x ? 3, z ? 17
x ? 3, z ? z0

57
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1
x ? 3, z ? 17
x ? 3, z ? z0


x ? 2, z ? 17
x ? 2, z ? z0
58
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1
x ? 3, z ? 17
x ? 3, z ? z0


x ? 2, z ? 17
x ? 2, z ? z0
2z01!2
x ? 2, z ? 17
x ? 2, z ? z0
59
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1
x ? 3, z ? 17
x ? 3, z ? z0


x ? 2, z ? 17
x ? 2, z ? z0
2z01!2
x ? 2, z ? 17
x ? 2, z ? z0
Backtrack Here Solve 2z012 No Solution
60
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1
x ? 3, z ? 17
x ? 3, z ? z0
x ? 2, z ? 17
x ? 2, z ? z0
x ? 2, z ? 17
x ? 2, z ? z0
Backtrack Here
61
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1
62
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1
x ? 2, z ? 17
x ? 2, z ? z0

63
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1
x ? 2, z ? 17
x ? 2, z ? z0

2z01!2

x ? 2, z ? 17
x ? 2, z ? z0
64
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1
x ? 2, z ? 17
x ? 2, z ? z0
t2
2z01!2

x ? 2, z ? 17
x ? 2, z ? z0
x ? 3, z ? 17
x ? 3, z ? z0
65
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1
x ? 2, z ? 17
x ? 2, z ? z0
t2
2z01!2

x ? 2, z ? 17
x ? 2, z ? z0
x ? 3, z ? 17
x ? 3, z ? z0
Backtrack Here Solve 2z012 No Solution
66
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1
x ? 2, z ? 17
x ? 2, z ? z0
t2
x ? 2, z ? 17
x ? 2, z ? z0
x ? 3, z ? 17
x ? 3, z ? z0
Backtrack Here
67
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1
t2
68
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1
x ? 2, z ? 17
x ? 2, z ? z0
t2
69
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1 ,t2
x ? 2, z ? 17
x ? 2, z ? z0
t2

x ? 3, z ? 17
x ? 3, z ? z0
70
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1 ,t2
x ? 2, z ? 17
x ? 2, z ? z0
t1 ,t2

x ? 3, z ? 17
x ? 3, z ? z0
2z01!3
x ? 3, z ? 17
x ? 3, z ? z0
71
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 17
x ? 0, z ? z0
t1 ,t2
x ? 2, z ? 17
x ? 2, z ? z0
t1 ,t2

x ? 3, z ? 17
x ? 3, z ? z0
2z01!3
x ? 3, z ? 17
x ? 3, z ? z0
Backtrack Here Solve 2z013 Solution z 1
72
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 1
x ? 0, z ? z0
t1 ,t2
t1 ,t2

73
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 1
x ? 0, z ? z0
t1 ,t2
x ? 2, z ? 1
x ? 2, z ? z0
t1 ,t2

x ? 3, z ? 1
x ? 3, z ? z0
2z013
x ? 3, z ? 1
x ? 3, z ? z0

74
jCUTE Example
Concrete State
Symbolic State
Path Constraint Postponed Set

• z input()
• Thread t1
• 1 x 3
• Thread t2
• 1 x 2
• 2 if (2z 1 x)
• 3 ERROR

x ? 0, z ? 1
x ? 0, z ? z0
t1 ,t2
x ? 2, z ? 1
x ? 2, z ? z0
t1 ,t2

x ? 3, z ? 1
x ? 3, z ? z0
2z013
x ? 3, z ? 1
x ? 3, z ? z0

Nothing to Backtrack
75
Race Detection

• Dynamic Vector Clock Algorithm FSE03,TACAS04
• Vector clock V Threads ! Nat
• Vi be vector clock for each thread ti.
• Vxa and Vxw vector clocks for each shared
  variable x.
• Algorithm
• if eik is a shared memory access, then
• Vii à Vii 1
• if eik is a read of a variable x then
• Vi à maxVi,Vxw
• Vxa à maxVxa,Vi
• if eik is a write of a variable x then
• Vxw à Vxa à Vi à maxVxa,Vi

Lemma For any two events e ¹ e iff Ve Ve
76
Race Flipping Hijack Thread Scheduler

• Ensure that only one thread is executing
• Create a tester thread (tsched)
• Associate a semaphore sem(t) with each thread t
• Before any shared memory access by t
• release control to the tester thread
• V(sem(tsched)) P(sem(t))
• Tester thread schedules a thread t
• V(sem(t)) P(sem(tsched))

77
jCUTE

• jCUTE can test multi-threaded Java programs
• URL
• http//osl.cs.uiuc.edu/ksen/cute/
• Next generation testing tools
• Combines Testing and Model-Checking
• jCUTE supports generation of JUnit test cases
• The tools also support replay of a buggy execution

78
Sun Microsystems JDK 1.4 Library

• java.util package provides thread-safe
  data-structure classes
• LinkedList, ArrayList, HashSet, TeeMap, etc.
• Widely used
• Found previously undocumented concurrency related
  problems
• Data race, Infinite Loop, Uncaught Exceptions,
  and Deadlocks

• List l1  
• Collections.synchronizedList(new LinkedList())
• List l2  
• Collections.synchronizedList(new LinkedList())
• l1.add(null)
• l2.add(null)
• // Create two threads
• // let thread 1 run
• l1.clear()
• // let thread 2 run
• l2.containsAll(l1)

79
Sun Microsystems JDK 1.4 Library
Name Runtime in seconds of Paths of Threads Branch Coverage of Functions Tested of Bugs Found data races deadlocks infinite loops exceptions
Vector 5519 20000 5 76.38 16 1902
ArrayList 6811 20000 5 75.00 16 3903
LinkedList 4401 11523 5 82.05 15 3311
LinkedHashSet 7303 20000 5 67.39 20 3902
TreeSet 7333 20000 5 54.93 26 4902
HashSet 7449 20000 5 69.56 20 19902
80
Honeywells DEOS real-time scheduling kernel

• Operating system developed for use in small
  business aircraft
• jCUTE found the subtle time-partitioning error in
  lt 1 minute
• Java Pathfinder from NASA Ames ran out of memory
  on the original program
• Had to test manually created abstraction
• Took 11 minutes to discover the same error in the
  abstraction