Iterators Revisited: Proof Rules and Implementation

1
Iterators Revisited Proof Rules and
Implementation

• Bart Jacobs, Erik Meijer,Frank Piessens, Wolfram
  Schulte

2
Outline

• Iterators in C
• How to specify and verify iterators and foreach
  loops?
• How to prevent interference between iterators and
  foreach loops?
• What are nested iterators?
• How to implement nested iterators efficiently?

3
The Iterator pattern in C 2.0

• public interface IEnumeratorltTgt
• T Current get
• bool MoveNext()
• public interface IEnumerableltTgt
• IEnumeratorltTgt GetEnumerator()

4
Foreach Loops

• foreach (T x in C) S
• is implemented as

IEnumerableltTgt c C IEnumeratorltTgt e
c.GetEnumerator() while (e.MoveNext()) T x
e.Current S
5
C 2.0 Iterator Methods

• IEnumerableltintgt FromTo(int a, int b)
• for (int x a x lt b x)
• yield return x
• is implemented as
• IEnumerableltintgt FromTo(int a, int b)
• return new FromTo_Enumerable(a, b)

Compiler-generated class
6
C 2.0 Iterator Methods

• class FromTo_Enumerator IEnumeratorltintgt
• int a int b int pc int x int current
• public FromTo_Enumerator(int _a, int _b) a
  _a b _b
• public int Current get return current
  
• public bool MoveNext()
• switch (pc)
• case 0 x a goto case 1
• case 1 if (!(x lt b)) goto case 4
• case 2 current x pc 3 return
  true
• case 3 x goto case 1
• case 4 pc 4 return false

7
How to specify and verify iterators?

• static IEnumerableltintgt FromTo(int a, int b)
• requires a lt b
• invariant forallint i in (0 b a)
  valuesi a i
• invariant values.Count lt b a
• ensures values.Count b a
• for (int x a x lt b x)
• invariant values.Count x a
• yield return x

Enumeration invariant must be proved at start of
iterator method
and after each yield return statement.
Ensures clause must be proved at end of method
(and at yield break statements)
8
How to specify and verify foreach loops?

• int sum 0
• Seqltintgt values new Seqltintgt()
• foreach (int x in FromTo(1, 3))
• invariant sum Math.Sum(values)
• free invariant forallint i in
  (0values.Count) valuesi1i
• free invariant values.Count lt 3 1
• int x havoc x values.Add(x)
• assume forallint i in (0values.Count)
  valuesi1i
• assume values.Count lt 3 1
• sum x
• assume values.Count 3 1
• assert sum 6

while ()
9
Interference

• Listltintgt xs new Listltintgt()
• xs.Add(1) xs.Add(2) xs.Add(3)
• int sum 0
• foreach (int x in xs)
• sum x xs.Remove(0)
• //assert sum 6

• class ListltTgt IEnumerableltTgt
• IEnumeratorltTgt GetEnumerator()
• int n Count
• for (int i 0 i lt n i)
• yield return thisi

ArgumentOutOfRangeException !
Parties execute in an interleaved fashion
But we wish to verify them as if they executed in
isolation
Proposed solution Prevent either party from
seeing the other partys effects
10
Proposed Solution
Enforced using an extension of the Boogie
methodology
Error unsatisfiedrequires this.readCount 0

• Listltintgt xs new Listltintgt()
• xs.Add(1) xs.Add(2) xs.Add(3)
• int sum 0
• foreach (int x in xs)
• sum x xs.Remove(0)
• //assert sum 6

• class ListltTgt IEnumerableltTgt
• IEnumeratorltTgt GetEnumerator()
• reads this
• int n Count
• for (int i 0 i lt n i)
• yield return thisi

reads clause declares the set ofpre-existing
objects the iterator method wishes to read
And the foreach loop body may not write the
objects in the reads clause
The iterator method may not read or write any
other pre-existing objects
11
The Boogie methodology

• Enforces object invariants
• Uses a dynamic ownership system
• Each object gets two extra fields
• bool inv
• bool writable
• o.f x requires o.writable !o.inv
• unpack o requires o.writable o.inv
• Sets o.inv false
• Makes owned objects writable
• pack o reverses the effect of unpack o

12
Adding read-only objects to the Boogie
methodology

• Each object gets three special fields
• bool inv
• bool writable
• int readCount // never negative
• o.f x requires
• o.writable o.readCount 0 !o.inv
• x o.f requires
• o.writable 0 lt o.readCount

13
Read-only Methods
read (o) S means assert o.writable 0 lt
o.readCount assert o.inv o.readCount foreach
(Owned field f of o) o.f.readCount S fore
ach (Owned field f of o) o.f.readCount-- o.
readCount--

• partial class ListltTgt
• IEnumerableltTgt
• IEnumeratorltTgt GetEnumerator()
• reads this
• int n Count
• for (int i 0 i lt n i)
• yield return thisi

• partial class ListltTgt
• Owned T elems
• T thisint index
• get
• requires
• inv
• (writable
• 0 lt readCount)
• read (this)
• return elemsindex

How is this call verified?
14
General Approach to Dependent Objects

• (Preliminary thoughts)
• Each object has a set of dependee objects
• Object may declare dependent invariants that
  dereference dependee objects
• Dependent invariants become requires clauses,
  unless the dependent object is in Reader mode
• Reader mode is statically nested within read
  blocks on dependee objects
• Generic user of Iterator interface will require
  that Iterator object is in Reader mode

15
What areNested Iterators?

• yield foreach E
• means
• foreach (T x in E) yield return x

but is implemented with better time complexity if
E evaluates to a nested iterator
and with less garbage generation if E is a
recursive call of the same iterator
16
Nested Enumerations

• class Tree
• int value ListltTreegt! children
• IEnumerableltTreegt Nodes get
• yield return this
• for (int i 0 i lt children.Count i)
• foreach (Tree t in childreni.Nodes)
• yield return t

Number of IEnumerableltTreegt and IEnumeratorltTreegt
objects created is O(n)
Number of recursive MoveNext calls is O(nlog(n))
Assume a balanced tree of n nodes
17
Nested Iterators

• class Tree
• int value ListltTreegt! children
• IEnumerableltTreegt Nodes get
• yield return this
• for (int i 0 i lt children.Count i)
• yield foreach childreni.Nodes

18
Nested Iterators
Space usage O(log(n))
Nb. of reallocations O(log(log(n))

• struct TreeStackFrame Tree self int pc int i
  
• class TreeEnumerator IEnumeratorltTreegt
• Tree current TreeStackFrame! stack new
  TreeStackFrame8 int top
• public TreeEnumerator(Tree self) Push(self,
  0, 0)
• public bool MoveNext()
• while (0 lt top)
• switch (stacktop.pc)
• case 0 current this
  stacktop.pc 1 return true
• case 1 stacktop.i 0 goto
  case 2
• case 2 if (!(stacktop.i lt
  stacktop.self.children.Count)) goto case 4
• stacktop.pc 3
  Push(stacktop.self.childrenstacktop.i, 0,
  0)
• break
• case 3 stacktop.i goto case
  2
• case 4 Pop() break
• return false
• public Tree Current get return current
  
• void Push(Tree self, int pc, int i)
  void Pop() top--

Total nb. of loop iterations across all MoveNext
calls O(n)
Nb. of TreeStackFrame copy operations O(log(n))
19
Nested Iterators
Balanced tree of n nodes Balanced tree of n nodes Linked list of length n Linked list of length n
Time Allocations Time Allocations
Plain iterators O(nlog(n)) O(n) O(n2) O(n)
Nested iterators O(n) O(n) O(n) O(n)
Recursive nested iterators O(n) O(log(log(n))) O(n) O(log(n))
But if we can statically detect tail recursion,
the nb. of allocations becomes O(1)