Introduction to Algorithms: Verification, Complexity, and Searching

1
Introduction to Algorithms Verification,
Complexity, and Searching

• Andy Wang
• Data Structures, Algorithms, and Generic
  Programming

2
Lecture Overview

• Components of an algorithm
• Sequential search algorithms
• Binary search algorithms
• Proving algorithm correctness
• Computational complexity
• TVector Retrospective

3
Algorithm Components

• Required components
• Assumptions
• Asserted outcomes
• Body
• Proof
• Optional components
• Time complexity
• Space complexity

4
Sequential Search

• Goal
• Find a specified value in a collection of values
• Idea
• Walk through the collection and test each value
• A simple, commonly used, algorithm

5
Sequential Search Requirements

• A way to differentiate things in the collection
• A starting position
• A way to move on to the next thing in the
  collection
• A way to stop

6
Sequential Search Algorithm

• Assumptions
• Collection L of data of type T
• Can iterate through L with begin(), next(),
  end()
• Outcomes
• Decide whether t is in L
• Return boolean (true/false)

7
Sequential Search Algorithm (2)

• Body (in pseudocode)
• for (T item begin(L) item ! end(L) item
  next(L))
• if (t item) return true
• if (t item) return true
• return false

8
Binary Search

• Goal
• Find a value in a collection of values
• Idea
• Divide and conquer

9
Binary Search (2)

• Requirements
• Collection must be array-like
• Can use an index to jump to any array element
• Collection must be sorted
• Efficiency
• Very fast
• No extra space required

10
Binary Searchthe idea
You are heading down to an exotic
restaurant with something exotic on your mind

Chocolate.Garlic.Pasta do not try this
at home
11
Binary Searchthe idea
0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

There are 8 items on the menu sorted
alphabetically
12
Binary Searchthe idea
0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

The list is quite long its time to do a
binary search
13
Binary Searchthe idea
Search range 0 - 7
0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

Compare the middle element ?
Chocolate garlic pasta 14
Binary Searchthe idea
Search range 0 - 3
0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream
Compare the middle element ?

Chocolate garlic pasta Blood pudding
15
Binary Searchthe idea
Search range 2 - 3
0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

Compare the middle element ?
Chocolate garlic pasta Chocolate garlic pasta
16
Binary Searchthe idea
Yum
17
Binary Search Algorithm

• Three versions
• Binary_search
• Lower_bound
• Upper_bound

18
Binary Search Algorithm (2)

• Assumptions
• Collection L of data type of T with size sz
• L is sorted
• Element t of type T
• Outcomes
• Binary_search true if t in L false, otherwise
• Lower_bound smallest j, where t
• Upper_bound smallest j, where t

19
Lower_bound Code

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid, high max
• while (low
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• return low

20
Lower_bound Code

• t Chocolate Garlic Pasta
• low 0
• high 7
• while (0
• mid (0 7) / 2 3
• if (L3
• low mid 1
• else
• high mid
• return low

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

21
Lower_bound Code

• t Chocolate Garlic Pasta
• low 0
• high 7
• while (0
• mid (0 7) / 2 3
• if (L3
• low mid 1
• else
• high mid
• return low

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

22
Lower_bound Code

• t Chocolate Garlic Pasta
• low 0
• high 3
• while (0
• mid (0 3) / 2 1
• if (L1
• low mid 1
• else
• high mid
• return low

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

23
Lower_bound Code

• t Chocolate Garlic Pasta
• low 0
• high 3
• while (0
• mid (0 3) / 2 1
• if (L1
• low mid 1
• else
• high mid
• return low

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

24
Lower_bound Code

• t Chocolate Garlic Pasta
• low 2
• high 3
• while (2
• mid (2 3) / 2 2
• if (L2
• low mid 1
• else
• high mid
• return low

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

25
Lower_bound Code

• t Chocolate Garlic Pasta
• low 2
• high 3
• while (2
• mid (2 3) / 2 2
• if (L2
• low mid 1
• else
• high mid
• return low

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

26
Lower_bound Code

• t Chocolate Garlic Pasta
• low 2
• high 2
• while (2
• return low 2

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

27
Upper_bound Code

• unsigned int upper_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid, high max
• while (low
• mid (low high) / 2
• if (t
• high mid
• else
• low mid 1
• return low

28
Upper_bound Code

• t Chocolate Garlic Pasta
• low 0
• high 7
• while (0
• mid (0 7) / 2 3
• if (t
• high mid
• else
• low mid 1
• return low

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

29
Upper_bound Code

• t Chocolate Garlic Pasta
• low 0
• high 7
• while (0
• mid (0 7) / 2 3
• if (t
• high mid
• else
• low mid 1
• return low

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

30
Upper_bound Code

• t Chocolate Garlic Pasta
• low 0
• high 3
• while (0
• mid (0 3) / 2 1
• if (t
• high mid
• else
• low mid 1
• return low

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

31
Upper_bound Code

• t Chocolate Garlic Pasta
• low 0
• high 3
• while (0
• mid (0 3) / 2 1
• if (t
• high mid
• else
• low mid 1
• return low

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

32
Upper_bound Code

• t Chocolate Garlic Pasta
• low 2
• high 3
• while (2
• mid (2 3) / 2 2
• if (t
• high mid
• else
• low mid 1
• return low

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

33
Upper_bound Code

• t Chocolate Garlic Pasta
• low 2
• high 3
• while (2
• mid (2 3) / 2 2
• if (t
• high mid
• else
• low mid 1
• return low

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

34
Upper_bound Code

• t Chocolate Garlic Pasta
• low 3
• high 2
• while (3
• return low 3

0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

35
Binary_search Code

• unsigned int binary_search(T L, unsigned sz, T
  t)
• unsigned int lb lower_bound(L, sz 1, t)
• if (lb
• if (t Llb)
• return true
• return false

36
If there are duplicate entries
0. Baby beer 1. Blood pudding 2. Chocolate garlic
pasta 3. Chocolate garlic pasta 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

Lower bound ?
Upper bound ?
37
If t is not in L
0. Baby beer 1. Blood pudding 2. Chocolate banana
crepe 3. Chocolate martini 4. Death by
Chocolate 5. Garlic ice cream 6. Popcorn-flavored
jelly beans 7. Saffron ice cream

Lower bound and upper bound ?
38
Issues of Proof

• Correctness
• Termination
• Correct outcome
• Performance
• Time complexity
• Space complexity

39
Correctness and Loop Invariants

• Correctness
• Loop termination
• State when entering the loop
• State when exiting the loop
• Loop invariants
• Conditions that remain true for each iteration
• Mathematical induction

40
What can go wrong?

• for (j 0 j
• compute(j)
• void compute(unsigned int j) --j
• n

41
InvariantsSequential Search

• boolean sequential_search
• for (T item first(L) item ! end(L) item
  next(L))
• // item is not the final item in the collection
• // current item has not been examined ?
  progress
• // L is finite
• if (t item)
• return true
• // t does not match the current item
• if (t item)
• return true
• return false

42
InvariantsBinary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid, high max
• while (low
• // (1) low
• // (2) Llow - 1 valid)
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• // (3) low
• // (4) high low has decreased
• // (5) Llow - 1 valid)
• return low

43
InvariantsBinary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid, high max
• while (low
• // (1) low
• // (2) Llow - 1 valid)
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• // (3) low
• // (4) high low has decreased
• // (5) Llow - 1 valid)
• return low

t does not have to be L
44
InvariantsBinary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid, high max
• while (low
• // (1) low
• // (2) Llow - 1
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• // (3) low
• // (4) high low has decreased
• // (5) Llow 1 valid)
• return low

45
InvariantsBinary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid, high max
• while (low
• // (1) low
• // (2) Llow 1
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• // (3) low
• // (4) high low has decreased
• // (5) Llow - 1 valid)
• return low

46
InvariantsBinary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid, high max
• while (low
• // (1) low
• // (2) Llow - 1 valid)
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• // (3) low
• // (4) high low has decreased
• // (5) Llow - 1 valid)
• return low

47
InvariantsBinary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid, high max
• while (low
• // (1) low
• // (2) Llow - 1 valid)
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• // (3) low
• // (4) high low has decreased
• // (5) Llow - 1 valid)
• return low

48
InvariantsBinary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid, high max
• while (low
• // (1) low
• // (2) Llow - 1 valid)
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• // (3) low
• // (4) high low has decreased
• // (5) Llow - 1 valid)
• return low

49
InvariantsBinary Search
high low mid low (low old_high)/2
low (old_high - low)/2 (old_high - old_low)/2
high low

unsigned int lower_bound(T L, unsigned max, T t)

unsigned int low 0, mid, high max

while (low

// (1) low

// (2) Llow - 1 valid)

mid (low high) / 2

if (Lmid

low mid 1

else

high mid

// (3) low

// (4) high low has decreased

// (5) Llow - 1 valid)

return low

50
InvariantsBinary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid, high max
• while (low
• // (1) low
• // (2) Llow - 1 valid)
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• // (3) low
• // (4) high low has decreased
• // (5) Llow - 1 valid)
• return low

Llow - 1 Lmid changed, t
51
InvariantsBinary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid, high max
• while (low
• // (1) low
• // (2) Llow - 1 valid)
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• // (3) low
• // (4) high low has decreased
• // (5) Llow - 1 valid)
• return low

Since low is unchanged, Llow Lmid t
52
InvariantsBinary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid, high max
• while (low
• // (1) low
• // (2) Llow - 1 valid)
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• // (3) low
• // (4) high low has decreased
• // (5) Llow - 1 valid)
• return low

Termination (3) shows that the loop can
terminate (4) shows progress
53
InvariantsBinary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid, high max
• while (low
• // (1) low
• // (2) Llow - 1 valid)
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• // (3) low
• // (4) high low has decreased
• // (5) Llow - 1 valid)
• return low

Return value (5) smallest t t
54
Computational Complexity

• Compares growth of two functions
• Independent of constant multipliers and
  lower-order effects
• Metrics
• Big O Notation
• Big Omega Notation
• Big Theta Notation

55
Big O Notation

• f(n)
• iff ? c, n0 0 0 n0

f(n)
56
Big Omega Notation

• ?(g(n))
• iff ? c, n0 0 0 n0

f(n)
57
Big Theta Notation

• f(n) ?(g(n))
• iff ? c1, c2, n0 0 0 ? n n0

f(n)
58
Examples

• 3n2 17
• ?(1), ?(n)
• O(n2), O(n3)
• ?(n2)

59
Analogous to Real Numbers

• f(n) O(g(n)) (a
• f(n) ?(g(n)) (a b)
• f(n) ?(g(n)) (a b)

60
Transitivity

• f(n) O(g(n)) (a
• f(n) ?(g(n)) (a b)
• f(n) ?(g(n)) (a b)
• f(n) O(g(n)) and g(n) O(h(n))
• f(n) O(h(n))

61
Anti-symmetry

• f(n) O(g(n)) (a
• f(n) ?(g(n)) (a b)
• f(n) ?(g(n)) (a b)
• f(n) O(g(n)) iff g(n) ?(f(n))

62
Symmetry

• f(n) O(g(n)) (a
• f(n) ?(g(n)) (a b)
• f(n) ?(g(n)) (a b)
• f(n) ?(g(n)) iff g(n) ?(f(n))

63
Reflexivity

• f(n) O(g(n)) (a
• f(n) ?(g(n)) (a b)
• f(n) ?(g(n)) (a b)
• f(n) O(f(n)), f(n) ?(f(n)), f(n) ?(f(n))

64
Dichotomy

• f(n) O(g(n)) (a
• f(n) ?(g(n)) (a b)
• f(n) ?(g(n)) (a b)
• If f(n) O(g(n)), g(n) ?(f(n)), f(n) ?(g(n))

65
Complexity Analysis

• Steps
• Find n size of input
• Find an atomic activity to count
• Find f(n) the number of atomic activities done
  by an input size of n
• Complexity of an algorithm complexity of f(n)

66
Algorithm ComplexityLoops

• for (j 0 j
• // 3 atomics
• Complexity ?(3n) ?(n)

67
Loops with Break

• for (j 0 j
• // 3 atomics
• if (condition) break
• Upper bound ?(3n) ?(n)
• Lower bound ?(3) ?(1)
• Complexity O(n)

68
Loops in Sequence

• for (j 0 j
• // 3 atomics
• for (j 0 j
• // 5 atomics
• Complexity ?(3n 5n) ?(n)

69
Nested Loops

• for (j 0 j
• // 2 atomics
• for (k 0 k
• // 3 atomics
• Complexity ?((2 3n)n) ?(n2)

70
Sequential Search

• for (T item begin(L) item ! end(L) item
  next(L))
• if (t item) return true
• if (t item) return true
• return false
• Input size n
• Atomic computation comparison
• Complexity O(n)

71
Binary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid,
• high max
• while (low
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• return low

• Input size n
• Atomic computation comparison
• Complexity
• k iterations x
• 1 comparison/loop
• ?(log(n))

72
Iteration Count for Binary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid,
• high max
• while (low
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• return low

• Iter search space
• 1 n
• 2 n/2
• 3 n/4
• k n/2(k-1)
• n/2(k-1) 1
• n 2(k-1)
• log2(n) k - 1

73
Iteration Count for Binary Search

• unsigned int lower_bound(T L, unsigned max, T t)
  
• unsigned int low 0, mid,
• high max
• while (low
• mid (low high) / 2
• if (Lmid
• low mid 1
• else
• high mid
• return low

• n/2(k-1) 1
• n 2(k-1)
• log2(n) k - 1
• log2(n) 1 k
• Complexity function f(n) log(n) iterations x 1
  comparison/loop ?(log(n))

74
TVector Retrospective

• Runtime complexity ?(1)
• PopBack(), Clear(), Front(), Back(), Empty()
• Size(), Capacity,
• Amortized runtime complexity ?(1)
• PushBack()
• Runtime complexity ?(n)
• Constructors, Destructor, Display(), Dump(),