Lectures on Recursive Algorithms

1
Lectures onRecursive Algorithms

• COMP 523 Advanced Algorithmic Techniques
• Lecturer Dariusz Kowalski

2
Announcements

• Web page
• www.csc.liv.ac.uk/darek/comp523.html
• Slides constantly updated based on your feedback
  check for a new version a few hours after the
  lecture(s)

3
Overview

• Previous lectures
• Algorithms correctness, termination, performance
• Asymptotic notation
• Popular asymptotic complexities log n, n, n log
  n, n2, ...
• Graphs - basic definitions and examples
• These lectures
• Recursive algorithms - basic recursion
• Searching algorithm in time O(log n)
• Finding majority in time O(n)
• Sorting in time O(n log n)
• Other examples

4
Algorithms based on recurrence

• Algorithmic scheme
• Reduce the input to smaller sub-input(s)
• Solve the problem for (some) sub-inputs
• Merge the obtained solution(s) into one global
  solution

5
Complexity analysis of recursive process

• Let T(m) denote time (or other complexity
  measure) of solving a given problem by a given
  algorithm working on input length m
• Simple recursive formula
• T(n) ? qT(n/2) f(n)
• Solve the problem for half of the input q times,
  and spend at most f(n) time for partitioning
  and/or merging

6
Example 1Searching in time O(log n)
7
Searching problem and algorithm

• Input sorted array A of n numbers and number x
• Problem find if x is in array A
• Solution Algorithm SEARCH(x,1,n)
• Procedure SEARCH(x,i,j)
• If i j then
• if x Ai then answer YES
• if x ? Ai then answer NO
• Else compare x with element A(ij)/2
• If x A(ij)/2 then answer YES
• If x lt A(ij)/2 then SEARCH(x, i, (ij)/2-1)
• If x gt A(ij)/2 then SEARCH(x, (ij)/21, j)

8
Tree structure of searching
root
4
find if value 5 is there
6
2
height 3
1
3
5
7
1
2
3
4
5
6
7
8
8 leaves
9
Time and memory consumption

• Each recursive call
• needs additional constant memory, and
• the size of input decreases by factor 2
• Formula for time ( comparisons)
• T(n) ? T(n/2) 4
• Additional memory size log n O(log n)
• In other words recursive algorithm must store a
  path from the root to the searched leaf, and it
  has logarithmic length

10
Case q 1, f(n) c

• T(n) ? T(n/2) c , T(2) ? c
• T(n) ? T(n/2) c
• ? (T((n/2)/2) c) c T(n/4) 2c
• ? (T((n/4)/2) c) 2c T(n/8) 3c
• ? (T(n/2i) c) (i-1) . c T(n/2i) i . c
• ? (T(n/2log n 1) c) (log n - 2) . c
• T(n/(n/2)) (log n - 2) . c T(2) (log n
  1) . c
• ? c (log n 1) . c c log n

11
Case q 1, f(n) cformal analysis

• T(n) ? T(n/2) c
• T(2) ? c
• Solution T(n) ? c log n
• Proof by induction.
• For n 2 straightforward T(2) ? c ? c log 2
• Suppose T(n/2) ? c log (n/2) then
• T(n) ? T(n/2) c ? c log (n/2) c
• ? c log n - c c ? c log n

12
Example 2Finding a majority in time O(n)
13
Finding a majority

• Input array A of n elements
• Problem find if there is an element x that
  occurs in array A more than n/2 times (a majority
  value)
• Solution Algorithm Majority(A) returning a
  majority element in A, if it exists, or null
  otherwise

14
Finding a majority algorithm

• Algorithm Majority(A1..n)
• If A 0 then output null, else if A 1 then
  output A1 else
• If n A is odd then
• check whether An is a majority in A by counting
  the number of occurrences of value An
• if yes then output An, otherwise decrease n by
  one
• Initialize additional array B of size A/2
• Set j to 0
• For i 1,2,,n/2 do
• if A2i-1 A2i then
• increase j by one
• set Bj to A2i
• Find if there is a majority in B1..j by
  executing Majority(B1..j)
• If a majority value x in B1..j is returned then
  check whether x is a majority in A, by going
  through array A and counting the number of
  occurrences of value x in A if successful output
  x otherwise null

15
Tree structure of finding majority
root
2
odd, no majority
height 2
2
2
1
2
2
2
2
1
2
1
1
8 leaves
16
Correctness of Majority Algorithm

• Lemma (to be proved later)
• If x is a majority in A then x is a majority in
  B.
• Correctness (proof by induction, based on the
  Lemma)
• Inductive assumption
• Majority(B) works correctly for all Bs of size
  smaller than A
• Case 1 There is a majority in A.
• Then it is a majority in B, by Lemma, therefore
  it will be found in the recursive stage of the
  algorithm Majority(B) (by inductive argument),
  and will be checked (successfully) in the last
  point of the algorithm Majority(A).
• Case 2 There is no majority in A.
• Then even if there is a candidate value found in
  the recursive stage Majority(B) of the algorithm,
  it will be rejected anyway in the last point of
  the algorithm Majority(A).

17
Proof of the Lemma

• Suppose, to the contrary, that x is a majority in
  A but is not a majority in B. Let
• m be the number of occurrences of x in A, and
• k be the number of occurrences of x in B.
• It follows that the other values appear at least
  k times in B.
• Consequently, the other values appear in A
• at least 2k times those pairs that are
  represented in B by a value different than x
  plus
• m-2k times each occurrence of x in A that is
  not paired by another x (there are k pairs xx in
  A, thus m-2k of other occurrences of x in A) is
  paired by some other value (different than x)
• which gives at least 2k(m-2k) m occurrences in
  total, and contradicts the fact that x is a
  majority in A. Contradiction! Therefore x must be
  also a majority in B.

18
Memory consumption

• Each call to the recursive procedure
• needs additional memory of half of the current
  input size
• reduces the size of the input by factor at least
  two
• Formula for time ( comparisons)
• T(n) ? n -1 3n/2 T(n/2) n T(n/2) 7n/2
• Additional memory used (for smaller arrays B)
• M(n) ? n/2 M(n/2) ? n/2 n/4 M(n/4)
• ? n/2 n/4 1 n - 1 O(n)
• Intuition about memory size the algorithm needs
  a binary tree of linear size, in the worst case,
  to store winners

19
Case q 1, f(n) cn

• T(n) ? T(n/2) cn , T(1) ? c
• T(n) ? T(n/2) cn
• ? (T(n/4) cn/2) cn T(n/4) (3/2)cn
• ? (T(n/8) cn/4) (3/2)cn T(n/8)
  (7/4)cn
• ? (T(n/2i) cn/2i-1) ((2i-1-1)/2i-2) . cn
• T(n/2i) ((2i-1)/2i-1) . cn
• ? T(n/2log n) ((2log n-1)/2log n-1) . cn
• T(1) (n 1)/(n/2) . cn ? c 2cn 2c
  ? 2cn

20
Case q 1, f(n) c nformal analysis

• T(n) ? T(n/2) c n
• T(1) ? c
• Solution T(n) ? 2c?n
• Proof by induction.
• For n 1 straightforward T(1) ? c ? 2c ? 1
• Suppose T(n/2) ? 2c?(n/2) then
• T(n) ? T(n/2) c?n ? 2c?(n/2) c n
• 2c?n

21
Textbook and exercises

• READING Chapter 5, up to Section 5.2
• EXERCISES
• Solve the following recursive formulas the best
  you can
• T(n) ? T(n/2) 4
• T(n) ? T(n/2) 5n
• T(n) ? T(n/2) 3n2
• T(n) ? (3/2)?T(n/2) 1
• Compare and sort the obtained (asymptotic)
  formulas according to the asymptotic order. For
  each of them try to find an algorithmic example
  with performance giving by this formula.
• Sort the following formulas according to
  big/small Oh order
• log (n1/2), log (9n), log (n3), 2log n, 23log n,
  2log (9n), n2, n log n
• Propose and analyse an algorithm for checking if
  there is a majority in a given sorted array A.

22
Example 3Sorting in time O(n log n)
23
Divide and conquer

• Algorithm
• Partition input into two halves in (at most)
  linear time
• Solve the problem for each half of the input
  separately
• Merge solutions into one in (at most) linear time
• Recursive formula on time complexity
• T(n) ? 2 T(n/2) c?n

24
Case q 2, f(n) cn

• T(n) ? 2T(n/2) cn , T(2) ? c
• T(n) ? 2T(n/2) cn
• ? 2(2T(n/4) cn/2) cn 4T(n/4) 2cn
• ? 4(2T(n/8) cn/4) 2cn 8T(n/8) 3cn
• ? 2i-1 (2T(n/2i) cn/2i-1) (i-1) . cn
• 2i T(n/2i) i . cn
• ? 2log n - 1 T(2) (log n - 1) . cn ? cn log
  n

25
Solution for timeformal analysis

• T(n) ? 2 T(n/2) c?n
• T(2) ? c
• Solution T(n) ? c?n?log n
• Proof by induction.
• For n 2 straightforward T(2) ? c ? c?2?log
  2
• Suppose T(n/2) ? c?(n/2)?log (n/2) then
• T(n) ? 2T(n/2) c?n ? 2c?(n/2)?log (n/2)
  c?n
• 2c?(n/2)?log n 2c?n/2 c?n c?n?log n

26
Example - sorting by merging

• Input list L of n numbers
• Problem Sort list L
• Algorithm MergeSort(L)
• If L has at most two elements then sort them by
  comparison and exit
• Else
• Split L into two lists prefix and suffix, each of
  size n/2
• Sort, by merging, the prefix and the suffix
  separately
• MergeSort(prefix) and MergeSort(suffix)
• Merge sorted prefix with sorted suffix as
  follows
• Initialize final list as empty
• Repeat until either prefix or suffix is empty
• Compare the first elements on the lists prefix
  and suffix and then move the smaller one to the
  end of the final list
• Concatenate final list with the remaining
  non-empty list (prefix or suffix)

27
MergeSort procedure
1
2
4
7
8
3
5
6
9
10
1
2
3
4
5
6
7
8
9
10
28
Tree structure of sorting
root
merging operations
height 3
2
4
1
7
8
3
6
5
8 leaves
29
Time and memory consumption

• Each recursive call
• needs additional constant memory, and
• the number of inputs increases by factor 2, and
• the size of each input decreases by factor 2
• Formula for time ( comparisons and list
  operations)
• T(n) ? n/2 2T(n/2) 2n 2T(n/2) 5n/2
• Additional memory size M(n) ? 2 M(n/2) 2
• M(n) ? 2 M(n/2) 2 ? 4 M(n/4) 4 2 ? 8 M(n/8)
  8 4 2
• ? ? n/2 n/4 4 2 n - 2 O(n)
• In other words divide and conquer algorithm must
  store the binary tree of pointers during the
  computation

30
Quick sort - algorithmic scheme

• Generic Quick Sort
• Select one element x from the input
• Partition the input into part containing elements
  not greater than x and part containing elements
  bigger than x
• Sort each part separately
• Concatenate these two sorted parts
• Problem how to choose element x to balance the
  sizes of these two parts? (to get the similar
  recursive equations as for MergeSort)

31
Why parts should be balanced?

• Suppose they are not balanced, but, say, the
  smallest element is chosen
• T(n) ? T(1) T(n-1) c n
• T(1) ? c
• Solution T(n) ? (c/2) . n2
• Proof by induction.
• For n 1 straightforward T(1) ? c ? (c/2) .
  12
• Suppose T(n-1) ? (c/2) . (n-1)2 then
• T(n) ? T(n-1) c cn ? (c/2) . (n-1)2 c
  (n1)
• ? (c/2) . (n-1)2 (c/2) . 2n ? (c/2) . n2

32
Two approaches to be fast

• Deterministic
• Additional tree structure of size O(n) needed for
  identifying the median value
• Time O(n log n), additional memory O(n)
• Randomized
• Select separation element x uniformly at random
• Time (expected) O(n log n), additional memory
  O(n)

33
Randomized approach - analysis

• Let T(n) denote the expected time sum of all
  possible values of time weighted by the
  probabilities of these values
• T(n) ? 1/n (T(n-1)T(1) T(n-2)T(2)
  T(0)T(n)) cn
• T(0) T(1) 1, T(2) ? c
• Solution T(n) ? d n log n, for some constant d
  ? 8c
• Proof by induction.
• For n 2 straightforward T(2) ? c ? d . 2
  . log 2
• Suppose T(m) ? d m log m, for every m lt n then
• (1-1/n)?T(n) ? (2/n)?(T(0) T(n-1)) c n
• ? (2d/n)?(1 1 2log 2
  (n-1)log(n-1)) c n
• ? (2d/n)?((n2/2) log n n2/8) c n
• ? d n log n - d (n/4) c n ? d n
  log n - d n/2
• T(n) ? (n/(n-1))?(d n log n - d n/2) ? d n log n

34
Tree structure of random execution
root
1
6
5
7
height 5
3
2
4
1
2
3
4
5
6
7
8
8 leaves
35
Textbook and Exercises

• READING Section 5.3
• EXERCISES
• Solved exercises 1, 2 from the textbook, chapter
  5 Divide and Conquer
• Prove that
• 1 1 2 log 2 (n-1)log(n-1) ? n(n/2)
  log(4n) , and
• (n/(n-1))?(d n log n - d n/2) ? d n log n , for n
  gt 16

36
Example 4Finding closest pair in time O(n log
n)
37
Problem

• Closest pair of points
• Input n points on the plane
• Output two points having the closest distance
• Remarks
• Exhaustive search algorithm gives time O(n2)
• Similar approach gives ALL pairs of closest points

38
Solution

• Preprocessing
• Sort points according to the first coordinate
  (obtain horizontal list H) and according to the
  second coordinate (obtain vertical list V)
• Main Algorithm
• Partition input list H into halves (H1,H2) in
  linear time (draw vertical line L containing
  medium point) split list V accordingly into
  V1,V2, where Vi contains the same elements as Hi
  and inherits the initial order from V (for i
  1,2)
• Solve the problem for each half of the input
  separately, obtaining two pairs of closest
  points let ? be the smallest distance from the
  obtained ones
• Find if there is a pair which has distance
  smaller than ? - if yes then find the smallest
  distance pair

39
Main difficulty

• Find if there is a pair which has distance
  smaller than ?
• if yes then find the smallest distance pair
• How to do it in linear time?

?
?
?
L
40
Closest pair in the strip

• Find a sub-list P of V containing all points with
  a distance at most ? from the line L
• go through V and remove elements not ?close to L
• 2. For each element in P check its distance to
  the next 8 elements and remember the closest pair
  ever found

?
?
?
P
L
41
Why to check only 8 points ahead?

• 1. Partition the strip into squares of length ?/2
  each, as shown in the picture
• 2. Each square contains at most 1 point - by
  definition of ?
• 3. If there is at least 2 full squares between
  points then they can not be the closest points
• 4. There are at most 8 squares to check

?
?/2
?/2
?/2
?/2
?/2
?/2
L
42
Time complexity

• Preprocessing sorting in time O(n log n)
• Main Algorithm recursion in time O(n log n)
• T(n) ? 3n/2 2T(n/2) 8n 2T(n/2) 9.5n
• T(2) 1

43
Example 5Integer multiplication in time
O(n1.59)
44
Beyond ?(n log n) integers multiplication

• Input two integers x and y, each consisting of
  at most n bits
• Output multiply these integers
• Naïve Algorithm
• Multiply each bit i of x by y, add (i-1) zeroes
  at the end
• Add the obtained at most n numbers
• Time ?(n2) of bit operations

45
Divide and Conquer approach

• Let x x1 2n/2 x2 and y y1 2n/2 y2
• Let p x1 x2 and q y1 y2. Then
• x y (x1 2n/2 x2) (y1 2n/2 y2)
• x1 y1 2n/2 (x2 y1 x1 y2) 2n x2 y2
• x1 y1 2n/2 (pq - x1 y1- x2 y2) 2n
  x2 y2
• Algorithm
• Compute p and q (in linear time)
• Compute recursively x1 y1 , x2 y2 , pq
• Perform x1 y1 2n/2 (pq - x1 y1- x2 y2) 2n x2
  y2 in linear time

46
Time complexity

• T(n) ? 3T(n/2) c n
• T(1) ? c
• Solution T(n) ? d nlog 3, for some constant d ?
  3c
• Proof For n 1 straightforward T(1) ? c ? d
  nlog 3
• General case
• T(n) ? 3T(n/2) c n ? 9T(n/4) 3c(n/2) c
  n
• ? ? c n (1 3/2 (3/2)2 (3/2)log n)
• c n 2(3/2)log n 1 ? 3c n? nlog 3 - 1
• 3c nlog 3 ? d? nlog 3 O(n1.59)

47
Textbook and Exercises

• READING Chapter 5 from Section 5.4
• OBLIGATORY EXERCISES
• How to add two n-bit integers in linear number of
  bit operations?
• ADDITIONAL EXERCISES
• Solved Exercise 1 from the textbook, chapter 5
  Divide and Conquer
• Exercises 1, 6, 7 from the textbook, chapter 5
  Divide and Conquer

48
Conclusions

• Searching in time O(log n)
• Finding majority value in time O(n)
• Divide and conquer in time O(n log n)
• sorting algorithms (MergeSort, RandQuickSort)
• closest points algorithm
• Beyond ?(n log n) O(nlog 3) time algorithm for
  integer multiplication