Introduction to Algorithms

1
Introduction to Algorithms

• Jiafen Liu

Sept. 2013
2
Todays task

• Develop more asymptotic notations
• How to solve recurrences

3
T-notation

• Math
• T(g(n)) f(n) there exist positive
  constants c1, c2, and n0 such that 0 c1g(n)
  f (n) c2g(n) for all n n0
• Engineering
• Drop low-order terms
• ignore leading constants.

4
O-notation

• Another asymptotic symbol used to indicate upper
  bounds.
• Math
• Ex 2n2 O(n3)
• (c 1, n0 2)
• means one-way equality

5
Set definition of O-notation

• Math
• Looks better?
• EX 2n2 ? O(n3)
• O-notation corresponds roughly to less than or
  equal to.

6
Usage of O-notation

• Macro substitution A set in a formula represents
  an anonymous function in the set, if O-notation
  only occurs on the right hand of formula.
• Ex
• f(n) n3 O(n2)
• Means f(n) n3 h(n)
• for some h(n) ?O(n2), here we can think of h(n)
  as error term.

7
Usage of O-notation

• If O-notation occurs on the left hand of formula,
  such as n2 O(n) O(n2), which means
• for any f(n) ?O(n)
• there exists some h(n) ?O(n2),
• makes n2 f(n) h(n)
• O-notation is an upper-bound notation.
• It makes no sense to say f(n) is at least O(n2).

8
O-notation

• How can we express a lower bound?
• Ex

9
T-notation

• O-notation is like
• O-notation is like .
• And T-notation is like
• Now we can give another definition of T-notation
• We also call T-notation as tight bound.

10
A Theorem on Asymptotic Notations

• Theorem 3.1
• For any two functions f(n) and g(n), we say
  f(n)T(g(n)) if and only if f(n)O(g(n)) and
  f(n)O(g(n)).
• Ex , how can we
  prove that?

11
Two More Strict Notations

• We have just said that
• O-notation and O-notation are like and .
• o-notation and ?-notation are like lt and gt.
• Difference with O-notation This inequality must
  hold for all c instead of just for 1.

12
What does it mean? for any constant c

• With o-notation, we mean that no matter what
  constant we put in front of g(x), f(x) will be
  still less than cg(x) for sufficiently large n.
  No matter how small c is.
• Ex 2n2 o(n3)
• An counter example
• 1/2n2 O (n2) does not hold for each c.

(n0 2/c)
13
Two More Strict Notations

• We have just said that
• O-notation and O-notation are like and .
• o-notation and ?-notation are like lt and gt.
• Difference with O-notation This inequality must
  hold for all c instead of just for 1.

14
Solving recurrences

• The analysis of merge sort from Lecture 1
  required us to solve a recurrence.
• Lecture 3 Applications of recurrences to
  divide-and-conquer algorithms.
• We often omit some details inessential while
  solving recurrences
• n is supposed to be an integer because the size
  of input is an integer typically .
• Ignore the boundary conditions for convenience.

15
Substitution method

• The most general method Substitution method
• Guess the form of the solution.
• Verify by induction.
• Solve for constants.
• Substitution method is used to determine the
  upper or lower bounds of recurrence.

16
Example of Substitution

• EXAMPLE T(n) 4T(n/2) n (n1)
• (Assume that T(1) T(1) )
• Can you guess the time complexity of it?
• Guess O(n3). (Prove O and O separately.)
• We will prove T(n) cn3 by induction.
• First ,we assume that T(k) ck3 for k lt n
• T(k) ck3 holds while kn/2 by assumption.

17
Example of Substitution

• All we need is
• this holds as long as c
  2
• for n 1

18
Base Case in Induction

• We must also handle the initial conditions, that
  is, ground the induction with base cases.
• Base T(n) T(1) for all n lt n0, where n0 is a
  suitable constant.
• For 1 nlt n0, we have T(1) cn3, if we pick c
  big enough.
• Here we are!

BUT this bound is not tight !
19
A tighter upper bound?

• We shall prove that T(n) O(n2) .
• Assume that T(k) ck2 for k lt n.
• Anybody can tell me
  why?
• Now we need
  n 0
• But it seems
  impossible
• for n 1

20
A tighter upper bound!

• IDEA Strengthen the inductive hypothesis.
  Subtract a low-order term.
• Inductive hypothesis T(k) c1k2 c2k for klt
  n.
• 
  Now we need
• 
  (c2-1) n 0,
• 
  it holds if c2 1

21
For the Base Case

• We have proved now that for any value of c1, and
  provided c2 1.
• Base We need T(1) c1 c2
• Assumed T(1) is some constant.
• We need to choose c1 to be sufficiently larger
  than c2, and c2 has to be at least 1.
• To handle the initial conditions, just pick c1
  big enough with respect to c2.

22
About Substitution method

• We have worked for upper bounds, and the lower
  bounds are similar.
• Try it yourself.
• Shortcomings
• We had to know the answer in order to find it,
  which is a bit of a pain.
• It would be nicer to just figure out the answer
  by some procedure, and that will be the next two
  techniques.

23
Recursion-tree method

• A recursion tree models the costs (time) of a
  recursive execution of an algorithm.
• The recursion-tree method can be unreliable, just
  like any method that uses dot,dot,dots ().
• The recursion-tree method promotes intuition,
  however.
• The recursion tree method is good for generating
  guesses for the substitution method.

24
Example of recursion tree

• Solve T(n) T(n/4) T(n/2) n2

25
Example of recursion tree

• Solve T(n) T(n/4) T(n/2) n2

26
Example of recursion tree

• Solve T(n) T(n/4) T(n/2) n2

27
Example of recursion tree

• Solve T(n) T(n/4) T(n/2) n2

?
How many leaves?
ltn
We just need an upper bound.
28
Example of recursion tree

• Solve T(n) T(n/4) T(n/2) n2
• lt2n2
• So T(n) T(n2)

?
?
?
?
Recall 11/21/41/8
29
The Master Method

• It looks like an application of the recursion
  tree method but with more precise.
• The sad part about the master method is it is
  pretty restrictive.
• It only applies to recurrences of the form
• T(n) a T(n/b) f(n) ,
• where a 1, b gt1, and f(n) is asymptotically
  positive.
• aT(n/b) means every problem you recurse on
  should be of the same size.

30
Three Common Cases
31
Three Common Cases

• Case 1

32
Three Common Cases

• Case 1
• Case 2

33
Three Common Cases

• Case 3

34
Examples

• Ex T(n) 4T(n/2) n
• a 4, b 2
• and

35
Examples

• Ex T(n) 4T(n/2) n2
• a 4, b 2
• and

36
Examples

• Ex T(n) 4T(n/2) n3
• a 4, b 2
• and

37
Homework

• Read Chapter 3 and 4 to be prepared for
  applications of recurrences.

38
Proof of Master Method
Height ?
logbn
(leaves) ?
39
Proof of Master Method

• CASE1 The weight increases geometrically from
  the root to the leaves. The leaves hold a
  constant fraction of the total weight.

Height ?
logbn
(leaves) ?
40
Proof of Master Method

• CASE2(k 0) The weight is approximately the same
  on each of the logbn levels.

Height ?
logbn
(leaves) ?
41
Proof of Master Method

• CASE3 The weight decreases geometrically from
  the root to the leaves. The root holds a constant
  fraction of the total weight.

Height ?
logbn
(leaves) ?
42
Have FUN !