Asymptotic Behavior

1
Asymptotic Behavior

• Algorithm Design Analysis
• 2

2
In the last class

• Goal of the Course
• Mathematical Background
• Probability
• Summations and Series
• Monotonic and Convex Functions
• Average and Worst-Case Analysis
• Lower Bounds and the Complexity of Problems

3
Asymptotic Behavior

• Asymptotic growth rate
• The Sets ?, ? and ?
• Complexity Class
• An Example Searching an Ordered Array
• Improved Sequential Search
• Binary Search
• Binary Search Is Optimal

4
Fundamentals of Cost

• Input
• Input size
• Number of items (sorting)
• Total number of bits (integer multiplying)
• Number of verticesnumber of edges (graph)
• Structure of input
• Running time
• Number of primitive operations executed
• Actual cost of each operation

5
How to Compare Two Algorithm?

• Simplifying the analysis
• assumption that the total number of steps is
  roughly proportional to the number of basic
  operations counted (a constant coefficient)
• only the leading term in the formula is
  considered
• constant coefficient is ignored
• Asymptotic growth rate
• large n vs. smaller n

6
Relative Growth Rate

O(g)functions that grow at least as fast as g
T(g)functions that grow at the same rate as g
g
?(g)functions that grow no faster as g
7
The Set Big Oh

• Definition
• Giving gN?R, then ?(g) is the set of fN?R,
  such that for some c?R and some n0?N, f(n)?cg(n)
  for all n?n0.
• A function f??(g) if limn??f(n)/g(n)clt?
• Note c may be zero. In that case, f??(g),
  little Oh
• Example let f(n)n2, g(n)nlgn, then
• f??(g), since limn??f(n)/g(n) limn??n2/nlgn
  limn??n/lgn limn??1/(1/nln2)?
• g??(f), since limn??g(n)/f(n)0

8
The Sets ? and ?

• Definition
• Giving gN?R, then ?(g) is the set of fN?R,
  such that for some c?R and some n0?N, f(n)?cg(n)
  for all n?n0.
• A function f??(g) if limn??f(n)/g(n)gt0
• Note the limit may be infinity
• Definition
• Giving gN?R, then ?(g) ?(g)?? ?(g)
• A function f??(g) if limn??f(n)/g(n)c,0ltclt?

9
How Functions Grow
Algorithm 1 2 2 2 3 3 4 4 4
Time function(ms) 33n 46n lg n 46n lg n 46n lg n 13n2 13n2 3.4n3 3.4n3 3.4n3 2n
Input size(n) Solution time Solution time Solution time Solution time Solution time Solution time Solution time Solution time Solution time Solution time
10 0.00033 sec. 0.00033 sec. 0.00033 sec. 0.0015 sec. 0.0015 sec. 0.0013 sec. 0.0013 sec. 0.0034 sec. 0.0034 sec. 0.001 sec.
100 0.0033 sec. 0.0033 sec. 0.0033 sec. 0.03 sec. 0.03 sec. 0.13 sec. 0.13 sec. 3.4 sec. 3.4 sec. 4?1016 yr.
1,000 0.033 sec. 0.033 sec. 0.033 sec. 0.45 sec. 0.45 sec. 13 sec. 13 sec. 0.94 hr. 0.94 hr.
10,000 0.33 sec. 0.33 sec. 0.33 sec. 6.1 sec. 6.1 sec. 22 min. 22 min. 39 days 39 days
100,000 3.3 sec. 3.3 sec. 3.3 sec. 1.3 min. 1.3 min. 1.5 days 1.5 days 108 yr. 108 yr.
Time allowed Maximum solvable input size (approx.) Maximum solvable input size (approx.) Maximum solvable input size (approx.) Maximum solvable input size (approx.) Maximum solvable input size (approx.) Maximum solvable input size (approx.) Maximum solvable input size (approx.) Maximum solvable input size (approx.) Maximum solvable input size (approx.) Maximum solvable input size (approx.)
1 second 30,000 30,000 2,000 2,000 280 280 67 67 20 20
1 minute 1,800,000 1,800,000 82,000 82,000 2,200 2,200 260 260 26 26
10
Increasing Computer Speed
Number of steps performed on input of size n f(n) Maximum feasible input size s Maximum feasible input size in t times as much time snew
lg n s1 s1t
n s2 t s2
n2 s3 ?t s3
n3 s4 s4lg t
11
Sorting a Array of 1 Million Numbers

Using merge sort, taking time 50nlogn 100
seconds!
Computer B 10 Mips
Computer A 1000 Mips
Using insertion sort, taking time 2n2 2000
seconds!
12
Properties of O, ? and ?

• Transitive property
• If f?O(g) and g?O(h), then f?O(h)
• Symmetric properties
• f?O(g) if and only if g??(f)
• f??(g) if and only if g??(f)
• Order of sum function
• O(fg)O(max(f,g))

13
Complexity Class

• Let S be a set of fN?R under consideration,
  define the relation on S as following fg iff.
  f??(g)
• then, is an equivalence.
• Each set ?(g) is an equivalence class, called
  complexity class.
• We usually use the simplest element as possible
  as the representative, so, ?(n), ?(n2), etc.

14
Comparison of Often Used Orders

• The log function grows more slowly than any
  positive power of n
• lgn ? o(n?) for any ?gt0
• The power of n grows more slowly than any
  exponential function with base greater than 1
• nk ? o(cn) for any cgt1
• (The commonly seen base is 2)

15
Order of Common Sums

• rk is the largest
  term in the sum

f(n)
Areanf(n)
f(rn)
n
rn (0ltrlt1)
Area(1-r)nf(rn)
16
Manipulation of Big-O
17
Roulette Wheel
If there are W winners with bonus a, and L losers
with penalty b, then the average winning will be
There are N slots , some winning and some losing
1
2
N
3
.
.
i
.
Winner
n
18
A Special Case with N1000
Let winning bonus be 5, and losing penalty be 1
19
Solution Generalized
Let the largest slot number be N, and
, then
20
Reasonable Approximation
Now, we have W?N/K?K2/25K/2-3, where K
21
If Only N Large Enough
N W
error
1,000 10,000 100,000 1,000,000 10,000,000 100,000,
000 1,000,000,000
150.0 696.2 3231.7 15000.0 69623.8 323165.2 150000
0.0
172 746 3343 15247 70158 324322 1502497
12.791 6.670 3.331 1.620 0.761 0.357 0.166
22
Searching an Ordered Array

• Problem
• Input
• an array E containing n entries of numeric type
  sorted in non-decreasing order
• a value K
• Output
• index for which KEindex, if K is in E, or, -1,
  if K is not in E
• Algorithm
• Int seqSearch(int E, int n, int K)
• 1. Int ans, index
• 2. Ans-1 // Assume failure
• 3. For (index0 indexltn index)
• 4. If (KEindex) ansindex//success!
• 5. break
• 6. return ans

23
Searching a Sequence (cont.)

• For a given K, there are two possibilities
• K in E (say, with probability q), then K may be
  any one of Ei (say, with equal probability,
  that is 1/n)
• K not in E (with probability 1-q), then K may be
  located in any one of gap(i) (say, with equal
  probability, that is 1/(n1))

24
Improved Sequential Search

• Since E is sorted, when an entry larger than K is
  met, no more comparison is needed
• Worst-case complexity n, unchanged
• Average complexity
• 
  Note A(n)??(n)

Roughly n/2
25
Divide and Conquer

• If we compare K to every jth entry, we can locate
  the small section of E that may contain K.
• To locate a section, roughly n/j steps at most
• To search in a section, j steps at most
• So, the worst-case complexity (n/j)j, with j
  selected properly, (n/j)j??(?n)
• However, we can use the same strategy in the
  small sections recursively

Choose j ?n
26
Binary Search

• int binarySearch(int E, int first, int last,
  int K)
• if (lastltfirst)
• index-1
• else
• int mid(firstlast)/2
• if (KEmid)
• indexmid
• else if (KltEmid)
• indexbinarySearch(E, first, mid-1, K)
• else if (KltEmid)
• indexbinarySearch(E, mid1, last, K)
• return index

27
Worst-case Complexity of Binary Search

• Observation with each call of the recursive
  procedure, only at most half entries left for
  further consideration.
• At most ?lg n? calls can be made if we want to
  keep the range of the section left not less than
  1.
• So, the worst-case complexity of binary search is
  ?lg n?1?lg(n1)?

28
Average Complexity of Binary Search

• Observation
• for most cases, the number of comparison is or is
  very close to that of worst-case
• particularly, if n2k-1, all failure position
  need exactly k comparisons
• Assumption
• all success position are equally likely (1/n)
• n2k-1

29
Average Complexity of Binary Search

• Average complexity
• Note We count the sum of st, which is the number
  of inputs for which the algorithm does t
  comparisons, and if n2k-1, st2t-1

30
Decision Tree

• An algorithms A that can do no other operations
  on the array entries except comparison can be
  modeled by a decision trees.
• A decision tree for A and a given input of size n
  is a binary tree whose nodes are labeled with
  numbers between 0 and n-1
• Root labeled with the index
• first compared
• If the label on a particular node
• is i, then the left child is labeled
• the index next compared if KltEi,
• the right child the index next
• compared if KgtEi, and no branch
• for the case of KEi.

31
Binary Search Is Optimal

• If the number of comparison in the worst case is
  p, then the longest path from root to a leaf is
  p-1, so there are at most 2p-1 node in the tree.
• There are at least n node in the tree.
• (We can prove that For all i?0,1,,n-1, exist a
  node with the label i.)
• Conclusion n? 2p-1, that is p?lg(n1)

32
Binary Search Is Optimal

• For all i?0,1,,n-1, exist a node with the
  label i.
• Proof
• if otherwise, suppose that i doesnt appear in
  the tree, make up two inputs E1 and E2, with
  E1iK, E2iK, KgtK, for any j?i, (0?j?n-1),
  E1jE2j. (Remember that both array are
  sorted). Since i doesnt appear in the tree, for
  both K and K, the algorithm behave alike
  exactly, and give the same outputs, of which at
  least one is wrong, so A is not a right
  algorithm.

33
Home Assignment

• pp.63
• 1.23
• 1.27
• 1.31
• 1.34
• 1.45