Asymptotic Notation

1
Asymptotic Notation

• CS 583
• Analysis of Algorithms

2
Outline

• Divide-and-Conquer Approach
• Merge Sort Algorithm
• Pseudocode
• Asymptotic Notation
• ?-notation
• O-notation
• ?-notation, etc.
• Randomized Algorithms

3
Divide-and-Conquer Approach

• Break the problem into several subproblems that
  are similar to the original problem, but smaller
  in size solve the subproblems recursively then
  combine the solutions.
• Divide the problem into a number of subproblems.
• Conquer the subproblems by solving them
  recursively.
• Combine he solutions into the solution of the
  original problem.

4
Merge Sort Algorithm

• Divide an n-element sequence to be sorted into
  two n/2-subsequences.
• Sort the two subsequences recursively using merge
  sort.
• Merge two subsequences to get the sorted answer.
• The key procedure is to merge Ap..q with
  Aq1..r assuming both sequences are sorted.
• Two arrays are merged by moving the smaller of
  two numbers to the resulting array at each step.
  There are at most n steps performed, so merging
  takes ?(n) time.
• A special sentinel number (max_int) is placed at
  the bottom to simplify the comparison.

5
Merge Sort Pseudocode
MERGE (A, p, q, r) 1 n1 q-p1 2 n2 r-q 3
// create left and right arrays 4 for i1 to
n1 5 Li Api-1 6 for i1 to n2 7
Ri Aqi 8 Ln11 max_int 9 Rn21
max_int 10 i1 11 j1 12 for kp to r 13 if
Li lt Rj 14 AkLi 15 i i1 16
else 17 AkRj 18 j j1
6
Merge Sort Pseudocode (cont.)
The main merge sort procedure sorts elements in
the subarray Ap..r   MERGE-SORT-RUN (A, p,
r) 1 if pltr 2 q ceil((pr)/2) 3
MERGE-SORT-RUN(A,p,q) 4 MERGE-SORT-RUN(A,q1,r)
5 MERGE(A,p,q,r)   The merge sort algorithm
simply runs the main procedure on an array
A1..n   MERGE-SORT (A, n) 1
MERGE-SORT-RUN(A,1,n)
7
Merge Sort Analysis

• Assume the original problem's size n2x.
• The divide step just computes the middle of an
  array
• This takes constant time c.
• We solve two subproblems, each of size n/2.
• Combining is a merge procedure that takes ?(n)
  time.

  c, if n 1 T(n) 2T(n/2) cn,
otherwise There are lg(n) recursive steps, each
takes cn time   T(n) c?n?lg(n)
8
Asymptotic Notation

• The notations are defined in terms of functions
  whose domains are the set of natural numbers
  N0,1,2,....
• Such notations are convenient for describing the
  worst-case running time function T(n).
• It can also be extended to the domain of real
  numbers.

9
?-notation
For a given function g(n) we denote by ?(g(n))
the set of functions   ?(g(n)) f(n) ? c1,
c2, n0 gt 0 (? ngt n0
0 lt c1 g(n) lt f(n) lt c2 g(n)))   f(n)
?(g(n)) ? f(n) ? ?(g(n))   g(n) is an
asymptotically tight bound for f(n)
10
?-notation Examples

• (n/100100) ?(n)
•  
• Find c1, c2, n0 such that
•  
• c1n lt n/100100ltc2n for all ngtn0
• c1 lt 1/100 100/n lt c2
•  
• For nn01 we have
•  
• c1 lt 100 1/100
• c2 gt 100 1/100
•  
• Choose c1 1/100 c2 100.001, then the above
  equation will hold for any ngt1.

11
?-notation Examples (cont.)

• f(n)1000 ? ?(n)
•  
• By contradiction, suppose there is c1 so that
•  
• c1?n lt 1000 for all ngtn0
•  
• n lt 1000/c1, which cannot hold for arbitrarily
  large n since c1 is constant.

12
O-notation
We use O-notation when we have only an asymptotic
upper bound   O(g(n)) f(n) ? c,n0 gt 0
(? ngt n0 (0 lt f(n) lt cg(n)))   Note
that, ?(g(n)) ? O(g(n)). For example, n
O(n2).   Since O-notation describes an upper
bound, when we use it to bound the worst-case
running time of an algorithm, we have a bound on
every input. For example, the O(n2) bound on the
insertion sort also applies to its running time
on every input However, ? (n2) on the insertion
sort would only apply to the worst-case input.
13
?-notation
?-notation provides an asymptotic lower bound n a
function   ?(g(n)) f(n) ? c,n0 gt 0
(? ngt n0 (0 lt cg(n) lt f(n)))   Since
?-notation describes a lower bound, it is useful
when applied to the best-case running time of
algorithms. For example, the best-case running
time of the insertion sort is ?(n), which implies
that the running time of insertion sort ?(n).
14
o-notation
This notation is used to denote an upper bound
that is not asymptotically tight   o(g(n))f(n)
?c gt 0 (? n0gt0 (0 lt
f(n) lt cg(n) for all ngtn0))   For example, 2n
o(n2). Intuitively   lim f(n)/g(n) 0  
15
?-notation
We use ?-notation to denote a lower bound that is
not asymptotically tight   ?(g(n))f(n) ?c gt
0 (? n0gt0 (0 lt cg(n) lt
f(n) for all ngtn0))   For example, n2/2 ? (n).
The relationship implies   lim f(n)/g(n) ?
16
The Hiring Problem

• The goal is to hire a new assistant through an
  employment agency.
• The agency sends one candidate each day.
• The commitment is to have the best person to do
  the job.
• When the interviewed person is better than the
  current assistant, he/she is hired in place of
  the current one.
• There is a small cost to pay for the interview.
• There is usually a larger cost associated with
  the fire/hire process.

17
The Hiring Problem Algorithm

• Hire-Assistant (n)
• 1 best 0 // candidate 0 is least qualified
• 2 for i 1 to n
• ltinterview candidate igt
• 4 if i is better than best
• 5 best i
• 6 lthire candidate igt
• Assume interviewing has cost ci, whereas more
  expensive hiring has cost ch. Let m be the number
  of people hired. Then the cost of the above
  algorithm is
• O(nci mch)
• The quantity m varies with each run and
  determines the overall cost of the algorithm. It
  is estimated using probabilistic analysis.

18
Indicator Random Variables
Assume sample space S and an event A. The
indicator random variable IA is defined as
1, if A occurs IA 0,
otherwise Given a sample space S and an event A,
denote XA a random variable associated with an
event being A, i.e. XA IA. The the expected
value of XA is EXA PrA Proof. EXA
EIA 1?PrA 0?Pr?A PrA
19
The Hiring Problem Analysis
Let Xi be the indicator random variable
associated with the event that the candidate i is
hired Xi Icandidate i is hired Let X be
the random variable whose value equals the number
of time we hire a new candidate X X1 ...
Xn Note that EXi PrXi Prcandidate i is
hired. We now need to compute Prcandidate i is
hired.
20
The Hiring Problem Analysis (cont.)
Candidate i is hired (line 5) when it is better
than any of the previous (i-1) candidates. Since
all candidates arrive in random order, each of
them have the same probability of being the best
so far. Therefore EXi PrXi 1/i We can
now compute EX EX EX1 ... Xn 1
½ ... 1/n ln(n) O(1) Hence, when
candidates are presented in random order, the
algorithm Hire-Assistant has a total hiring
cost O(ch ln(n))
21
Randomized Algorithms
In a randomized algorithm the distribution of
inputs is imposed. In particular, in the
randomized version of the Hire-Assistant
algorithm we randomly permute the
candidates Randomized-Hire-Assistant (n) 1
ltrandomly permute the list of candidatesgt 2 best
0 // candidate 0 is least qualified 3 for i 1
to n 4 ltinterview candidate igt 5 if i is
better than best 6 best i 7 lthire
candidate igt According to the earlier
computations, the expected cost of the above
algorithm is O(ncichln(n)).