BigO Analyzing Algorithms Asymptotically

1
Big-OAnalyzing Algorithms Asymptotically

• CS 226
• 5 February 2002
• Noah Smith (nasmith_at_cs)

2
Comparing Algorithms
P1
P2
Should we use Program 1 or Program 2? Is Program
1 fast? Fast enough?
3
You and Igor the empirical approach
Implement each candidate
Run it
Time it
That could be lots of work also error-prone.
Which inputs?
What machine/OS?
4
Toward an analytic approach
Today is just math How to solve which
algorithm problems without machines, test data,
or Igor!
5
The Big Picture
Algorithm
How long does it take for the algorithm to finish?
6
Primitives

• Primitive operations
• x 4 assignment
• ... x 5 ... arithmetic
• if (x lt y) ... comparison
• x4 index an array
• x dereference (C)
• x.foo( ) calling a method
• Others
• new/malloc memory usage

7
How many foos?

• for (j 1 j lt N j)
• foo( )

1 N
8
How many foos?

• for (j 1 j lt N j)
• for (k 1 k lt M k)
• foo( )

1 NM
9
How many foos?

• for (j 1 j lt N j)
• for (k 1 k lt j k)
• foo( )

N (N 1)
1
j
2
10
How many foos?

• for (j 0 j lt N j)
• for (k 0 k lt j k)
• foo( )
• for (j 0 j lt N j)
• for (k 0 k lt M k)
• foo( )

N(N 1)/2
NM
11
How many foos?
T(0) T(1) T(2) 1 T(n) 1 T(n/2) if n gt
2 T(n) 1 (1 T(n/4)) 2
T(n/4) 2 (1 T(n/8)) 3
T(n/8) 3 (1 T(n/16)) 4
T(n/16) log2 n

• void foo(int N)
• if(N lt 2) return
• foo(N / 2)

12
The trick

• ank bnk-1 yn z
• ank
• nk

13
Big O

• Definition Let f and g be functions mapping N
  to R. We say that f(n) is O(g(n)) if there exist
• c ? R, c gt 0
• and
• n0 ? N, n0 1
• such that
• f(n) cg(n) for all n ? N, n n0

14
Example 1
g(n)
f(n)
n0
15
Example 2
h(n)
g(n)
n0
16
Example 3
h(n)
k(n)
17
Example 3
k(n)
h(n)
18
Example 3
k(n)
h(n)
19
Example 4
3n2
n2
20
Some complexity classes
21
Dont be confused

• We typically say,
• f(n) is O(g(n)) or f(n) O(g(n))
• But O(g(n)) is really a set of functions.
• It might be more clear to say,
• f(n) ? O(g(n))
• But I dont make the rules.
• Crystal clear f(n) is order (g(n)

22
Intuitively

• To say f(n) is O(g(n)) is to say that
• f(n) is less than or equal to g(n)
• We also have (GT pp. 118-120)

23
Big-Omega and Big-Theta

• O is just like O except that f(n) cg(n)
• f(n) is O(g(n)) ?? g(n) is O(f(n))
• T is both O and O (and the constants need not
  match)
• f(n) is O(g(n)) ??? f(n) is O(g(n)) ? f(n) is
  T(g(n))

24
little o

• Definition Let f and g be functions mapping N
  to R. We say that f(n) is o(g(n)) if
• for any c ? R, c gt 0
• there exists
• n0 ? N, n0 gt 0
• such that
• f(n) cg(n) for all n ? N, n n0
• (little omega, ?, is the same but with )

25
Multiple variables

• for(j 1 j lt N j)
• for(k 1 k lt N k)
• for(l 1 l lt M l)
• foo()
• for(j 1 j lt N j)
• for(k 1 k lt M k)
• for(l 1 l lt M l)
• foo()

O(N2M NM2)
26
Multiple primitives

• for(j 1 j lt N j)
• sum Aj
• for(k 1 k lt M k)
• sum2 Bjk
• Cjk Bjk Aj 1
• for(l 1 l lt k l)
• Bjk - Bjl

27
Tradeoffs an example
0, 0 0, 1 0, 2 0, 3 1, 0 1, 1 1, 2 1, 3 N,
0 N, 1 N, 2 N, N
0 0
0 1
0 2
(lots of options in between)

N N
?
28
Another example
0 1 2 3 1,000,000

• I have a set of integers between 0 and 1,000,000.
• I need to store them, and I want O(1) lookup,
  insertion, and deletion.
• Constant time and constant space, right?

29
Big-O and Deceit

• Beware huge coefficients
• Beware key lower order terms
• Beware when n is small

30
Does it matter? Let n 1,000, and 1 ms /
operation.
31
Worst, best, and average

• Gideon is a fast runner
• up hills.
• down hills.
• on flat ground.

• Gideon is the fastest swimmer
• on the JHU team.
• in molasses.
• in our research lab.
• in 5-yard race.
• on Tuesdays.
• in an average race.

32
Whats average?

• Strictly speaking, average (mean) is relative to
  some probability distribution.
• mean(X) Pr(x) ? x
• Unless you have some notion of a probability
  distribution over test cases, its hard to talk
  about average requirements.

33
Now you know

• How to analyze the run-time (or space
  requirements) of a piece of pseudo-code.
• Some new uses for Greek letters.
• Why the order of an algorithm matters.
• How to avoid some pitfalls.