Analysis of Algorithms

1
Analysis of Algorithms

• Issues
• Correctness
• Time efficiency
• Space efficiency
• Optimality
• Approaches
• Theoretical analysis
• Empirical analysis

2
Theoretical analysis of time efficiency

• Time efficiency is analyzed by determining the
  number of repetitions of the basic operation as a
  function of input size
• Basic operation the operation that contributes
  most towards the running time of the algorithm.
• T(n) copC(n)

3
Input size and basic operation examples
Problem Input size measure Basic operation
Search for key in list of n items Number of items in list n Key comparison
Multiply two matrices of floating point numbers Dimensions of matrices Floating point multiplication
Compute an n Floating point multiplication
Graph problem vertices and/or edges Visiting a vertex or traversing an edge
4
Empirical analysis of time efficiency

• Select a specific (typical) sample of inputs
• Use physical unit of time (e.g., milliseconds)
• OR
• Count actual number of basic operations
• Analyze the empirical data

5
Best-case, average-case, worst-case

• For some algorithms efficiency depends on type of
  input
• Worst case W(n) maximum over inputs of size
  n
• Best case B(n) minimum over inputs of
  size n
• Average case A(n) average over inputs of
  size n
• Number of times the basic operation will be
  executed on typical input
• NOT the average of worst and best case
• Expected number of basic operations repetitions
  considered as a random variable under some
  assumption about the probability distribution of
  all possible inputs of size n

6
Example Sequential search

• Problem Given a list of n elements and a search
  key K, find an element equal to K, if any.
• Algorithm Scan the list and compare its
  successive elements with K until either a
  matching element is found (successful search) or
  the list is exhausted (unsuccessful search)
• Worst case
• Best case
• Average case

7
Types of formulas for basic operation count

• Exact formula
• e.g., C(n) n(n-1)/2
• Formula indicating order of growth with specific
  multiplicative constant
• e.g., C(n) 0.5 n2
• Formula indicating order of growth with unknown
  multiplicative constant
• e.g., C(n) cn2

8
Order of growth

• Most important Order of growth within a constant
  multiple as n?8
• Example
• How much faster will algorithm run on computer
  that is twice as fast?
• How much longer does it take to solve problem of
  double input size?
• See table 2.1

9
Table 2.1
10
Asymptotic growth rate

• A way of comparing functions that ignores
  constant factors and small input sizes
• O(g(n)) class of functions f(n) that grow no
  faster than g(n)
• T(g(n)) class of functions f(n) that grow at
  same rate as g(n)
• O(g(n)) class of functions f(n) that grow at
  least as fast as g(n)
• see figures 2.1, 2.2, 2.3

11
Big-oh
12
Big-omega
13
Big-theta
14
Establishing rate of growth Method 1 using
limits

• limn?8 T(n)/g(n)

• Examples
• 10n vs. 2n2
• n(n1)/2 vs. n2
• logb n vs. logc n

15
LHôpitals rule

• If
• limn?8 f(n) limn?8 g(n) 8
• The derivatives f, g exist,
• Then

• Example logn vs. n

16
Establishing rate of growth Method 2 using
definition

• f(n) is O(g(n)) if order of growth of f(n)
  order of growth of g(n) (within constant
  multiple)
• There exist positive constant c and non-negative
  integer n0 such that
• f(n) c g(n) for every n n0
• Examples
• 10n is O(2n2)
• 5n20 is O(10n)

17
Basic Asymptotic Efficiency classes
1 constant
log n logarithmic
n linear
n log n n log n
n2 quadratic
n3 cubic
2n exponential
n! factorial
18
Time efficiency of nonrecursive algorithms

• Steps in mathematical analysis of nonrecursive
  algorithms
• Decide on parameter n indicating input size
• Identify algorithms basic operation
• Determine worst, average, and best case for input
  of size n
• Set up summation for C(n) reflecting algorithms
  loop structure
• Simplify summation using standard formulas (see
  Appendix A)

19
A simple example

• Algorithm MaxElement(A0..n-1)
• // Determines the value of the largest element in
  a given array
• // Input An array A0..n-1 of real numbers
• // Output The value of the largest element in A
• maxval ? A0
• for i?1 to n-1 do
• if Aigtmaxval
• maxval ? Ai
• return maxval

20
Another simple example

• Algorithm UniqueElement(A0..n-1)
• // Check whether all the elements in a array are
  distinct
• // Input An array A0..n-1 of real numbers
• // Output Returns true if all the elements are
  distinct
• for i?0 to n-2 do
• for j?i1 to n-1 do
• if AiAj return false
• return true

21
Other examples

• Matrix multiplication
• Selection sort
• Insertion sort
• Mystery Algorithm

22
Matrix multipliacation
23
A third simple example
Algorithm Binary(n) // Input A positive decimal
integer n // Output The number of binary digits
in ns binary representation count?1 while ngt1
do count ? count1 n ? n/2 return count
24
Selection sort
25
Insertion sort
26
Mystery algorithm

• for i 1 to n-1 do
• max i
• for j i1 to n do
• if Aj,i gt Amax,i then max j
• for k i to n1 do
• swap Ai,k with Amax,k
• for j i1 to n do
• for k n1 downto i do
• Aj,k Aj,k - Ai,kAj,i/Ai,i

27
Time efficiency of recursive algorithms

• Steps in mathematical analysis of recursive
  algorithms
• Decide on parameter n indicating input size
• Identify algorithms basic operation
• Determine worst, average, and best case for input
  of size n
• Set up a recurrence relation and initial
  condition(s) for C(n)-the number of times the
  basic operation will be executed for an input of
  size n (alternatively count recursive calls).
• Solve the recurrence to obtain a closed form or
  estimate the order of magnitude of the solution
  (see Appendix B)

28
Recursive evaluation of n !

• Definition n ! 12(n-1)n
• Recursive definition of n! n ! (n-1)!n
• Algorithm
• if n0 then F(n) 1
• else F(n) F(n-1) n
• return F(n)
• Recurrence for number of multiplications
• M(n) M(n-1) 1 for ngt0
• M(0) 0
• Method backward substitution

29
Towers of Hanoi

• Problem to count the number of moves
• Recurrence for number of moves
• M(n) M(n-1) 1 M(n-1) for ngt1
• M(1) 1
• Alternatively construct a tree for the number of
  recursive calls
• Comment inherent inefficiency

30
Revisiting the Binary algorithm

• Algorithm BinRec(n)
• // Input A positive decimal integer n
• // Output The number of binary digits in ns
  binary representation
• if n1 return 1
• else return BinRec(n/2)1
• Recurrence for number of additions
• A(n) A(n/2) 1 for ngt1
• A(1) 0
• Apply the smoothness rule n2k

31
Computing Fibonacci numbers (1)

1. Definition based recursive algorithm
2. Nonrecursive brute-force algorithm
3. Explicit formula algorithm
4. Algorithm based on matrix multiplications

32
Computing Fibonacci numbers (2)

• Leonardo Fibonacci, 2002.
• The Fibonacci sequence 0, 1, 1, 2, 3, 5, 8, 13,
  21,
• Fibonacci recurrence
• F(n) F(n-1) F(n-2)
• F(0) 0
• F(1) 1
• Algorithm F(n)
• // Computes the n-th Fibonacci number recursively
• // Input A nonnegative integer n
• // Output The n-th Fibonacci number
• if n lt 1 return n
• else return F(n-1)F(n-2)

33
Computing Fibonacci numbers (3)

• 2nd order linear homogeneous recurrence relation
  with constant coefficients
• Characteristic equation
• Solution
• F(n) fn/v5 fn/v5
• f (1v5)/2 (golden ratio) f (1-v5)/2
• F(n) fn/v5
• Construct a tree for the number of recursive
  calls
• Recursive algorithms solutions, not a panacea

34
Computing Fibonacci numbers (4)

• Algorithm F(n)
• // Computes the n-th Fibonacci number iteratively
• // Input A nonnegative integer n
• // Output The n-th Fibonacci number
• F0 ? 0 F1? 1
• for i ? 2 to n do
• Fi ? Fi-1 Fi-2
• return F(n)
• Complexity ?

35
Computing Fibonacci numbers (5)

• Alternatively algorithmic use of equation F(n)
  fn/v5
• with smart rounding
• Relies on the exponentiation algorithm
• Solutions T(n), T(logn)

36
Computing Fibonacci numbers (6)

• It holds that
• for n1,
• Assuming an efficient way of computing matrix
  powers, solution in T(logn)

37
Important recurrence types

• One (constant) operation reduces problem size by
  one.
• T(n) T(n-1) c T(1) d
• Solution T(n) (n-1)c d
  linear
• A pass through input reduces problem size by one.
• T(n) T(n-1) cn T(1) d
• Solution T(n) n(n1)/2 1 c d
  quadratic
• One (constant) operation reduces problem size by
  half.
• T(n) T(n/2) c T(1) d
• Solution T(n) c lg n d
  logarithmic
• A pass through input reduces problem size by
  half.
• T(n) 2T(n/2) cn T(1) d
• Solution T(n) cn lg n d n
  n log n

38
A general divide-and-conquer recurrence

• T(n) aT(n/b) f (n) where f (n) ? T(nk)
  
• a lt bk T(n) ? T(nk)
• a bk T(n) ? T(nk lg n )
• a gt bk T(n) ? T(nlog b a)
• Note the same results hold with O instead of T.

39
Solving linear homogeneous recurrence relations
with constant coefficients

• Easy first 1st order LHRRCCs
• C(n) a C(n -1) C(0) t
  Solution C(n) t an
• Extrapolate to 2nd order
• L(n) a L(n-1) b L(n-2)
  A solution? L(n) r n
• Characteristic equation (quadratic)
• Solve to obtain roots r1 and r2 e.g. A(n)
  3A(n-1) - 2(n-2)
• General solution to RR linear combination of r1n
  and r2n
• Particular solution use initial conditions
  e.g.A(0) 1 A(1) 3