Analysis of Algorithms

1
Topic 9

• Analysis of Algorithms
• (before Sorting)

2
Chapter Objectives
3
Run Time of a Program

• What factors can affect the actual run time of a
  program?
• number of instructions executed per input
• number of data items (size of input)
• e.g. sorting 5 items vs 1000 items
• contents of the input
• e.g. integer data vs more complex structure
• optimization made by compiler
• e.g. compiler may put frequently used variables
  into registers
• speed of various instructions in the machine used

4
Analysis of Algorithms

• it is more productive to analyze the efficiency
  of an algorithm, at the design stage
• efficiency of an algorithm can be measured in
  terms of
• time complexity the amount of time required to
  execute an algorithm
• space complexity memory storage space required

5
Time Complexity

• How can we measure the time required to execute
  an algorithm?
• implement the algorithm and run the program
• what are some problems with doing this?
• analyse the algorithm and the input, and predict
  the runtime behaviour of the program
• determine feasibility of the program
• compare different methods before doing
  implementation

6
Time Complexity Measurement

• essentially based on the number of basic
  operations in an algorithm
• do the number of basic operations actually have
  to be counted?
• no, they are proportional to such things as
• number of arithmetic operations performed
• number of comparisons
• number of times through a critical loop
• number of array elements accessed
• etc.

7
Example Polynomial Evaluation
8
Method of Analysis
9
Analysis of Brute Force Method
10
Analysis of Horners Method
11
Big-O Notation

• formally the time complexity T(n) of an
  algorithm is O(f(n)) (we say of the order
  f(n))if, for some constant C and for all but
  finitely many values of n,T(n) lt C f(n)
• what does this mean? this gives an upper bound on
  the number of operations, for sufficiently large
  n

12
Time Complexity of Brute Force Method
13
Time Complexity of Horners Method
14
Time Complexity and Input

• run time can depend only on the size of the input
  (e.g. sorting 5 items vs 1000 items)
• run time can also depend on the particular input
  (e.g. suppose the input is already sorted)
• this leads to several kinds of time complexity
  analysis
• worst case analysis
• average case analysis
• best case analysis

15
Worst/Average/Best Case Analysis

• worst case analysis considers the maximum of the
  time over all inputs of size n
• average case analysis considers the average of
  the time over all inputs of size n
• best case analysis considers the minimum of the
  time over all inputs of size n

16
Discussion

• problems with average case analysis
• hard to determine
• depends on distribution of inputs
• so, we usually use worst case analysis(why not
  best case analysis?)

17
Example Linear Search
18
Example Binary Search
19
Big-O in General

• to determine the time complexity of an algorithm
• look at the loop structure
• identify the basic operation(s)
• express the number of operations asf1(n) f2(n)
  
• identify the dominant term (by definition, the fi
  such that fj is O(fi) for all j)
• then the time complexity of the algorithm is O(fi)

20
Dominant Terms

• examples of dominant terms
• n dominates log (n) (log base 2)
• n log(n) dominates n
• n2 dominates n log(n)
• nm dominates nk when m gt k
• an dominates nm for any a gt 1 and m gt 0
• we can write this as log (n) lt n lt n log(n) lt
  n2 lt lt nm lt anfor a gt 1 and m gt 2

21
Examples of Big-O Analysis

• independent nested loops
• int x 0for (int i 1 i lt n/2 i) for
  (int j 1 j lt nn j) x x i
  j
• number of iterations of inner loop is independent
  of the value of i
• how many times through outer loop?
• how many times through inner loop?
• time complexity of algorithm?

22
Examples (contd)

• dependent nested loops
• int x 0for (int i 1 i lt n i) for
  (int j 1 j lt 3i j) x x j
  
• number of iterations of inner loop depends on the
  value of i in the outer loop
• on kth iteration of outer loop, how many times
  through inner loop?
• total number of iterations of inner loop sum
  for k running from 1 to n
• time complexity of algorithm?

23
Usefulness of Big-O

• we can compare algorithms for efficiency
• examples?
• we can estimate actual run times if we know the
  time complexity of the algorithm(s) we are
  analyzing

24
Estimating Run Times

• assuming a million operations per second on a
  computer, here are some typical complexity
  functions f(n) and their associated
  runtimesf(n) n 103 n 105 n
  106---------------------------------------------
  ---------------------------------
• log(n) 10-5 sec. 1.710-5 sec. 210-5 sec.
• n 10-3 sec. 0.1 sec. 1 sec.
• n log(n) 0.01 sec. 1.7 sec. 20 sec.
• n2 1 sec. 3 hours 12 days
• n3 17 mins. 32 years 317 centuries
• 2n 10285 cent. 1010000 yrs. 10100000 yrs.

25
Discussion

• Suppose we want to perform a sort that is O(n2).
  What happens if the number of items to be sorted
  is 100000?
• Compare this to a sort that is O(n log(n)). Now
  what can we expect?
• Is an O(n3) algorithm practical for large n?
• What about an O(2n) algorithm, even for small n?
  e.g. for a Pentium, runtimes are n 30 n
  40 n 50 n 60 11 sec. 3 hours 130
  days 365 years

26
Mathematical Induction

• mathematical induction a formal proof technique
  that is typically used to prove that some
  property P holds for all positive integers(or,
  for positive integers from some point on, i.e.
  greater than or equal to some constant B)
• often used in time complexity analysis

27
Inductive Proofs

• an inductive proof involves the following steps
• induction basis show that the property P holds
  for some base case B
• often B0 or B1
• induction hypothesis assume that P holds for
  some value k gt B
• induction step using the induction hypothesis,
  show that P also holds for k1

28
Discussion

• Why does induction work?

29
Induction Example 1

• Using mathematical induction, provethe sum of
  the first n integers greater than 0 is n(n1)/2
• induction basis for n 1
• the sum of 1 is 1
• on the other hand, n (n1)/2 1
• therefore, the property holds for n 1
• induction hypothesis assume that the sum of the
  first k integers isk(k1)/2 for some k gt 1

30
Induction Example 1 (contd)

• induction step show that the sum of the first
  (k1) integers is (k1)(k2)/2
• the sum of the first (k1) integers (k1)
  (the sum of the first k integers) (k1)
  (k(k1)/2) by induction hypothesis
  (k1)(k2)/2
• thus, by mathematical induction, we have proved
  that the property holds for all n gt 1

31
Induction Example 2

• Using mathematical induction, provethe sum of
  the first n powers of 2, for n gt0,is n(n1)/2
• What is the induction basis?
• What is the induction hypothesis?
• What is the induction step?

32
Time Complexity forTowers of Hanoi

• Use mathematical induction to prove that solving
  the n-disk problem requires 2n moves, for n gt 1
• insert code for T of H

33
Contd

• induction basis n 1
• in the algorithm, if n 1, one move is required
• on the other hand, if n 1 then2n 1 1
• so, the property holds for n 1
• induction hypothesis assume that, for some k gt
  1, the towers of Hanoi problem requires 2k 1
  moves

34
Contd

• induction step show that the (k1)-disk problem
  is solved in 2(k1) 1 moves
• in the algorithm, the if condition is false
• therefore, we have the k-disk problem followed by
  a single move followed by the k-disk problem
• using the induction hypothesis, the total number
  of moves required is (2k 1) 1 (2k 1)
  2(k1) 1
• thus, by mathematical induction, we have proved
  that the number of moves is 2(n-1) for all n gt 1

35
Discussion

• How does this function grow? (try it for n 1,
  2, 3, 10, 100, )
• What is the time complexity of the towers of
  Hanoi problem?
• Could we have come up with this without knowing
  the exact formula for the number of moves first?
• when an algorithm is recursive, we can use a
  recurrence relation to determine the time
  complexity instead

36
Using Recurrence Relations

• a recurrence relation requires two formulas
• one to express the number of operations for the
  base case
• one to express the number of operations for the
  general case, in terms of a smaller version of
  the problem

37
Using Recurrence Relation for Towers of Hanoi
Analysis

• base case when n 1, one move is needed
• so, our formula for the base case is T(1)
  1
• general case for n gt 1, we have the
  formulaT(n) T(n-1) 1 T(n-1) 1
  2T(n-1)

38
Contd

• iterating this formula, we getT(n) 1 2(1
  2T(n-2)) 1 2 22 T(n-2) 1 2 22
  (1 2T(n-3)) 1 2 22 23T(n-3)
  20 21 22 2(n-2)
  2(n-1)T(1) sum of the powers of 2, from 0
  to n-1 2n 1
• so, what is the time complexity for the Towers of
  Hanoi?

39
Space Complexity

• space complexity of an algorithm is the number of
  memory locations required for storage
• may also be measured using Big-O notation
• Time and space tradeoff
• Sometimes using more space can save time, and
  using more time can save space .