Analysis of Algorithms Efficiency II

1

• LECTURE 5
• Analysis of Algorithms Efficiency (II)

2
Last week we saw

• Which are the basic steps in analyzing the time
  efficiency of an algorithm
• Identify the input size
• Identify the dominant operation
• Count how many times the dominant operation is
  executed (estimate the running time)
• If this number depends on the properties of input
  data we analyze
• Best case gt lower bound of the running time
  
• Worst case gt upper bound of the running time
  
• Average case gt averaged running time

3
Today we will see that

• the main aim of efficiency analysis is to find
  out how the running time increases when the
  problem size increases
• we dont need very detailed expressions of the
  running time, but to identify
• The order of growth of the running time
• The efficiency class to which an algorithm
  belongs

4
Outline

• What is the order of growth ?
• What is asymptotic analysis ?
• Some asymptotic notations
• Efficiency analysis of basic processing
  structures
• Efficiency classes
• Empirical analysis of the algorithms

5
What is the order of growth?

• In the running time expression, when n becomes
  large a term will become significantly larger
  than the other ones
• this is the so-called
  dominant term

Dominant term a n Dominant term a log
n Dominant term a n2 Dominant term an
T1(n)anb T2(n)a log nb T3(n)a
n2bnc T4(n)anb n c (agt1)
6
What is the order of growth?

• Let us analyze what happens with the dominant
  term when the input size is multiplied by k

T1(kn) a knk T1(n) T2(kn)a
log(kn)T2(n)alog k T3(kn)a (kn)2k2
T3(n) T4(kn)akn(an)k T4(n)k
T1(n)an T2(n)a log n T3(n)a n2 T4(kn)an
7
What is the order of growth?

• The order of growth expresses how increases the
  dominant term of the running time with the input
  size

Order of growth Linear Logarithmic Quadratic Ex
ponential
T1(kn) a knk T1(n) T2(kn)a
log(kn)T2(n)alog k T3(kn)a (kn)2k2
T3(n) T4(kn)akn(an)k
8
How can be interpreted the order of growth?

• Between two algorithms it is considered that the
  one having a smaller order of growth is more
  efficient
• However, this is true only for large enough input
  sizes
• Example. Let us consider
• T1(n)10n10 (linear order of growth)
• T2(n)n2 (quadratic order of growth)
• If nlt10 then T1(n)gtT2(n)
• Thus the order of growth is relevant only for ngt10

9
A comparison of orders of growth
The multiplicative constants in the dominant term
can be ignored
10
Comparing orders of growth
The order of growth of two running times T1(n)
and T2(n) can be compared by computing the limit
of T1(n)/T2(n) when n goes to infinity.If the
limit is 0 then T1(n) has a smaller order of
growth than T2(n)If the limit is a constant c
(cgt0) then T1(n) and T2(n) have the same order of
growthIf the limit is infinity then T1(n) has a
larger order of growth than T2(n)
11
Outline

• What is the order of growth ?
• What is asymptotic analysis ?
• Some asymptotic notations
• Efficiency analysis of basic processing
  structures
• Efficiency classes
• Empirical analysis of the algorithms

12
What is asymptotic analysis ?

• The differences between orders of growths are
  more significant for larger input size
• Analyzing the running times on small inputs does
  not allow us to distinguish between efficient and
  inefficient algorithms
• Asymptotic analysis deals with analyzing the
  properties of the running time when the input
  size goes to infinity
• (this means a large input size)

13
What is asymptotic analysis ?

• Depending on the behavior of the running time
  when the input size becomes large, the algorithm
  can belong to different classes of efficiency
• To formalize, some classes of functions and some
  specific notations have been introduced
• ? (big Theta)
• O (big O)
• O (big Omega)

14
Outline

• What is the order of growth ?
• What is asympotic analysis ?
• Some asymptotic notations
• Efficiency analysis of basic processing
  structures
• Efficiency classes
• Empirical analysis of the algorithms

15
? - notation

• Let f,g N-gt R
• Definition.
• f(n) ??(g(n)) iff there exist c1, c2 ? R and
  n0 ?N such that
• c1g(n) f(n) c2g(n) for
  all nn0
• Notation. f(n) ? (g(n)) (same order of growth
  as g(n))
• Examples.
• T(n) 3n3 ? T(n) ? ? (n)
• c12, c24, n03, g(n)n
• T(n) n210 nlgn 5 ? T(n) ? ? (n2)
• c11, c22, n040, g(n)n2

16
? - notation
Graphical illustration. f(n) is bounded both
above and below by some positive constant
multiples of g(n) for all large n
c2g(n)2n2
c1g(n) f(n) c2g(n)
f(n) n210 nlgn 5
Doesnt matter
c1g(n)n2
?(n2)
n0
17
? - notation. Properties

• If T(n)aknkak-1nk-1a1na0 then T(n) ?
  ?(nk)
• Proof. Since T(n)gt0 for all n it follows that
  akgt0.
• Then T(n)/nk -gtak (as n-gt?).
• Thus for all egt0 there exists N(e) such that
• T(n)/nk- aklt e gt ak- eltT(n)/nkltak e
  for all ngtN(e)
• Let us suppose that ak- egt0.
• Then by taking c1(ak- e), c2ak e and n0N(e)
  one obtains
• c1nk lt T(n) ltc2nk for all ngtn0, i.e. T(n)
  ? ?(nk)

18
? - notation. Properties

• 2. ?(c g(n)) ?(g(n)) for all constant c
• Proof. Let f(n) ? ?(cg(n)).
• Then c1cg(n) f(n) c2cg(n) for all nn0.
• By taking c1 cc1 and c2 c c2 we obtain that
  f(n) ? ?(g(n)).
• Thus ?(cg(n))? ?(g(n)).
• Similarly we can prove that ? (g(n)) ? ?
  (cg(n)), so ? (cg(n)) ? (g(n)).
• Particular cases
• a) ?(c) ?(1)
• b) ?(logah(n)) ?(logbh(n)) for all a,b gt1
• The logarithms base is not significant in
  specifying the efficiency class. We will use
  logarithms in base 2

19
? - notation. More examples

• 3nltT(n) lt4n-1 ? T(n) ? ?(n)
• c13, c24, n01
• Multiplying two matrices T(m,n,p)4mnp5mp4m2
• Extension of the definition
• f(m,n,p) ? ?(g(m,n,p)) iff
• there exist c1, c2 ? R and
  m0,n0 ,p0 ? N such that
• c1g(m,n,p) ltf(m,n,p) ltc2g(m,n,p) for all
  mgtm0, ngtn0, pgtp0
• Thus T(m,n,p) ? ?(mnp)
• Sequential search 6lt T(n) lt 3(n1)
• If T(n)6 then we cannot find c1 such that 6 gt
  c1n for large values of n.
• Thus T(n) does not belong to ?(n).
• There exist running times which do not belong to
  a big-theta class.

20
O - notation

• Definition.
• f(n)? O(g(n)) iff there exist c ? R and n0 ? N
  such that
• f(n) ltc g(n) for all
  ngtn0
• Notation. f(n) O(g(n)) (an order of growth at
  most as that of g(n))
• Examples.
• T(n) 3n3 ? T(n) ? O(n)
• c4, n03, g(n)n
• 2. 6lt T(n) lt 3(n1)? T(n) ? O(n)
• c4, n03, g(n)n

21
O - notation

• Graphical illustration. f(n) is bounded above
  by some positive constant multiples of g(n) for
  all large n

cg(n)n2
f(n)ltcg(n)
doesnt matter
f(n) 10nlgn 5
O(n2)
n036
22
O notation. Properties

• If T(n)aknkak-1nk-1a1na0
• then T(n) ? O(nd) for all dgtk
• Proof. Since T(n)gt0 for all n it follows that
  akgt0.
• Then T(n)/nk -gt ak (as n-gt?).
• Thus for all egt0 there exists N(e) such that
• T(n)/nk lt ak e for all
  ngtN(e)
• Hence T(n) lt (ak e)nk lt (ake)nd
• Then by taking cak e and n0N(e) one obtains
• T(n) ltcnd for all ngtn0, i.e. T(n) ? O(nd)
• Example.
• n ? O (n2)
• (it is correct but is more useful to
  write n ? O (n))

23
O notation. Properties

• 2. ?(g(n)) is a subset of O(g(n)
• Proof. It suffices to consider only the upper
  bound from big-theta definition
• Remark. The inclusion is a strict one there
  exist elements of O(g(n)) which do not belong to
  ?(g(n))
• Example
• f(n)10nlgn5, g(n)n2
• f(n)ltg(n) for all ngt36 ? f(n)?O(g(n))
• But it dont exist constants c and n0 such that
• cn2 lt 10nlgn5 for all n gt n0

24
O notation. Properties

• When by a worst case analysis we obtain that
• T(n) g(n) we can say that the running time of
  the algorithm belongs to O(g(n))
• Example. Sequential search 6lt T(n) lt 3(n1)
• Thus the running time of sequential search
  belongs to O(n)

25
O notation

• Definition.
• f(n)? O(g(n)) iff there exist c ? R and n0 ? N
  such that
• cg(n) lt f(n) for all
  ngtn0
• Notation. f(n) O(g(n)) (an order of growth at
  least as that of g(n))
• Examples.
• T(n) 3n3 ? T(n) ? O(n)
• c3, n01, g(n)n
• 2. 6lt T(n) lt 3(n1)? T(n) ? O(1)
• c6, n01, g(n)1

26
O notation

• Graphical illustration. f(n) is bounded below
  by a positive constant multiple of g(n) for all
  large n

f(n)10nlgn5
cg(n)ltf(n)
Doesnt matter
cg(n)20n
O(n)
n07
27
O notation. Properties

• If T(n)aknkak-1nk-1a1na0
• then T(n) ? O(nd) for all dltk
• Proof. Since T(n)gt0 for all n it follows that
  akgt0.
• Then T(n)/nk -gt ak (as n-gt?).
• Thus for all egt0 there exists N(e) such that
• ak - e lt T(n)/nk for all
  ngtN(e)
• Hence (ak -e)nd lt(ak-e)nk lt T(n)
• Then by taking cak- e and n0N(e) one obtains
• cnd lt T(n) for all ngtn0, i.e. T(n) ?
  O(nd)
• Example.
• n2 ? O (n)

28
O notation. Properties

• 2. ?(g(n))? O(g(n)
• Proof. It suffices to consider only the lower
  bound from big-theta definition
• Remark. The inclusion is a strict one there
  exist elements of O(g(n)) which do not belong to
  ?(g(n))
• Example
• f(n)10nlgn5, g(n)n
• f(n) gt 10g(n) for all ngt1 ? f(n) ? O(g(n))
• But it dont exist constants c and n0 such that
• 10nlgn5ltcn for all n gt n0
• 3. ?(g(n))O(g(n))?O(g(n)

29
Outline

• What is the order of growth ?
• What is asympotic analysis ?
• Some asymptotic notations
• Efficiency analysis of basic processing
  structures
• Efficiency classes
• Empirical analysis of the algorithms

30
Efficiency analysis of basic processing structures

• Sequential structure
• P
• P1 ?(g1(n)) O(g1(n)) O(g1(n))
• P2 ?(g2(n)) O(g2(n)) O(g2(n))
• Pk ?(gk(n)) O(gk(n)) O(gk(n))
• --------------------------------------------------
  --
• ?(maxg1(n),g2(n), , gk(n))
• O(maxg1(n),g2(n), , gk(n))
• O(maxg1(n),g2(n), , gk(n))

31
Efficiency analysis of basic processing structures

• Conditional statement
• P
• IF ltconditiongt
• THEN P1 ?(g1(n))
  O(g1(n)) O(g1(n))
• ELSE P2 ?(g2(n))
  O(g2(n)) O(g2(n))
• --------------------------------------------------
  -------------------------
• ?(maxg1(n),g2(n))
• O(maxg1(n),g2(n))
• O(ming1(n),g2(n))

32
Efficiency analysis of basic processing structures

• Loop statement
• P
• FOR i1, n DO
• P1 ?(1) ? ?(n)
• FOR i1,n DO
• FOR j1,n DO
• P1 ?(1) ?
  ?(n2)
• Remark If the counting variables vary between 1
  and n the complexity is nk (k is the number
  superposed loops)

33
Efficiency analysis of basic processing structures

• Remark.
• If the limits of the counters are
  modified inside the loop body then the analysis
  need to be modified
• Example
• m1
• FOR i1,n DO
• m3m m3i
• FOR j1,m DO
• processing step from ?(1)
• The complexity of the sequence is
• 3323n (3n1-1)/2-1
• The complexity ?(3n)

34
Outline

• What is the order of growth ?
• What is asymptotic analysis ?
• Some asymptotic notations
• Efficiency analysis of basic processing
  structures
• Efficiency classes
• Empirical analysis of the algorithms

35
Efficiency classes

• Some frequently encountered efficiency classes

36
Outline

• What is the order of growth ?
• What is asympotic analysis ?
• Some asymptotic notations
• Efficiency analysis of basic processing
  structures
• Efficiency classes
• Empirical analysis of the algorithms

37
Empirical analysis of the algorithms

• Sometimes the mathematical analysis of efficiency
  is too difficult to apply in these cases the
  empirical analysis could be useful
• It can be used to
• Develop a hypothesis about the algorithms
  efficiency
• Compare the efficiency of several algorithms
  designed to solve the same problem
• Establish the efficiency of an algorithms
  implementation
• Check the accuracy of a theoretical assertion
  about algorithms efficiency

38
General plan for empirical analysis

• Establish the aim of the analysis
• Choose an efficiency measure (e.g. number of
  executions of some operations or time needed to
  execute a sequence of processing steps)
• Decide on the characteristics of the input sample
  (size, range )
• Implement the algorithm in a programming
  language
• Generate a set of data inputs
• Execute the program for each data sample and
  record the results
• Analyze the obtained results

39
General plan for empirical analysis

• Efficiency measure. It is chosen depending on
  the aim of the empirical analysis
• If the aim is to estimate the efficiency class an
  adequate efficiency measure is the number of
  operations
• If the aim is to analyze/compare the
  implementation of an algorithm on a given machine
  an adequate efficiency measure is the physical
  time

40
General plan for empirical analysis

• Set of input data. Different input data must be
  generated in order to conduct a useful empirical
  analysis
• Some rules in generating input data
• The input data in the set should be of different
  sizes and values (the entire range of values
  should be represented)
• All characteristics of input data should be
  represented in the sample set (different
  configurations)
• The data should be typical (not only exceptions)

41
General plan for empirical analysis

• Algorithms implementation. Some monitoring
  processing step should be included
• Counting variables (when the efficiency measure
  is the number of executions)
• Calls of some functions which return the current
  time (in order to estimate the time needed to
  execute a processing sequence)

42
Next lecture will be on

• elementary sorting algorithms
• on their correctness and
• on their efficiency
• MONDAY instead of Logics (only for the next
  week)