Introduction to complexity

1
Introduction to complexity
2
Analysis of Algorithms

• Why do we need to analyze algorithms?
• To measure the performance
• Comparison of different algorithms
• Can we find a better one? Is this the best
  solution?
• Running time analysis
• Memory usage

3
Complexity a measure of the performance of an
algorithm
An algorithms performance depends on internal
and external factors

• External
• Size of the input to the algorithm
• Speed of the computer on which it is run
• Quality of the compiler

• Internal
• The algorithms efficiency, in terms of
• Time required to run
• Space (memory storage) required to run

Complexity measures the internal factors (usually
more interested in time than space)
4
Running Time Analysis
Algorithm 1 T1(N)1000N Algorithm 2 T2(N)N2
Running Time T1(n)
N
Algorithm 1
5
Running Time Analysis
Algorithm 2
Running Time T(N)
Algorithm 1
N
1000
6
Summary of Running Times
N T1 T2
10 10-2 sec 10-4 sec
100 10-1 sec 10-2 sec
1000 1 sec 1 sec
10000 10 sec 100 sec
100000 100 sec 10000 sec
For the values of N that are less than 1000 the
difference bw. running times of the two
algorithms can be ignored
7
Growth rates and big-O notation

• Growth rates capture the essence of an
  algorithms performance
• Big-O notation indicates the growth rate. It is
  the class of mathematical formula that best
  describes an algorithms performance, and is
  discovered by looking inside the algorithm
• Big-O is a function with parameter N, where N is
  usually the size of the input to the algorithm
• For example, if an algorithm depending on the
  value n has performance an2 bn c (for
  constants a, b, c) then we say the algorithm has
  performance O(N2)
• For large N, the N2 term dominates. Only the
  dominant term is included in big-O

8
big-O notation Asympthotic Upper Limit

• Running time of an algorithm
• T(N)O(f(n))

This is not a function, but a notation.

• If there are c and n0 values satisfying T(N) ?
  c f(n) for N ? n0 then T(N) ? c f(n)
• f(n), is an asymptotic upper bound for T(N) and
  T(N)O(f(n)).

9
big-O notation

• big-O notation is written in the simplest form
• Example
• 3n22n5 O(n2)
• The followings are correct, but not used
• 3n22n5 O(3n22n5)
• 3n22n5 O(n2n)
• 3n22n5 O(3n2)

10
Common growth rates
Time complexity Example
O(1) constant Adding to the front of a linked list
O(log N) log Finding an entry in a sorted array
O(N) linear Finding an entry in an unsorted array
O(N log N) n-log-n Sorting n items by divide-and-conquer
O(N2) quadratic Shortest path between two nodes in a graph
O(N3) cubic Simultaneous linear equations
O(2N) exponential The Towers of Hanoi problem
11
Growth rates
O(N2)
O(Nlog N)
Time
For a short time N2 is better than NlogN
Number of Inputs
12
Calculating the actual time taken by a program
(example)

• A program takes 10ms to process one data item
  (i.e. to do one operation on the data item)
• How long would the program take to process 1000
  data items, if time is proportional to
• log10 N
• N
• N log10 N
• N2
• N3

• (time for 1 item) x (big-O( ) time complexity
  of N items)

13
How do we calculate big-O?
Five guidelines for finding out the time
complexity of a piece of code

• 1 Loops
• 2 Nested loops
• 3 Consecutive statements
• 4 If-then-else statements
• 5 Logarithmic complexity

14
Guideline 1 Loops
The running time of a loop is, at most, the
running time of the statements inside the loop
(including tests) multiplied by the number of
iterations.
for (i1 iltn i) m m 2
Total time a constant c n cn O(N)
15
Guideline 2 Nested loops
Analyse inside out. Total running time is the
product of the sizes of all the loops.
for (i1 iltn i) for (j1 jltn j)
k k1
Total time c n n cn2 O(N2)
16
Guideline 3 Consecutive statements
Add the time complexities of each statement.
x x 1 for (i1 iltn i) m m
2 for (i1 iltn i) for (j1 jltn
j) k k1
Total time c0 c1n c2n2 O(N2)
17
Guideline 4 If-then-else statements
Worst-case running time the test, plus either
the then part or the else part (whichever is the
larger).
if (depth( ) ! otherStack.depth( ) ) return
false else for (int n 0 n lt depth( )
n) if (!listn.equals(otherStack.listn))
return false
another if constant constant (no else part)
Total time c0 c1 (c2 c3) n O(N)
18
Guideline 5 Logarithmic complexity
An algorithm is O(log N) if it takes a constant
time to cut the problem size by a fraction
(usually by ½)
Example algorithm (binary search) finding a word
in a dictionary of n pages

• Look at the centre point in the dictionary
• Is word to left or right of centre?
• Repeat process with left or right part of
  dictionary until the word is found

19
big-O notation - Example 1

• Prove/disprove that 3n22n5 O(n2)
• 10 n2 3n2 2n2 5n2
• ? 3n2 2n 5 for n ? 1
• c 10, n0 1

The number of (n0,c) pairs is not important.
Having only one pair satisfying the condition is
enough.
20
big-O notation - Example 2

• If T(N) can be expressed as T(N)O(7n25n4),
  T(N) can be anything from the followings
• T(N)n2
• T(N)4n7
• T(N)1000n22n300
• T(N) O(7n25n4) O(n2)

21
big-O notation - Example 3

• Write the big-O notations for the follwing
  running times
• f1(n) 10 n 25 n2
• f2(n) 20 n log n 5 n
• f3(n) 12 n log n 0.05 n2
• f4(n) n1/2 3 n log n

• O(n2)
• O(n log n)
• O(n2)
• O(n log n)

22
Analysis of Running Times

• Strategy Upper and lower bounds

Upper bound
Running time function
Lower bound
23
Analysis of Running Times

• Determing the running times exactly is a
  difficult task. Therefore
• Analysis of the best cases
• Analysis of the average cases which is quite
  difficult
• Analysis of the worst cases which is easier
• have to be considered

24
Best, average, worst-case complexity

• In some cases, it is important to consider the
  best, worst and/or average (or typical)
  performance of an algorithm
• E.g., when sorting a list into order, if it is
  already in order then the algorithm may have very
  little work to do
• The worst-case analysis gives a bound for all
  possible input (and may be easier to calculate
  than the average case)

• Worst, O(N) or o(N) ? or gt true function
• Typical, T(N) ? true function
  
• Best, O(N) ? true function
  

These approximations are true only after N has
passed some value
25
? Notation- Asymptotic Bound

• Reverse of big-O notation
• If there exist positive c and n0 constatnts can
  be found for T(N) ? c f(n) where N ? n0 then
  T(N)?(f(n))
• f(n) is an asymptotic lower bound of T(N)

f(n)
c g(n)
n0
26
? Notation- Example 1

• 7n23n5 O(n4)
• 7n23n5 O(n3)
• 7n23n5 O(n2)
• 7n23n5 ?(n2)
• 7n23n5 ?(n)
• 7n23n5 ?(1)

27
Algorithm 1

• int Sum (int N)
• int i, PartialSum
• PartialSum0
• for(i1 iltN i)
• PartialSumiii
• return PartialSum

1
1(N1)N
NN2N
1
Running Time 6N4O(N)
28
Algorithm 2

• for(i0 iltN i)
• for(j1 jltN i)
• k

Running Time O(N2)
for(i0 iltN i) Ai0 for(i0 iltN
i) for(j1 jltN i) AiAjij
Running Time O(N2)
29
Algorithm 3

• If( condition )
• S1
• Else
• S2

Running Time max_ Running_Time (S1,S2)
30
Algorithm 4

• int binary search(A,key,N)
• low0, highN-1
• while(low?high)
• mid(lowhigh)/2
• if(Amidltkey)
• lowmid1
• if(Amidgtkey)
• highmid-1
• if(Amidkey)
• return mid
• Return not found

Running Time O(logN). Since the number of input
is halved in each iteration.
31
Algorithm 5

• int binarysearch(A,key,low,high)
• if (lowgthigh)
• Return not found
• else
• mid(lowhigh)/2
• if(Amidltkey)
• Return binarysearch(A,key,mid1,high)
• if(Amidgtkey)
• Return binary search(A,key,low,mid-1)
• if (Amidkey)
• Return mid

T(N)T(N/2)O(1)
32
Algorithm 6

• MaxSubsequenceSum(const int A, int n)
• ThisSumMaxSum0
• for(j0jltNj)
• ThisSumAj
• if (ThisSumltMaxSum)
• MaxSumThisSum
• else if(ThisSumlt0)
• ThisSum0
• Return MaxSum

Running Time O(N)
33
Performance isnt everything!

• There can be a tradeoff between
• Ease of understanding, writing and debugging
• Efficient use of time and space
• So, maximum performance is not always desirable
• However, it is still useful to compare the
  performance of different algorithms, even if the
  optimal algorithm may not be adopted