Mathematical Analysis of Algorithms

1
Mathematical Analysisof Algorithms
A study of methods used to determine the quality
of algorithms for particular problems How does a
specific algorithm perform as the size of the
data set increases?
2
Algorithm Performance

• An algorithms performance is dependent on the
  size of data and order of data.
• If we send random data thru an algorithm we will
  be studying the average-case performance.
• If we send a carefully designed sequence thru an
  algorithm we are often looking at either worst
  case or best case performance.

3
Growth of Functions

• Most algorithms have a primary parameter N that
  affects the runtime of the algorithm.
• This may be the degree of a polynomial
• The size of the data being sorted.
• The number of characters in a string
• Some may have two or more parameters.( IE
  Union/Find had two M and N)

4
Typical Functions Encountered(1)

• 1 (constant) Here the number of instructions
  executed is constant no matter the size of the
  data set size N.
• lg N (logarithmic) Here the amount of work is
  increased by 1 if the size of N doubles.
• N (linear) Here a constant amount of processing
  is done on each input element.

5
Typical Functions Encountered(2)

• N lg N often occurs when we divide a problem
  into smaller pieces, solve those and recombine
  the sub-solutions.
• N2 (quadratic) We are slowly getting worse. Here
  if the data set size double the amount of work is
  multiplied by 4. Double loops!
• N3 ( cubic ) Practical for only small problems
• 2N (exponential) When N doubles the running time
  squares. VERY bad!

6
Practical Complexity
7
Practical Complexity
8
Practical Complexity
9
Just an Interesting Question
What is the optimal base to use in the
representation of numbers ? n? Example
with base x we have _ _ _ _ _ _ _ _
_
We minimize
X values
slots
10
Logarithm Review
Ln loge is called the natural logarithm Lg
log2 is called the binary logarithm How many
bits are required to represent the number n in
binary
11
Logarithm Rules
The logarithm to the base b of x denoted logbx is
defined to that number y such that
by x
logb(x1x2) logb x1 logb x2 logb(x1/x2)
logb x1 - logb x2 logb xc c logbx
logbx gt 0 if x gt 1 logbx 0 if x 1 logbx lt 0
if 0 lt x lt 1
12
Additional Rules
For all real agt0, bgt0 , cgt0 and n logb a logca
/ logc b logb (1/a) - logb a logb
a 1/ logab a logb n n logb
a
13
Additional Rules
For all real agt0, bgt0 , cgt0 and n logb a logca
/ logc b logb (1/a) - logb a logb
a 1/ logab a logb n n logb
a
14
Big O Definition
A function g(n) is said to be O(f(n) if
there exists constants c and an n0 such that
g(n) ? c f(n) for every n ? n0
f(n)
g(n)
n0
15
Big O example
Show using the definition that 5n4 ?O(n) Where
g(n)n First we must find a c and an n0 We now
need to show that f(n) ? c g(n)
for every n ? n0 clearly 5n 5 ? 6n
whenever n ? 6 Hence c6 and n06 satisfy the
requirements.
16
Upper Bound Notation

• We say InsertionSorts run time is O(n2)
• Properly we should say run time is in O(n2)
• Read O as Big-O (youll also hear it as
  order)
• In general a function
• f(n) is O(g(n)) if there exist positive constants
  c and n0 such that f(n) ? c ? g(n) for all n ? n0
• Formally
• O(g(n)) f(n) ? positive constants c and n0
  such that f(n) ? c ? g(n) ? n ? n0

17
Arithmetic Series Review
1 2 3 . . . n ?
Sn a ad a2d a3d . . .
a(n-1)d Sn 2a(n-1)d 2a(n-2)d 2a(n-3)d
a 2Sn 2a(n-1)d 2a(n-1)d 2a(n-1)d
. . . 2a(n-1)d Sn n/22a (n-1)d
consequently
123n n(n1)/2
18
Problems
Find the sum of the following 135 . . .
121 ? The first 50 terms of -3 3 9 15
1 3/2 2 5/2 . . . 25?
19
Geometric Series Review
1 2 4 8 . . . 2n 1 1/2 1/4 .
. . 2-n
Theorem
Sn a ar ar2 . . . arn rSn ar
ar2 . . . arn arn1 Sn-rSn a - arn1

What about the case where -1lt r lt 1 ?
20
Geometric Problems
What is the sum of 39/4 27/16 . . . 1/2
- 1/4 1/8 - 1/16 . . .
21
Harmonic Series
This is Eulers constants
22
"There's nothing in your head the sorting hat
can't see. So try me on and I will tell you where
you ought to be." -The Sorting Hat, Harry
Potter and the Sorcerer's Stone
23
What is Sorting

• Problem Arrange a list of numbers, words or
  records into appropriate order
• numbers are placed in numerical order
• words are placed in lexicographic order
• records are place in order as defined by a key
  field
• A sorting Algorithm is a program that does the
  above using a specific method . Some
  algorithms/methods are good some are not!

24
Example Sorting Algorithms

• Bubble Sort- the worst of all!
• Insertion Sort- poor choice for large data sets
• Selection Sort- poor choice for large data sets
• Shell Sort- We are not sure how good this is
• Merge Sort- Fast but uses a lot of memory
• Heap Sort- Faster. Uses priority queues
• Quick Sort- Statistically faster than the above

25
Selection Sort
For i1 to n do // On the ith pass, find the
smallest element in sublist i to n smallPosi
smallestxsmallPos For ji1 to n-1 do if
xjltsmallest smallPosj smallestxsmallPos
//Now interchange smallest with first of
sub-list xsmallPosxI sIsmallest
Sorted
i
n
26
Work done by the selection sort
First pass n checks for size Second pass n-1
checks Third pass n-2 checks . . . Nth pass 1
check N(N1)/2 (Why?)
i
n
Amount of work increases with respect to the
square of N
27
Insertion Sort
In this algorithm the program does the following
for i 2 to n do // Insert the ith
element into the sorted list from 1 to i-1
2
5
7
11
17
6
21
1
15
46
36
3
14
8
i
Remove 6 and then slide 17,11, and 7 over to the
right. Then place 6 in the third slot.
Sorted up to i-1
28
Insertion Sort
For i2 to n do //Insert xi into its proper
position from 1 to i-1 nextElementxi x0
nextElement ji While nextElementltxj-1
do //Shift element to the right to open a
spot xjxj-1 jj-1 sjnextElement
i
Sorted
Number of accesses used here??
29
Work Done by Insertion Sort

• On the first loop we look at one value
• On the second loop we look at two values
• On the third loop we look at three values
• On the nth loop we look at n values
• 1 2 3 4 5 . . . n-1 n n(n1)/2

30
The Bubble Sort
for (i0 iltn-1 i) // Check the subscripting
here closely for (j0 jltn-1-i j) if
(xj1 lt xj) / compare the two neighbors /
tmp xj / swap aj and aj1 /
xj xj1 xj1 tmp
This algorithm also used n2 accesses
31
Complexity
All three algorithms actually use on the order of
n2 accesses to the array for an array of size
n. What does this really mean?
It means that if we double the size of the data
set then the execution time will be multiplied by
4.
Suppose that the number of accesses is n3 What
does this really mean?
It means that if we double the size of the data
set then the execution time will be multiplied
by 8.
32
Complexity

• We refer to n2 algorithms as having O(n2)
  complexity.
• Complexity is a measure of the work done as a
  function of the size of the data set.
• Efficiency is not the same as Complexity.

What is the complexity of merging two array?
33
The Merge Algorithm
Next value to compare with B
2
4
7
11
17
21
29
A
2
3
4
7
3
6
9
16
23
27
31
B
Next value to compare with A
Homework Code a C function the does the merge
operation.
What is the complexity of the the merge operation?
O(n)
34
Faster SortingMerge Sort
MergeSort(A, left, right) if (left lt right)
mid floor((left right) /
2) MergeSort(A, left, mid) MergeSort(A,
mid1, right) Merge(A, left, mid,
right) // Merge() takes two sorted
subarrays of A and// merges them into a single
sorted subarray of A.// Merge()takes O(n) time,
n length of A
35
MergeSort Operation Tree
n items
Work Done to merge
n
n/2
n/2
n/4
n/4
n
n/4
n/4
n
n/8 items
n/16 items
n
n/32 items
n
How many ns are there here???
36
Mergesort Complexity

• The Complexity of Merge sort is O(nlog2n)
• This is the best complexity you can obtain when
  you use comparisons in your sort
• A drawback is that it requires an extra array
  thereby doubling memory requirements
• It is usually used to sort external files
• Generally written recursively.

37
Other O(nlog2n) Sorts

• Heap Sort
• Worst case is always nlog2n
• Average case is nlog2n
• Quick Sort
• average case is nlog2n
• worst case is n2