Chapter 9: Algorithm Efficiency and Sorting

1
Chapter 9 Algorithm Efficiency and Sorting

• Chien Chin Chen
• Department of Information Management
• National Taiwan University

2
Measuring the Efficiency of Algorithms (1/5)

• Measuring an algorithms efficiency is very
  important.
• Your choice of algorithm for a application often
  has a great impact.
• Components that contribute to the cost of a
  computer program
• The cost of human time.
• Time of development, maintenance
• The cost of program execution.
• The amount of computer time and space that the
  program requires to execute.

3
Measuring the Efficiency of Algorithms (2/5)

• Analysis of algorithms
• Provides tools for contrasting the efficiency of
  different algorithms.
• Time efficiency, space efficiency
• Should focus on significant differences in
  efficiency.
• Should not consider reductions in computing costs
  due to clever coding tricks.

4
Measuring the Efficiency of Algorithms (3/5)

• This chapter focuses on time efficiency.
• Comparison implement different programs check
  which one is faster.
• Three difficulties with comparing programs
  instead of algorithms
• How are the algorithms coded?
• What computer should you use?
• What data should the programs use?
• The most important difficulty.

5
Measuring the Efficiency of Algorithms (4/5)

• Algorithm analysis should be independent of
• Specific implementations.
• Computers.
• Data.
• How?
• Counting the number of significant operations in
  a particular solution.

6
Measuring the Efficiency of Algorithms (5/5)

• Counting an algorithm's operations is a way to
  assess its time efficiency.
• An algorithms execution time is related to the
  number of operations it requires.
• Example Traversal of a linked list of n nodes.
• n 1 assignments, n 1 comparisons, n writes.
• Example The Towers of Hanoi with n disks.
• 2n - 1 moves.

? 1 assignment
Node cur head while (cur ! NULL) cout ltlt
cur-gtitem ltlt endl cur cur-gtnext
? n1 comparisons
? n writes
? n assignments
7
Algorithm Growth Rates (1/3)

• An algorithms time requirements can be measured
  as a function of the problem size (instance
  characteristic).
• Number of nodes in a linked list.
• Size of an array.
• Number of items in a stack.
• Number of disks in the Towers of Hanoi problem.
• Algorithm efficiency is typically a concern for
  large problems only.

8
Algorithm Growth Rates (2/3)
Varies with different computers and
implementations.

• Algorithm A requires time proportional to n2
• Algorithm B requires time proportional to n

9
Algorithm Growth Rates (3/3)

• An algorithms growth rate
• How quickly the algorithms time requirement
  grows as a function of the problem size.
• Algorithm A requires time proportional to n2.
• Algorithm B requires time proportional to n.
• Algorithm B is faster than algorithm A.
• n2 and n are growth-rate functions.
• A mathematical function used to specify an
  algorithms order in terms of the size of the
  problem.
• Algorithm A is O(n2) - order n2.
• Algorithm B is O(n) - order n.
• Big O notation.

10
Order-of-Magnitude Analysis and Big O Notation
(1/5)

• Definition of the order of an algorithm
• Algorithm A is order f(n) denoted O (f(n))
  if constants k and n0 exist such that A requires
  no more than k f(n) time units to solve a
  problem of size n n0.

11
Order-of-Magnitude Analysis and Big O Notation
(2/5)
O(1) constant
12
Order-of-Magnitude Analysis and Big O Notation
(3/5)
13
Order-of-Magnitude Analysis and Big O Notation
(4/5)

• Order of growth of some common functions
• O(1) lt O(log2n) lt O(n) lt O(n log2n) lt O(n2) lt
  O(n3) lt O(2n).
• Properties of growth-rate functions
• O(n3 3n) is O(n3) ignore low-order terms.
• O(5 f(n)) O(f(n)) ignore multiplicative
  constant in the high-order term.
• O(f(n)) O(g(n)) O(f(n) g(n)).

14
Order-of-Magnitude Analysis and Big O Notation
(5/5)

• An algorithm can require different times to solve
  different problems of the same size.
• Average-case analysis
• A determination of the average amount of time
  that an algorithm requires to solve problems of
  size n.
• Best-case analysis
• A determination of the minimum amount of time
  that an algorithm requires to solve problems of
  size n.
• Worst-case analysis
• A determination of the maximum amount of time
  that an algorithm requires to solve problems of
  size n.
• Is easire to calculate and is more common.

15
Keeping Your Perspective (1/2)

• Only significant differences in efficiency are
  interesting.
• Frequency of operations
• When choosing an ADTs implementation, consider
  how frequently particular ADT operations occur in
  a given application.
• However, some seldom-used but critical operations
  must be efficient.
• E.g., an air traffic control system.

16
Keeping Your Perspective (2/2)

• If the problem size is always small, you can
  probably ignore an algorithms efficiency.
• Order-of-magnitude analysis focuses on large
  problems.
• Weigh the trade-offs between an algorithms time
  requirements and its memory requirements.
• Compare algorithms for both style and efficiency.

17
The Efficiency of Searching Algorithms (1/2)

• Sequential search
• Strategy
• Look at each item in the data collection in turn.
• Stop when the desired item is found, or the end
  of the data is reached.
• Efficiency
• Worst case

O(n)
18
The Efficiency of Searching Algorithms (2/2)

• Binary search of a sorted array
• Strategy
• Repeatedly divide the array in half.
• Determine which half could contain the item, and
  discard the other half.
• Efficiency
• Worst case
• For large arrays, the binary search has an
  enormous advantage over a sequential search.
• At most 20 comparisons to search an array of one
  million items.

O(log2n)
19
Sorting Algorithms and Their Efficiency

• Sorting
• A process that organizes a collection of data
  into either ascending or descending order.
• The sort key.
• The part of a data item that we consider when
  sorting a data collection.
• Categories of sorting algorithms.
• An internal sort
• Requires that the collection of data fit entirely
  in the computers main memory.
• An external sort
• The collection of data will not fit in the
  computers main memory all at once, but must
  reside in secondary storage.

20
Selection Sort (1/9)

• Strategy
• Select the largest (or smallest) item and put it
  in its correct place.
• Select the next largest (or next smallest) item
  and put it in its correct place.
• And so on.
• Until you have selected and put n-1 of the n
  items.
• Analogous to card playing.

21
Selection Sort (2/9)

• Shaded elements are selected boldface element
  are in order.

29 37 13
Initial array
29 37 13
Select
swap
29 13 37
Partial sorted array
29 13 37
Select
swap
13 29 37
Sorted array
22
/ Finds the largest item in an array. _at_pre
theArray is an array of size items, size gt 1.
_at_post None. _at_param theArray The given array.
_at_param size The number of elements in
theArray. _at_return The index of the largest
item in the array. The arguments are
unchanged. / int indexOfLargest(const DataType
theArray, int size) int indexSoFar 0
// index of largest item
// found so far for (int currentIndex 1
currentIndex lt size
currentIndex) if
(theArraycurrentIndex gt theArrayindexSoFar)
indexSoFar currentIndex // end
for return indexSoFar // index of largest
item // end indexOfLargest
typedef int DataType
0
1
indexSoFar
currentIndex
size-1
4 9 ...
23
Selection Sort (4/9)
/ Swaps two items. _at_pre x and y are the
items to be swapped. _at_post Contents of actual
locations that x and y represent are swapped.
_at_param x Given data item. _at_param y Given
data item. / void swap(DataType x, DataType
y) DataType temp x x y y
temp // end swap
24
Selection Sort (5/)
/ Sorts the items in an array into ascending
order. _at_pre theArray is an array of n items.
_at_post The array theArray is sorted into
ascending order n is unchanged.
_at_param theArray The array to sort. _at_param n
The size of theArray. / void selectionSort(DataTy
pe theArray, int n) // last index of the
last item in the subarray of // items
yet to be sorted, // largest index of the
largest item found for (int last n-1 last
gt 1 --last) // select largest item in
theArray0..last int largest
indexOfLargest(theArray, last1) // swap
largest item theArraylargest with //
theArraylast swap(theArraylargest,
theArraylast) // end for // end
selectionSort
25
Selection Sort (6/9)

• Analysis
• Sorting in general compares, exchanges, or moves
  items.
• We should count these operations.
• Such operations are more expensive than ones that
  control loops or manipulate array indexes,
  particularly when the data to be sorted are
  complex.
• The for loop in the function selectoinSort
  executes n-1 times.
• indexOfLargest and swap are called n-1 times.

26
Selection Sort (7/9)

• Each call to indexOfLargest causes its loop to
  execute last (or size 1) times.
• Cause the loop to execute a total of
• (n-1) (n-2) 1 times.
• Each execution of the loop performs one
  comparison
• The calls of indexOfLargest require n(n-1)/2
  comparisons.

n(n-1)/2
27
Selection Sort (8/9)

• The n-1 calls to swap require
• 3 (n-1) moves.
• Together, a selection sort of n items requires
• n(n-1)/2 3(n-1)
• n2/2 5n/2 -3 major operations.
• Thus, selection sort is O(n2).

28
Selection Sort (9/9)

• Does not depend on the initial arrangement of the
  data.
• Best case worst case average case O(n2)
• Only appropriate for small n!!

29
Bubble Sort (1/4)

• Strategy
• Compare adjacent elements and exchange them if
  they are out of order.
• Moves the largest (or smallest) elements to the
  end of the array.
• Repeating this process eventually sorts the array
  into ascending (or descending) order.

30
Bubble Sort (2/4)
Pass 1
Pass 2
Initial array
29 10 14 37
Initial array
10 29 14 37
29 10 14 37
10 29 14 37
swap
10 29 14 37
10 29 14 37
swap
swap
10 14 29 37
10 14 29 37
10 29 14 37
31
Bubble Sort (3/)
/ Sorts the items in an array into ascending
order. _at_pre theArray is an array of n items.
_at_post theArray is sorted into ascending order
n is unchanged. _at_param theArray The given
array. _at_param n The size of theArray. / void
bubbleSort(DataType theArray, int n) bool
sorted false // false when swaps occur
for (int pass 1 (pass lt n) !sorted
pass) sorted true // assume
sorted for (int index 0 index lt n-pass
index) int nextIndex index
1 if (theArrayindex gt
theArraynextIndex) // exchange
items swap(theArrayindex,
theArraynextIndex) sorted false
// signal exchange // end if
// end for // end for // end bubbleSort
You can terminate the process if no exchanges
occur during any pass.
32
Bubble Sort (4/4)

• Analysis
• In the worst case, the bubble sort requires at
  most n-1 passes through the array.
• Pass 1 requires n-1 comparisons and at most n-1
  exchanges.
• Pass 2 require n-2 comparisons and at most n-2
  exchanges.
• ...
• Require a total of
• (n-1) (n-2) 1 n(n-1)/2 comparisons.
• (n-1) (n-2) 1 n(n-1)/2 exchanges.
• Each exchange require 3 moves.
• Thus, altogether there are 2 n(n-1) O(n2)
• In the best case require only one pass.
• n-1 comparisons and no exchanges O(n)

33
Insertion Sort (1/4)

• Strategy
• Partition the array into two regions sorted and
  unsorted.
• At each step,
• Take the first item from the unsorted region.
• Insert it into its correct order in the sorted
  region.

insert


sorted
unsorted
34
Insertion Sort (2/4)

• You can omit the first step by considering the
  initial sorted region to be theArray0 and the
  initial unsorted region to be theArray1...n-1.

Initial array
29 14 10 37
Initial array
14 29 10 37
14 29 37
shift
shift
29 10 37
14 29 37
shift
insert
14 29 10 37
insert
10 14 29 37
35
Insertion Sort (3/)
/ Sorts the items in an array into ascending
order. _at_pre theArray is an array of n items.
_at_post theArray is sorted into ascending ordern
is unchanged. _at_param theArray The given
array. _at_param n The size of theArray. / void
insertionSort(DataType theArray, int n)
for (int unsorted 1 unsorted lt n unsorted)
DataType nextItem theArrayunsorted
int loc unsorted for ((loc gt 0)
(theArrayloc-1gt nextItem) --loc)
// shift theArrayloc-1 to the right
theArrayloc theArrayloc-1 // insert
nextItem into Sorted region theArrayloc
nextItem // end for // end insertionSort
nextItem
5
unsorted
... 3 ...
9
5
9
7
7
5
loc
36
Insertion Sort (4/4)

• Analysis
• In the worst case, the outer for loop executes
  n-1 times.
• This loop contains an inner for loop that
  executes at most unsorted times.
• unsorted ranges from 1 to n-1.
• Number of comparisons and moves
• 2 12(n-1) n(n-1).
• The outer loop moves data items twice per
  iteration, or 2(n-1) times.
• Together, there are n(n-1) 2(n-1) n2n-2
  major operations in the worst case, O(n2).
• Prohibitively inefficient for large arrays.

37
Mergesort (1/9)

• A recursive sorting algorithm.
• Strategy
• Divide an array into halves.
• Sort each half (by calling itself recursively).
• Merge the sorted halves into one sorted array.
• Base case an array of one item IS SORTED.

38
Mergesort (2/9)
theArray
8 1 4 3 2
Divide the array in half and conquer
...
sorted array
1 4 8
2 3
Merge the halves

tempArray
1
2
3
4
8
copy
theArray
1 2 3 4 8
39
Mergesort (3/9)
/ Sorts the items in an array into ascending
order. _at_pre theArrayfirst..last is an
array. _at_post theArrayfirst..last is sorted
in ascending order. _at_param theArray The given
array. _at_param first The first element to
consider in theArray. _at_param last The last
element to consider in theArray. / void
mergesort(DataType theArray, int first, int
last) if (first lt last) int mid
(first last)/2 // index of midpoint
mergesort(theArray, first, mid)
mergesort(theArray, mid1, last) // merge
the two halves merge(theArray, first, mid,
last) // end if // end mergesort
40
Mergesort (4/)
const int MAX_SIZE 10000 void merge(DataType
theArray, int first, int mid, int last)
DataType tempArrayMAX_SIZE // temporary
array // initialize the local indexes to
indicate the subarrays int first1 first
// beginning of first subarray int last1
mid // end of first subarray int
first2 mid 1 // beginning of second
subarray int last2 last // end of
second subarray // while both subarrays are
not empty, copy the // smaller item into the
temporary array int index first1 // next
available location in
// tempArray for ( (first1 lt last1)
(first2 lt last2) index) if
(theArrayfirst1 lt theArrayfirst2)
tempArrayindex theArrayfirst1
first1 else
tempArrayindex theArrayfirst2
first2 // end if // end for
first1
last1
first2
last2
... 5 8 3 9 ... ...
anArray
first
mid
last
index
... ... ...
3
5
tempArray
41
Mergesort (5/9)
// finish off the nonempty subarray //
finish off the first subarray, if necessary
for ( first1 lt last1 first1, index)
tempArrayindex theArrayfirst1 //
finish off the second subarray, if necessary
for ( first2 lt last2 first2, index)
tempArrayindex theArrayfirst2 // copy
the result back into the original array for
(index first index lt last index)
theArrayindex tempArrayindex // end
merge
42
Mergesort (6/9)
void mergesort(DataType theArray, int first,
int last) if (first lt last) int mid
(first last)/2 mergesort(theArray,
first, mid) mergesort(theArray, mid1,
last) merge(theArray, first, mid, last)

mergesort(theArray,0,5)
mergesort(theArray,3,5)
mergesort(theArray,0,2)
mergesort(theArray,1,1)
mergesort(theArray,0,1)
43
Mergesort (7/9)

• Analysis (worst case)
• The merge step of the algorithm requires the most
  effort.
• Each merge step merges theArrayfirstmid and
  theArraymid1last.
• If the number of items in the two array segment
  is m
• Merging the segments requires at most
• m-1 comparisons.
• m moves from the original array to the temporary
  array.
• m moves from the temporary array back to the
  original array.
• Each merge requires 3m-1 major operations.

44
Mergesort (8/9)
Level 0
n
Merge n items 3n-1 operations or O(n)
Level 1
n/2
n/2
Merge two n/2 items 2(3n/2-1) operations 3n-2
operations or O(n)
Level 2
n/4
n/4
n/4
n/4
Each level requires O(n) operations
. . .
Level log2n (or 1 log2n rounded down)
1
1
1
1
1
1
Each level O(n) operations O(log2n) levels ?
O(nlog2n)
45
Mergesort (9/9)

• Analysis
• Worst case
• Average case
• Performance is independent of the initial order
  of the array items.
• Advantage
• Mergesort is an extremely fast algorithm.
• Disadvantage
• Mergesort requires a second array as large as the
  original array.

O(n log2n).
O(n log2n).
46
Quicksort (1/15)

• A divide-and-conquer algorithm.
• Strategy
• Choose a pivot.
• Partition the array about the pivot.
• Pivot is now in correct sorted position.
• The items in first pivotIndex-1 remain in
  positions first through pivotIndex-1 when the
  array is properly sorted.
• Sort the left section and Sort the right
  section. (solve small problems)

47
Quicksort (2/15)

• Partition algorithm
• To partition an array segment theArrayfirstlast
  .
• Choose a pivot.
• If the items in the array are arranged randomly,
  you can choose a pivot at random.
• Choose theArrayfirst as the pivot.
• Three regions S1, S2, and unknown.

48
Quicksort (3/15)

• Initially, all items except the pivot
  (theArrayfirst) constitute the unknown region.
• Conditions
• lastS1 first
• firstUnknown first 1
• At each stop, you examine one item of the unknown
  region.
• i.e., theArrayfirstUnknown.
• Determine in which of the S1 or S2 it belongs,
  and place it there.
• Then decrease the size of unknown by 1 (i.e.,
  firstUnknown).

p ?
first lastS1
last
firstUnknown
49
Quicksort (4/15)

• Move theArrayfirstUnknown into S1
• Swap theArrayfirstUnknown with
  theArraylastS11.
• Increment lastS1.
• Increment firstUnknown.

swap
S1
S2
unknown
p
ltp
?p
?p
ltp
ltp
?p
?
first
last
lastS1
firstUnknown
50
Quicksort (5/15)

• Move theArrayfirstUnknown into S2
• Simply increment firstUnknown by 1.

S1
S2
unknown
p
?p
ltp
?p
?
first
last
lastS1
firstUnknown
51
Quicksort (6/15)

• After you have moved all items from unknown
  region into S1 and S2.
• i.e., firstUnknown gt last.
• You have to place the pivot between S1 and S2.
• swap theArraylastS1 with the pivot.
• Then pivot index is lastS1.

swap
S1
S2

ltp
ltp
ltp
?p
p
p
first
last
lastS1
firstUnknown
52
Quicksort (7/15)
/ Chooses a pivot for quicksort's partition
algorithm and swaps it with the first item in
an array. _at_pre theArrayfirst..last is an
array first lt last. _at_post theArrayfirst is
the pivot. _at_param theArray The given array.
_at_param first The first element to consider in
theArray. _at_param last The last element to
consider in theArray. / void choosePivot(DataType
theArray, int first, int last)
53
Quicksort (8/15)
void partition(DataType theArray,
int first, int last, int pivotIndex) //
place pivot in theArrayfirst
choosePivot(theArray, first, last) DataType
pivot theArrayfirst // copy pivot //
initially, everything but pivot is in unknown
int lastS1 first // index of last
item in S1 int firstUnknown first 1 //
index of first item in
// unknown
p ?
first lastS1
last
firstUnknown
54
Quicksort (9/15)
// move one item at a time until unknown
region is empty for ( firstUnknown lt last
firstUnknown) // move item from
unknown to proper region if
(theArrayfirstUnknown lt pivot) // item
from unknown belongs in S1 lastS1
swap(theArrayfirstUnknown,
theArraylastS1) // end if //
else item from unknown belongs in S2 // end
for // place pivot in proper position and
mark its location swap(theArrayfirst,
theArraylastS1) pivotIndex lastS1 //
end partition
55
Quicksort (10/15)
void quicksort(DataType theArray, int first,
int last) int pivotIndex if (first lt
last) // create the partition S1, pivot,
S2 partition(theArray, first, last,
pivotIndex) // sort regions S1 and S2
quicksort(theArray, first, pivotIndex-1)
quicksort(theArray, pivotIndex1, last) //
end if // end quicksort
56
Quicksort (11/15)

• The major effort in the quicksort function occurs
  during the partition step.
• Worst case
• The worst case behavior for quicksort occurs when
  the partition produces one sub-problem with n-1
  elements and one with 0 elements.
• And unbalanced partition arises in each recursive
  call.
• When the array is already sorted and the smallest
  item is chosen as the pivot.
• Because S2 decreases in size by one at each
  recursive call to quicksort. The maximum number
  of recursive calls to quicksort will occur.

57
Quicksort (12/15)

• For an array of n items
• In the worst case, the 1st partition requires n-1
  comparisons to partition the n items in the
  array.
• On the next recursive call to quicksort
• Partition is passed n-1 items.
• So it will require n-2 comparisons to partition
  them.
• Therefore, quicksort requires
• (n-1) (n-2) 1 n(n-1)/2 comparisons.
• O(n2).

58
Quicksort (13/15)

• In the average case, when S1 and S2 contain
  (nearly) the same number of items arranged at
  random, fewer recursive calls to quicksort occur.
• As in mergesort, there are either log2n or
  1log2n levels of recursive calls to quicksort.

59
Quicksort (14/15)

• At level 0
• n 1 comparisons for n items.
• At level 1
• n-3 comparisons for n/2, n/2 1 items.
• n-3 (n/2 1) (n/2 1 1).
• At level m
• 2m calls to quicksort.
• Each (n/2m 1) comparisons.
• Total 2m(n/2m 1) n 2m comparisons.
• Each level requires O(n) comparisons. So the
  average-case behavior of quicksort is O(nlogn).

60
Quicksort (15/15)

• Quicksort is usually extremely fast in practice.
• Even if the worst case occurs, quicksorts
  performance is acceptable for moderately large
  arrays.

61
Radix Sort (1/5)

• The radix sorting algorithm is quite different
  from the others.
• It treats each data element as a character
  string.
• 327 as 327.
• Strategy
• Repeatedly (right-to-left) organizes the data
  into groups according to the ith character in
  each element.

62
Radix Sort (2/5)

• Begin by organizing the data into groups
  according to their rightmost letters.
• The string in each group end with the same
  letter.
• The groups are order by that letter.
• The string within each group retain their
  relative order from the original list of string.

Example You can pad numbers on the left with
zeros, making them all appear to the same length.
63
Radix Sort (3/5)

• Combine the groups into one.
• Next, for new groups as before, but this time use
  the next-to-last digits.

64
Radix Sort (4/5)

• To sort n d-digit numbers, Radix sort requires
• n moves each time it forms groups.
• n moves to combine them into one group.
• Totally, 2 n d moves.
• O(n).

65
Radix Sort (5/5)

• Despite its efficiency, radix sort has some
  difficulties that make it inappropriate as a
  general-purpose sorting algorithm.
• To sort integers, you need to accommodate 10
  groups (0, 1, , 9).
• Each group must be able to hold n strings.
• For large n, this requirement demands substantial
  memory!!

66
A Comparison of Sorting Algorithms
67
Summary

• Order-of-magnitude analysis and Big O notation
  measure an algorithms time requirement as a
  function of the problem size by using a
  growth-rate function.
• Worst-case and average-case analyses.
• Sorting algorithms
• Selection sort, bubble sort, and insertion sort
  are all O(n2) algorithms
• Quicksort and mergesort are two very fast
  recursive sorting algorithms.

68
Homework 8

• Exercises 16 and 18. (due date 12/17)