CS 3343: Analysis of Algorithms

1
CS 3343 Analysis of Algorithms

• Lectures 11-12 Heap, Heap Sort

2
Outline

• Exam 1 statistics
• Review of quick sort
• Heap
• Heap sort

3
Exam1 total
4
Problems 1 - 4
5
Problems 5 - 7
6
Overall score (hw exam1)
Hw 10 Exam1 15 Overall 25 A gt 20 (22) B gt
17 (20) C gt 14 (17) D gt 12 (15) F lt 11 (14)
7
Quick sort

• Quicksort an n-element array
• Divide Partition the array into two subarrays
  around a pivot x such that elements in lower
  subarray x elements in upper subarray.
• Conquer Recursively sort the two subarrays.
• Combine Trivial.

Key Linear-time partitioning subroutine.
8
Pseudocode for quicksort
QUICKSORT(A, p, r) if p lt r then q ? PARTITION(A,
p, r) QUICKSORT(A, p, q1) QUICKSORT(A, q1, r)
Initial call QUICKSORT(A, 1, n)
9
Partition Code

• Partition(A, p, r)
• x Ap // pivot is the first element
• i p
• j r 1
• while (TRUE)
• repeat
• i
• until Ai gt x
• repeat
• j--
• until Aj lt x
• if (i lt j)
• Swap (Ai, Aj)
• else
• break
• swap (Ap, Aj)
• return j

Scan
partition() runs in O(n) time
10
Partition
6
10
5
8
13
3
2
11
2
3
8
10
3
6
11
Worst-case of quicksort

• Input sorted or reverse sorted.
• Partition around min or max element.
• One side of partition always has no elements.

(arithmetic series)
12
Worst-case recursion tree
T(n) T(0) T(n1) n
n
n
(n1)
Q(1)
Q(1)
(n2)
T(n) Q(n) Q(n2) Q(n2)
Q(1)
Q(1)
13
Best-case analysis
(For intuition only!)
If were lucky, PARTITION splits the array evenly
T(n) 2T(n/2) Q(n) Q(n log n)
(same as merge sort)
What is the solution to this recurrence?
14
Analysis of almost-best case
15
Analysis of almost-best case
n
16
Analysis of almost-best case
n
17
Analysis of almost-best case
n


O(n) leaves
Q(1)
Q(1)
18
Analysis of almost-best case



O(n) leaves
Q(1)
Q(1)
Q(n log n)
19
Quicksort Runtimes

• Best case runtime Tbest(n) ? O(n log n)
• Worst case runtime Tworst(n) ? O(n2)
• Average runtime Tavg(n) ? O(n log n )
• Expected runtime of randomized quicksort is O(n
  log n)
• Typically twice faster than merge sort (why?)

20
Randomized quicksort

• Randomly choose an element as pivot
• Every time need to do a partition, throw a die to
  decide which element to use as the pivot
• Each element has 1/n probability to be selected

Partition_random(A, p, r) d random() //
a random number between 0 and 1 index p
floor((r-p1) d) // pltindexltr
swap(Ap, Aindex) return partition(A, p,
r)
21
Running time of randomized quicksort
T(0) T(n1) dn if 0 n1 split, T(1)
T(n2) dn if 1 n2 split, M T(n1) T(0)
dn if n1 0 split,
T(n)

• The expected running time is an average of all
  cases

Expectation
22
Expected running time of Quicksort

• Guess
• So we have to prove
  for some c and sufficiently large n
• Use T(n) instead of for convenience

23

• Fact
• Need to Prove T(n) c n log (n)
• Assumption T(k) ck log (k) for 0 k n-1
• Proof by substitution

If c 4
24

• Heap sort

25
Heap sort

• Another T(n log n) sorting algorithm
• In practice quick sort wins
• The heap data structure and its variants are very
  useful for many algorithms

26
Selection sort
lt
Sorted
lt
Find minimum
lt
Sorted
27
Selection sort
gt
Sorted
gt
Find maximum
gt
Sorted
28
Selection sort

• SelectionSort(A1..n)
• for (i n i gt 0 i--)
• index max_element(A1..i)
• swap(Ai, Aindex)
• end

Whats the time complexity?
If max_element takes T(n), selection sort takes
?i1n i T(n2)
29
Heap

• A heap is a data structure that allows you to
  quickly retrieve the largest (or smallest)
  element
• It takes time T(n) to build the heap
• If you need to retrieve largest element, second
  largest, third largest, in long run the time
  taken for building heaps will be rewarded

30
Idea of heap sort

• HeapSort(A1..n)
• Build a heap from A
• For i n down to 1
• Retrieve largest element from heap
• Put element at end of A
• Reduce heap size by one
• end

• Key
• Build a heap in linear time
• Retrieve largest element (and make it ready for
  next retrieval) in O(log n) time

31
Heaps

• A heap can be seen as a complete binary tree
• What makes a binary tree complete?
• Is the example above complete?

Perfect binary tree
32
Heaps

• In practice, heaps are usually implemented as
  arrays

16
14
10
8
7
9
3
2
4
1
33
Heaps

• To represent a complete binary tree as an array
• The root node is A1
• Node i is Ai
• The parent of node i is Ai/2 (note integer
  divide)
• The left child of node i is A2i
• The right child of node i is A2i 1

16
14
10
8
7
9
3
2
4
1
A

34
Referencing Heap Elements

• So
• Parent(i)
• return ?i/2?
• Left(i)
• return 2i
• right(i)
• return 2i 1

35
Heap Height

• Definitions
• The height of a node in the tree the number of
  edges on the longest downward path to a leaf
• The height of a tree the height of its root
• What is the height of an n-element heap? Why?
• ?log2(n)?. Basic heap operations take at most
  time proportional to the height of the heap

h3
h1
h2
h0
h1
h0
36
The Heap Property

• Heaps also satisfy the heap property
• AParent(i) ? Ai for all nodes i gt 1
• In other words, the value of a node is at most
  the value of its parent
• The value of a node should be greater than or
  equal to both its left and right children
• And all of its descendents
• Where is the largest element in a heap stored?

37
Are they heaps?
16
4
10
14
7
9
3
2
8
1

16
10
14
7
8
9
3
2
4
1

Violation to heap property a node has value less
than one of its children How to find that? How to
resolve that?
38
Heap Operations Heapify()

• Heapify() maintain the heap property
• Given a node i in the heap with children l and r
• Given two subtrees rooted at l and r, assumed to
  be heaps
• Problem The subtree rooted at i may violate the
  heap property
• Action let the value of the parent node sift
  down so subtree at i satisfies the heap property
  
• Fix up the relationship between i, l, and r
  recursively

39
Heap Operations Heapify()

• Heapify(A, i)
• // precondition subtrees rooted at l and r are
  heaps
• l Left(i) r Right(i)
• if (l lt heap_size(A) Al gt Ai)
• largest l
• else
• largest i
• if (r lt heap_size(A) Ar gt Alargest)
• largest r
• if (largest ! i)
• Swap(A, i, largest)
• Heapify(A, largest)
• // postcondition subtree rooted at i is a heap

Among Al, Ai, Ar, which one is largest?
If violation, fix it.
40
Heapify() Example
16
4
10
14
7
9
3
2
8
1
16
4
10
14
7
9
3
2
8
1
A
41
Heapify() Example
16
4
10
14
7
9
3
2
8
1
16
10
14
7
9
3
2
8
1
A
4
42
Heapify() Example
16
4
10
14
7
9
3
2
8
1
16
10
7
9
3
2
8
1
A
4
14
43
Heapify() Example
16
14
10
4
7
9
3
2
8
1
16
14
10
7
9
3
2
8
1
A
4
44
Heapify() Example
16
14
10
4
7
9
3
2
8
1
16
14
10
7
9
3
2
1
A
4
8
45
Heapify() Example
16
14
10
8
7
9
3
2
4
1
16
14
10
8
7
9
3
2
1
A
4
46
Heapify() Example
16
14
10
8
7
9
3
2
4
1
16
14
10
8
7
9
3
2
4
1
A
47
Analyzing Heapify() Informal

• Aside from the recursive call, what is the
  running time of Heapify()?
• How many times can Heapify() recursively call
  itself?
• What is the worst-case running time of Heapify()
  on a heap of size n?

48
Analyzing Heapify() Formal

• Fixing up relationships between i, l, and r takes
  ?(1) time
• If the heap at i has n elements, how many
  elements can the subtrees at l or r have?
• Draw it
• Answer 2n/3 (worst case bottom row 1/2 full)
• So time taken by Heapify() is given by
• T(n) ? T(2n/3) ?(1)

49
Analyzing Heapify() Formal

• So we have
• T(n) ? T(2n/3) ?(1)
• By case 2 of the Master Theorem,
• T(n) O(lg n)
• Thus, Heapify() takes logarithmic time

50
Heap Operations BuildHeap()

• We can build a heap in a bottom-up manner by
  running Heapify() on successive subarrays
• Fact for array of length n, all elements in
  range A?n/2? 1 .. n are heaps (Why?)
• So
• Walk backwards through the array from n/2 to 1,
  calling Heapify() on each node.
• Order of processing guarantees that the children
  of node i are heaps when i is processed

51
BuildHeap()

• // given an unsorted array A, make A a heap
• BuildHeap(A)
• heap_size(A) length(A)
• for (i ?lengthA/2? downto 1)
• Heapify(A, i)

52
BuildHeap() Example

• Work through exampleA 4, 1, 3, 2, 16, 9, 10,
  14, 8, 7

4
1
3
2
16
9
10
14
8
7
4
1
3
2
16
9
10
14
8
7
A
53
4
1
3
2
16
9
10
14
8
7
4
1
3
2
16
9
10
14
8
7
A
54
4
1
3
2
16
9
10
14
8
7
4
1
3
2
16
9
10
14
8
7
A
55
4
1
3
14
16
9
10
2
8
7
4
1
3
14
16
9
10
2
8
7
A
56
4
1
3
14
16
9
10
2
8
7
4
1
3
14
16
9
10
2
8
7
A
57
4
1
10
14
16
9
3
2
8
7
4
1
10
14
16
9
3
2
8
7
A
58
4
1
10
14
16
9
3
2
8
7
4
1
10
14
16
9
3
2
8
7
A
59
4
16
10
14
1
9
3
2
8
7
4
16
10
14
1
9
3
2
8
7
A
60
4
16
10
14
7
9
3
2
8
1
4
16
10
14
7
9
3
2
8
1
A
61
4
16
10
14
7
9
3
2
8
1
4
16
10
14
7
9
3
2
8
1
A
62
16
4
10
14
7
9
3
2
8
1
16
4
10
14
7
9
3
2
8
1
A
63
16
14
10
4
7
9
3
2
8
1
16
14
10
4
7
9
3
2
8
1
A
64
16
14
10
8
7
9
3
2
4
1
16
14
10
8
7
9
3
2
4
1
A
65
Analyzing BuildHeap()

• Each call to Heapify() takes O(lg n) time
• There are O(n) such calls (specifically, ?n/2?)
• Thus the running time is O(n lg n)
• Is this a correct asymptotic upper bound?
• Is this an asymptotically tight bound?
• A tighter bound is O(n)
• How can this be? Is there a flaw in the above
  reasoning?

66
Analyzing BuildHeap() Tight

• To Heapify() a subtree takes O(h) time where h is
  the height of the subtree
• h O(lg m), m nodes in subtree
• The height of most subtrees is small
• Fact an n-element heap has at most ?n/2h1?
  nodes of height h
• CLR 6.3 uses this fact to prove that BuildHeap()
  takes O(n) time

67
Heapsort

• Given BuildHeap(), an in-place sorting algorithm
  is easily constructed
• Maximum element is at A1
• Discard by swapping with element at An
• Decrement heap_sizeA
• An now contains correct value
• Restore heap property at A1 by calling
  Heapify()
• Repeat, always swapping A1 for Aheap_size(A)

68
Heapsort

• Heapsort(A)
• BuildHeap(A)
• for (i length(A) downto 2)
• Swap(A1, Ai)
• heap_size(A) - 1
• Heapify(A, 1)

69
Heapsort Example

• Work through exampleA 4, 1, 3, 2, 16, 9, 10,
  14, 8, 7

4
1
3
2
16
9
10
14
8
7
4
1
3
2
16
9
10
14
8
7
A
70
Heapsort Example

• First build a heap

16
14
10
8
7
9
3
2
4
1
16
14
10
8
7
9
3
2
4
1
A
71
Heapsort Example

• Swap last and first

1
14
10
8
7
9
3
2
4
16
1
14
10
8
7
9
3
2
4
16
A
72
Heapsort Example

• Last element sorted

1
14
10
8
7
9
3
2
4
16
1
14
10
8
7
9
3
2
4
16
A
73
Heapsort Example

• Restore heap on remaining unsorted elements

14
8
10
4
7
9
3
2
1
16
Heapify
14
8
10
4
7
9
3
2
1
16
A
74
Heapsort Example

• Repeat swap new last and first

1
8
10
4
7
9
3
2
14
16
1
8
10
4
7
9
3
2
14
16
A
75
Heapsort Example

• Restore heap

10
8
9
4
7
1
3
2
14
16
10
8
9
4
7
1
3
2
14
16
A
76
Heapsort Example

• Repeat

9
8
3
4
7
1
2
10
14
16
9
8
3
4
7
1
2
10
14
16
A
77
Heapsort Example

• Repeat

8
7
3
4
2
1
9
10
14
16
8
7
3
4
2
1
9
10
14
16
A
78
Heapsort Example

• Repeat

1
2
3
4
7
8
9
10
14
16
1
2
3
4
7
8
9
10
14
16
A
79
Analyzing Heapsort

• The call to BuildHeap() takes O(n) time
• Each of the n - 1 calls to Heapify() takes O(lg
  n) time
• Thus the total time taken by HeapSort() O(n)
  (n - 1) O(lg n) O(n) O(n lg n) O(n lg n)

80
Priority Queues

• Heapsort is a nice algorithm, but in practice
  Quicksort usually wins
• The heap data structure is incredibly useful for
  implementing priority queues
• A data structure for maintaining a set S of
  elements, each with an associated value or key
• Supports the operations Insert(), Maximum(),
  ExtractMax(), changeKey()
• What might a priority queue be useful for?

81
Your personal travel destination list

• You have a list of places that you want to visit,
  each with a preference score
• Always visit the place with highest score
• Remove a place after visiting it
• You frequently add more destinations
• You may change score for a place when you have
  more information
• Whats the best data structure?

82
Priority Queue Operations

• Insert(S, x) inserts the element x into set S
• Maximum(S) returns the element of S with the
  maximum key
• ExtractMax(S) removes and returns the element of
  S with the maximum key
• ChangeKey(S, i, key) changes the key for element
  i to something else
• How could we implement these operations using a
  heap?

83
Implementing Priority Queues

• HeapMaximum(A)
• return A1

84
Implementing Priority Queues

• HeapExtractMax(A)
• if (heap_sizeA lt 1) error
• max A1
• A1 Aheap_sizeA
• heap_sizeA --
• Heapify(A, 1)
• return max

85
HeapExtractMax Example
16
14
10
8
7
9
3
2
4
1
16
14
10
8
7
9
3
2
4
1
A
86
HeapExtractMax Example

• Swap first and last, then remove last

1
14
10
8
7
9
3
2
4
16
14
10
8
7
9
3
2
4
16
A
1
87
HeapExtractMax Example

• Heapify

14
8
10
4
7
9
3
2
1
16
10
7
9
3
2
16
A
14
8
4
1
88
Implementing Priority Queues

• HeapChangeKey(A, i, key)
• if (key Ai) // decrease key
• Ai key
• heapify(A, i)
• else // increase key
• Ai key
• while (igt1 Aparent(i)ltAi)
• swap(Ai, Aparent(i)

Sift down
Bubble up
89
HeapChangeKey Example

• Increase key

16
14
10
8
7
9
3
2
4
1
16
14
10
8
7
9
3
2
4
1
A
90
HeapChangeKey Example

• Increase key

16
14
10
15
7
9
3
2
4
1
16
14
10
7
9
3
2
4
1
A
15
91
HeapChangeKey Example

• Increase key

16
15
10
14
7
9
3
2
4
1
16
10
7
9
3
2
4
1
A
14
15
92
Implementing Priority Queues

• HeapInsert(A, key)
• heap_sizeA
• i heap_sizeA
• Ai -8
• HeapChangeKey(A, i, key)

93
HeapInsert Example

• HeapInsert(A, 17)

16
14
10
8
7
9
3
2
4
1
16
14
10
8
7
9
3
2
4
1
A
94
HeapInsert Example

• HeapInsert(A, 17)

16
14
10
8
7
9
3
2
4
1
-8
-8 makes it a valid heap
16
14
10
8
7
9
3
2
4
1
A
-8
95
HeapInsert Example

• HeapInsert(A, 17)

16
14
10
8
7
9
3
2
4
1
17
Now call changeKey
16
10
8
9
3
2
4
1
A
17
14
7
96
HeapInsert Example

• HeapInsert(A, 17)

17
16
10
8
14
9
3
2
4
1
7
17
10
8
9
3
2
4
1
A
7
16
14
97

• Heapify T(log n)
• BuildHeap T(n)
• HeapSort T(nlog n)
• HeapMaximum T(1)
• HeapExtractMax T(log n)
• HeapChangeKey T(log n)
• HeapInsert T(log n)

98
If we use a sorted array

• Sort T(n log n)
• Afterwards
• arrayMaximum T(1)
• arrayExtractMax T(n)
• arrayChangeKey T(n)
• arrayInsert T(n)

99
If we use a linked list

• Sort T(n log n)
• Afterwards
• listMaximum T(1)
• listExtractMax T(1)
• listChangeKey T(n)
• listInsert T(n)