2IL05 Data Structures 2IL06 Introduction to Algorithms

1
2IL05 Data Structures 2IL06 Introduction to
Algorithms

• Spring 2009Lecture 4 Sorting in linear time

2
Building a heap
One more time
3
Building a heap

• Build-Max-Heap(A)
• heap-size ? lengthA
• for i ? lengthA downto 1
• do Max-Heapify(A,i)

4
Building a heap

• Build-Max-Heap2(A)
• heap-sizeA ? 1
• for i ? 2 to lengthA do Max-Heap-Insert(A,
  Ai)
• Max-Heap-Insert(A, key)
• heap-sizeA ? heap-sizeA 1
• Aheap-sizeA ? - infinity
• Increase-Key(A, heap-sizeA, key)
• Lower bound worst case running time?

14
3
8
11
2
24
35
28
16
5
20
21
Ai moves up until it reaches the correct
position
5
Building a heap

• Build-Max-Heap2(A)
• heap-sizeA ? 1
• for i ? 2 to lengthA do Max-Heap-Insert(A,
  Ai)
• Running time T(1) ?2i n (time for
  Max-Heap-Insert(A, Ai))
• Max-Heap-Insert (A, Ai ) takes O(log i) O(log
  n) time
• ? worst case running time is O(n log n)
• If A is sorted in increasing order, then Ai is
  always the largest element when
  Max-Heap-Insert(A, Ai)) is called and must move
  all the way up the tree
• ? Max-Heap-Insert (A, Ai) takes O(log i) time.
• Worst case running time T(1) ?2i n
  O(log i) O(1 ?2i n log i)
• O (n log n)
• since ?2i n log i ?n/2i n log (n/2)
  n/2 log (n/2)

6
Quiz

• log2 n T(log n2) ?
• vn O(log4 n) ?
• 2lg n O(n2) ?
• 2n O(n2) ?
• log(vn) T(log n) ?

• no
• yes
• no
• yes
• yes

7
Sorting in linear time
8
The sorting problem

• Input a sequence of n numbers a1, a2, , an
• Output a permutation of the input such that ai1
  ain
• Why do we care so much about sorting?
• sorting is used by many applications
• (first) step of many algorithms
• many techniques can be illustrated by studying
  sorting

9
Can we sort faster than T(n log n) ??

• Worst case running time of sorting algorithms
• InsertionSort T(n2)
• MergeSort T(n log n)
• HeapSort T(n log n)
• Can we do this faster? T(n loglog n) ?
  T(n) ?

10
Upper and lower bounds

• Upper bound
• How do you show that a problem (for example
  sorting) can be solved in T(f(n)) time?
• ? give an algorithm that solves the problem in
  T(f(n)) time.
• Lower bound
• How do you show that a problem (for example
  sorting) can not be solved faster than in T(f(n))
  time?
• ? prove that every possible algorithm that
  solves the problem needs O(f(n)) time.

11
Lower bounds

• Lower bound
• How do you show that a problem (for example
  sorting) can not be solved faster than in T(f(n))
  time?
• ? prove that every possible algorithm that
  solves the problem needs O(f(n)) time.

Model of computation which operations is the
algorithm allowed to use?
Bit-manipulations?Random-access (array
indexing) vs. pointer-machines?
12
Comparison-based sorting

• InsertionSort(A)
• for j ? 2 to lengthA
• do begin
• 6.
• 7. end
• Which steps precisely the algorithm executes
  and hence, which element ends up where only
  depends on the result of comparisons between the
  input elements.

key ? A j i ? j -1 while i gt 0 and A i
gt key do begin A i1 ? A i i ?
i -1 end A i 1 ? key
13
Decision tree for comparison-based sorting
exchange of elements, assignments, etc.
A. lt A.
A. lt A.
A. lt A.
A. lt A.
A. lt A.
A. lt A.
A. lt A.
14
Comparison-based sorting

• every permutation of the input follows a
  different path in the decision tree
• ? the decision tree has at least n! leaves
• the height of a binary tree with n! leaves is at
  last log(n!)
• worst case running time
• longest path from root to leaf
• log(n!) O(n log n)

15
Lower bound for comparison-based sorting

• TheoremAny comparison-based sorting algorithm
  requires O(n log n) comparisons in the worst
  case.
• ? The worst case running time of MergeSort and
  HeapSort is optimal.

16
Sorting in linear time

• Three algorithms which are faster
• CountingSort
• RadixSort
• BucketSort
• (not comparison-based, make assumptions on the
  input)

17
CountingSort

• Input array A1..n of numbers
• Assumption the input elements are integers in
  the range 0 to k, for some k
• Main idea count for every Ai the number of
  elements less than Ai ? position of Ai
  in the output array
• Beware of elements that have the same
  value!
• position(i) number of elements less than Ai
  in A1..n
• number of elements equal
  to Ai in A1..i

18
CountingSort

• position(i) number of elements less than Ai
  in A1..n
• number of elements equal
  to Ai in A1..i

5
3
10
5
4
5
7
7
9
3
10
8
5
3
3
8
3
3
3
3
4
5
5
5
5
7
7
8
8
9
10
10
numbers lt 5
third 5 from left position ( less than 5) 3
19
CountingSort

• position(i) number of elements less than Ai
  in A1..n
• number of elements equal
  to Ai in A1..i
• LemmaIf every element Ai is placed on
  position(i), then the array is sorted and the
  sorted order is stable.

Numbers with the same value appear in the same
order in the output array as they do in the input
array.
20
CountingSort
Ci will contain the number of elements i

• CountingSort(A,k)
• ? Input array A1..n of integers in the range
  0..k
• ? Output array B1..n which contains the
  elements of A, sorted
• for i ? 0 to k do Ci ? 0
• for j ? 1 to lengthA do CAj ? CAj
  1
• ? Ci now contains the number of elements equal
  to i
• for i ? 1 to k do Ci ? Ci Ci-1
• ? Ci now contains the number of elements less
  than or equal to i
• for j ? lengthA downto 1
• do BCA j ? Aj CA j
  ? CA j 1

21
CountingSort

• CountingSort(A,k)
• ? Input array A1..n of integers in the range
  0..k
• ? Output array B1..n which contains the
  elements of A, sorted
• for i ? 0 to k do Ci ? 0
• for j ? 1 to lengthA do CAj ? CAj
  1
• ? Ci now contains the number of elements equal
  to i
• for i ? 1 to k do Ci ? Ci Ci-1
• ? Ci now contains the number of elements less
  than or equal to i
• for j ? lengthA downto 1
• do BCA j ? Aj CA j
  ? CA j 1
• Correctness lines 6/7 Invariant
• Inv(m) for m i n Bposition(i) contains
  Ai
• for 0 i k Ci ( numbers
  smaller than i )
• (
  numbers equal to i in A1..m-1)
• Inv(m1) holds before loop is executed with j m,
  Inv(m) holds afterwards

22
CountingSort running time

• CountingSort(A,k)
• ? Input array A1..n of integers in the range
  0..k
• ? Output array B1..n which contains the
  elements of A, sorted
• for i ? 0 to k do Ci ? 0
• for j ? 1 to lengthA do CAj ? CAj
  1
• ? Ci now contains the number of elements equal
  to i
• for i ? 1 to k do Ci ? Ci Ci-1
• ? Ci now contains the number of elements less
  than or equal to i
• for j ? lengthA downto 1
• do BCA j ? Aj CA j
  ? CA j 1
• line 1 ?0ik T(1) T(k)
• line 2 ?1in T(1) T(n)
• line 4 ?0ik T(1) T(k)
• lines 6/7 ?1in T(1) T(n)

Total T(nk) ? T(n) if k O(n)
23
CountingSort

• TheoremCountingSort is a stable sorting
  algorithm that sorts an array of n integers in
  the range 0..k in T(nk) time.

24
RadixSort

• Input array A1..n of numbers
• Assumption the input elements are integers with
  d digits
• example (d 4) 3288, 1193, 9999, 0654,
  7243, 4321
• RadixSort(A, d)
• for i ? 1 to d
• do use a stable sort to sort array A on
  digit i

1st digit
dth digit
25
RadixSort example
329 457 657 839 436 720 355
sort on 1st digit
sort on 2nd digit
sort on 3rd digit

• Correctness Assignment 4

26
RadixSort

• Running time If we use CountingSort as stable
  sorting algorithm
• ? T(n k) per digit
• TheoremGiven n d-digit numbers in which each
  digit can take up to k possible values, RadixSort
  correctly sorts these numbers in T(d (n k))
  time.

each digit is an integer in the range 0..k
27
BucketSort

• Input array A1..n of numbers
• Assumption the input elements lie in the
  interval 0..1) (no integers!)
• BucketSort is fast if the elements are uniformly
  distributed in 0..1)

28
BucketSort

• Throw input elements in buckets, sort buckets,
  concatenate

input array A1..n numbers in 0..1)
auxiliary array B0..n-1 bucket Bi contains
numbers in i/n (i1)/n
29
BucketSort

• Throw input elements in buckets, sort buckets,
  concatenate

input array A1..n numbers in 0..1)
auxiliary array B0..n-1 bucket Bi contains
numbers in i/n (i1)/n
0
1
0.1
0.15
0.13
0.287
0.256
0.734
0.792
n-1
30
BucketSort

• BucketSort(A)
• ? Input array A1..n of numbers with 0 Ai
  lt 1
• ? Output sorted list, which contains the
  elements of A
• n ? lengthA
• initialize auxiliary array B0..n-1 each Bi
  is a linked list of numbers
• for i ? 1 to n
• do insert Ai into list B nAi
• for i ? 0 to n-1
• do sort list Bi, for example with
  InsertionSort
• concatenate the lists B0, B1, , Bn-1
  together in order

31
BucketSort

• Running time?
• Define ni number of elements in bucket Bi
• ? running time T(n) ?0in-1 T(ni2)
• worst case
• best case
• expected running time if the numbers are
  randomly distributed ?

all numbers fall into the same bucket ? T(n2)
all numbers fall into different buckets ? T(n)
32
BucketSort expected running time

• Define ni number of elements in bucket Bi
• ? running time T(n) ?0in-1 T(ni2)
• Assumption Pr Aj falls in bucket Bi
  1/n for each i
• E running time E T(n) ?0in-1
  T(ni2)
• T ( n ?0in-1 E
  ni2 )
• What is E ni2 ?
• (some math with indicator random variables see
  book for details)
• ? E ni2 2 - 1/n T(1)
• ? expected running time T(n)

but E ni2 ? E ni 2
We have E ni 1
33
Tutorials this week

• Small tutorials on Tuesday 34.
• Wednesday 78 big tutorial.
• Small tutorial Friday 78.