MS 101: Algorithms

1
MS 101 Algorithms

• Instructor
• Neelima Gupta
• ngupta_at_cs.du.ac.in

2
Table Of Contents

• Introduction to some tools to designing
  algorithms through Sorting
• Iterative
• Divide and Conquer

3
Iterative Algorithms Insertion Sort an example

• x1,x2,........., xi-1,xi,.......,xn
• For I 2 to n
• Insert the ith element xi in the partially
  sorted list x1,x2,........., xi-1.

(at rth position)
4
An Example Insertion Sort
15
8
7
10
12
5
At Iteration 1 key 8
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
5
An Example Insertion Sort
8
15
7
10
12
5
At Iteration 2 key 7
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
6
An Example Insertion Sort
7
8
15
10
12
5
At Iteration 3 key 10
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
7
An Example Insertion Sort
7
8
10
15
12
5
At Iteration 4 key 12
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
8
An Example Insertion Sort
7
8
10
12
15
5
At Iteration 5 key 5
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
9
An Example Insertion Sort
5
7
8
10
12
15
Final Output
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
10
Analysis Insertion Sort
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j -
1 Aj1 key
11
Running Time Analysis
Statement C N
InsertionSort(A, n) for i 2 to n
c1 n key Ai c2 (n-1) j i -
1 c3 (n-1) while (j gt 0) and (Aj gt
key) c4 S(Ti 1) Aj1
Aj c5 S Ti j j - 1 c6 S
Ti Aj1 key c7 (n-1)
where Ti is number of while expression
evaluations for the ith for loop iteration Ci is
the constant time required for 1 execution of the
statement N is the number of times the statement
is executed
12
Total time

• T(n) (c1 c2 c3 c7 )n (c2 c3 c7)
  (c4 c5 c6) Ti c4

n
?
i2
13
Worst Case
Worst case Ti i 1 i.e. Ti
(i 1) n(n-1)/2 hence, T(n) (c1 c2
c3 c4 c7 )n (c2 c3 c4 c7) (c4 c5
c6)n(n-1/2) an2 bn c where, a
1/2 (c4 c5 c6) b -1/2 (c4 c5
c6) (c1 c2 c3 c4 c7 ) c -(c2 c3
c4 c7)
n
n
?
?
i2
i2
14
Best Case
Best case Ti 1 i.e. Ti
1 (n 1) hence, T(n) (c1 c2 c3 c4
c7 )n (c2 c3 c4 c7) (c4 c5
c6)(n-1) an b where, a c1
c2 c3 2c4 c5 c6 c7 b -(c2 c3
2c4 c5 c6 c7 )
n
n
?
?
i2
i2
15
Analysis of Algorithms

• Before we move ahead, let us define the notion of
  analysis of algorithms more formally

16
Input Size

• Time and space complexity
• This is generally a function of the input size
• How we characterize input size depends
• Sorting number of input items
• Multiplication total number of bits
• Graph algorithms number of nodes edges
• Etc

17
Lower Bounds

• Please understand the following statements
  carefully.
• Any algorithm that sorts by removing at most one
  inversion per comparison does at least n(n-1)/2
  comparisons in the worst case.
• Hence Insertion Sort is optimal in this category
  of algorithms.

18
Optimal?

• What do you mean by the term Optimal?
• Answer If an algorithm runs as fast in the worst
  case as it is possible (in the best case), we say
  that the algorithm is optimal. i.e if the worst
  case performance of an algorithm matches the
  lower bound, the algorithm is said to be optimal

19
Inversion -

• Example 4, 2, 3
• No. of pairs 3C2 , out of which
• (4,2) is out of order i.e. inversion
• (2,3) is in order
• (4,3) inversion

20

• In n elements there will be n(n-1)/2 inversions
  in worst case.
• Thus, if an algorithm sorts by removing at most
  one inversion per comparison then it must do at
  least n(n-1)/2 comparisons in the worst case.

21

• Insertion Sort
• x1,x2,...,xk-1, xk,.............,xi-1,xi
• Let xi is inserted after xk-1
• No. of comparisons (i-1) (k 1) 1 i k
  1
• No. of inversions removed (i-1) (k 1) i
  k
• No. of inversions removed/comparison
• (1-k1)/(i-k)
• lt 1

22

• Thus Insertion sort falls in the category of
  comparison algorithms that remove at most one
  inversion per comparison.
• Insertion sort is optimal in this category of
  algorithms .

23
SELECTION SORT

• The algorithm works as follows
• Find the maximum value in the array.
• Swap it with the value in the last position
• Repeat the steps above for the remainder of the
  array.

24
Selection Sort An Example

• For i n to 2
• Select the maximum in a1.......ai.
• b) Swap it with ai.

25

• Selection Sort
• x1,x2,, xk,..........., xn
• Let the maximum is located at xk
• Swap it with xn
• Continue

26

• Selection Sort
• x1,x2,, xk,..........., xn-i1,xn
• Suppose we are at the ith iteration
• Let the maximum is located at xk
• Swap it with xn-i1

27
An Example Selection Sort
5 20 6 4 15 3 10 2
28
An Example Selection Sort
5 2 6 4 15 3 10 20

29
An Example Selection Sort
5 2 6 4 10 3 15 20
30
An Example Selection Sort
5 2 6 4 3 10 15 20
31
An Example Selection Sort
5 2 3 4 6 10 15 20
Thanks to MCA 2012 Chhaya Nathani(9)
32
An Example Selection Sort
4 2 3 5 6 10 15 20
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
33
An Example Selection Sort
3 2 4 5 6 10 15 20
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
34
An Example Selection Sort
2 3 4 5 6 10 15 20
DONE!!!!
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
35
ANALYSISSELECTION SORT

• SelectionSort(A, n)
• for i n to 2 maxi
  for ji-1 to 1
• If ajgtamax
• Maxj
• If(max!i)
• Swap aiamax

36
ANALYSIS SELECTION SORT

• Selecting the largest element requires scanning
  all n elements (this takes n - 1 comparisons)
• and then swapping it into the last position.
  Finding the next largest element requires
  scanning the remaining n - 1 elements and so
  on...
• (n - 1)  (n - 2)  ...  2  1  n(n - 1) / 2
• i.e T(n2) comparisons
• T(n) T(n2)

37

• Selection Sort
• x1,x2,, xk,..........., xn-i1,xn
• Suppose we are at the ith iteration
• Let the maximum is located at xk
• No. of comparisons (n i 1) - 1 n - i
• No. of inversions removed (n i 1) 1
  (k-1) n i k 1
• No. of inversions removed/comparison
• ( n i k 1)/ (n i)
• lt 1 (since k gt 1)

38

• Thus Selection sort also falls in category of
  comparison algorithms that remove at most one
  inversion per comparison.
• Selection sort is also optimal in this category
  of algorithms .

39
Merge Sort(Divide and Conquer Technique)

• Divide the list into nearly equal halves
• Sort each half recursively
• Merge these lists

40
18 9 15 13 20 10 7 5
Divide into 2 Subsequence
18 9 15 13
20 10 7 5
18 9
15 13
20 10
7 5
18
9
15
13
20
10
7
5
41
Next Sort the 2 subsequences and merge them.
42
Sorted Sequence
5 7 9 10 13 15 18 20
Sort merge 2 Subsequence
9 13 15 18
5 7 10 20
9 18
13 15
10 20
5 7
18
9
15
13
20
10
7
5
Initial Subsequence
43
Merging

• Let we have two sorted lists
• A
• B
• Compare a1 with b1 and put smaller one in new
  array C and increase the index of array having
  smaller element.

a1
a2
a3
.
an
b1
b2
b3
.
bm
44

• Let a1 lt b1
• A
• B
• Sorted Array C
• C

a1
a2
a3
.
an
b1
b2
b3
.
bm
a1
45

• Example 1
• A
• B
• Sorted Array C
• C

20
40
60
80
10
30
35
38
45
50
52
10
20
46
Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
Thanks toGaurav Gulzar (MCA -11)
47
Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
Thanks toGaurav Gulzar (MCA -11)
48
Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
40
45
Thanks toGaurav Gulzar (MCA -11)
49
Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
40
45
50
Thanks toGaurav Gulzar (MCA -11)
50
Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
40
45
50
52
Thanks toGaurav Gulzar (MCA -11)
51
Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
40
45
50
52
60
80
Thanks toGaurav Gulzar (MCA -11)
52
Worst Case Analysis

• If no. of elements in first list is n in
  second list is m then
• Total No. of comparisons n-1m
• i.e. O(m n) in worst case and

53
What is the best case?

• Number of Comparisons in the best case
• min(n , m), i.e. O(min(n , m))

54

• Example 2
• A
• B
• Sorted Array C
• C

5
8
10
30
35
38
45
50
52
5
8
55

• Example 2
• A
• B
• Sorted Array C
• C

5
8
10
30
35
38
45
50
52
5
8
10
30
56

• Example 2
• A
• B
• Sorted Array C
• C

5
8
10
30
35
38
45
50
52
5
8
10
30
35
57

• Example 2
• A
• B
• Sorted Array C
• C

5
8
10
30
35
38
45
50
52
5
8
10
30
35
38
58

• Example 2
• A
• B
• Sorted Array C
• C

5
8
10
30
35
38
45
50
52
5
8
10
30
35
38
45
59

• Example 2
• A
• B
• Sorted Array C
• C

5
8
10
30
35
38
45
50
52
5
8
10
30
35
38
45
50
60

• Example 2
• A
• B
• Sorted Array C
• C

5
8
10
30
35
38
45
50
52
5
8
10
30
35
38
45
50
52
61
What is the best case?

• Arrays
• Total number of steps O(mn)copying the rest
  of the elements of the bigger array.
• Linked List
• Total number of steps
• min(n , m), i.e. O(min(n , m)) in best case.

62
Analysis of Merge Sort

• Since size of both the lists to be merged is
  n/2, in either case (arrays or linked list), time
  to merge the two lists is T(n).
• If T(n) no. of comparisons performed on an
  Input of size n then
• T(n) T(n/2) T(n/2) T(n)
• 2T(n/2) cn ? ngt n0
• ? T(n) O(nlogn)

63

• If T(n) no. of comparisons performed on an
  Input of size n then
• T(n) T(n/2) T(n/2) T(n)
• 2T(n/2) cn ? ngt n0
• ? T(n) O(nlogn)

64
Final Points

• Merging is T(m n) in case of an Array
• If we use link list then it is O(min(m , n))
• Time for merging is O(n) and O(n/2) i.e. T(n)

65
Conclusion

• Merge Sort T(nlogn)
• Have we beaten the lower bound?
• No,
• It just means that merge sort does not fall in
  the previous category of algorithms. It removes gt
  1 inversions per comparisons.

66

• What is the
• Worst case
• Best Case
• for Merge Sort?

67
Merge Sort Vs Insertion Sort

• What is the advantage of merge sort?
• What is the advantage of insertion sort?

68
Merge Sort Vs Insertion Sort contd..

• Merge Sort is faster but does not take advantage
  if the list is already partially sorted.
• Insertion Sort takes advantage if the list is
  already partially sorted.

69
Lower Bound

• Any algorithm that sorts by comparison only does
  at least ?(n lg n) comparisons in the worst case.

70
Decision Trees

• Provides an abstraction of comparison sorts. In
  a decision tree, each node represents a
  comparison.
• Insertion sort applied on x1, x2, x3

x1x2
x1ltx2
x1gtx2
x2x3
x1x3
x2gtx3
x3gtx1
x3gtx2
x1gtx3
x2x3
x3x1
x1ltx2ltx3
x3gtx1gtx2
x2gtx3
x3gtx2
x3gtx1
x3ltx1
x2gtx1gtx3
x2gtx3gtx1
x1gtx2gtx3
x1gtx3gtx2
71
Decision Trees

• What is the minimum number of leaves in a
  decision tree?
• Longest path of the tree gives us the height of
  the tree, and actually represents the worst case
  scenario for the algorithm.
• height of a decision tree O (n log n)
• i.e. any comparison sort will perform at least
  (n logn) comparisons in the worst case.

72

• Proof
• Let h denotes the height of the tree.
• Whats the maximum of leaves of a binary tree
  of height h? 2h
• 2h gt number of leaves gt n!
• (where n no. of elements and n! is a lower
  bound on the no. of leaves in the decision tree)
• gt hgt log(n!)
• gt n log n ( By Stirlings Approximation)
• ( SA n! v (2.p.n) .(n/e)n
• gt (n/e)n
• Thus, log(n!) gt n log n n log e gt n logn. )

73
Merge Sort is Optimal

• Thus the time to comparison sort n elements is
  ?(n lg n)
• Corollary Merge-sort is asymptotically optimal
  comparison sorts.
• Later well see another sorting algorithm in this
  category namely heap-sort that is also optimal.
• Well also see some algorithms which beat this
  bound. Needless to say those algorithms are not
  purely based on comparisons. They do something
  extra.

74
Quick Sort(Divide and Conquer Technique)

• Pick a pivot element x
• Partition the array into two subarrays around the
  pivot x such that elements in left subarray is
  less than equal to x and element in right
  subarray is greater than x
• Recursively sort left subarray and right subarray

x
  x
gtx
75
Quick Sort (Algorithm)

• QUICKSORT(A, p, q)
• if p lt q
• kPARTITION(A, p, q)
• QUICKSORT(A, p, k-1)
• QUICKSORT(A, k1, q)

76
Quick Sort (Example)

• Let we have following elements in our array -
• i j
• i j
• i j

27 14 9 22 8 41 56 31 15 53 99 11 30 24
14 27 9 22 8 41 56 31 15 53 99 11 30 24
14 9 27 22 8 41 56 31 15 53 99 11 30 24
77
Quick Sort (Example)

• i j
• i j
• i j

14 9 22 27 8 41 56 31 15 53 99 11 30 24
14 9 22 8 27 41 56 31 15 53 99 11 30 24
14 9 22 8 27 24 56 31 15 53 99 11 30 41
78
Quick Sort (Example)

• i j
• i j
• i j

14 9 22 8 24 27 56 31 15 53 99 11 30 41
14 9 22 8 24 27 30 31 15 53 99 11 56 41
14 9 22 8 24 27 11 31 15 53 99 30 56 41
79
Quick Sort (Example)

• i j
• i j
• i j

14 9 22 8 24 11 27 31 15 53 99 30 56 41
14 9 22 8 24 11 27 99 15 53 31 30 56 41
14 9 22 8 24 11 27 53 15 99 31 30 56 41
80
Quick Sort (Example)

• i j
• i j (stop)
• (recursive call on left array) (recursive call
  on right array)

14 9 22 8 24 11 27 15 53 99 31 30 56 41
14 9 22 8 24 11 15 27 53 99 31 30 56 41
14 9 22 8 24 11 15 27 53 99 31 30 56 41
81

• i j i j
• i j i j
• i j i j
• i j i
  j
• i j
  i j
• i j i j (stop)
• i j (st op)

14 9 22 8 24 11 15
53 99 31 30 56 41
27
9 14 22 8 24 11 15
27
53 41 31 30 56 99
9 14 15 8 24 11 22
27
41 53 31 30 56 99
9 14 11 8 24 15 22
27
41 31 53 30 56 99
9 11 14 8 24 15 22
27
41 31 30 53 56 99
9 11 8 14 24 15 22
27
41 31 30 53 56 99
9 11 8 14 24 15 22
41 31 30 53 56 99
27
82
9 11 8 14 24 15 22 27 41 31 30 53 56 99
i j i j i j
i j
9 8 11 14 15 24 22 27 31 41 30 53 56 99
i j i j i
j i j (stop)
8 9 11 14 15 22 24 27 31 30 41 53 56 99
(stop) i j i j (stop) (stop) i j
i j (stop)
8 9 11 14 15 22 24 27 31 30 41 53 56 99
Sorted Array
83
Analyzing Quicksort

• Worst Case?
• Partition is always unbalanced
• Worst Case input?
• Already-sorted input, if the first element is
  always picked as the pivot
• Best case?
• Partition is perfectly balanced
• Best Case input?
• ?
• Worst Case and Best Case input when the middle
  element is always picked as the pivot?

84
Worst Case of Quicksort

• In the worst case
• T(1) ?(1)
• T(n) T(n - 1) ?(n)
• Does the recursion look familiar?
• T(n) ?(n2)

85
Best Case of Quicksort

• In the best case
• T(n) 2T(n/2) ?(n)
• Does the recursion familiar?
• T(n) ?(n lg n)

86
Why does Qsort works well in practice?

• Suppose that partition() always produces a 9-to-1
  split. This looks quite unbalanced!
• The recurrence is
• T(n) T(9n/10) T(n/10) n
• T(n) ? (n log n)
• Such an imbalanced partition and ?(n log n) time?

87
Why does Qsort works well in practice?

• Intuitively, a real-life run of quicksort will
  produce a mix of bad and good splits
• Pretend for intuition that they alternate between
  best-case (n/2 n/2) and worst-case (n-1 1)
• What happens if we bad-split root node, then
  good-split the resulting size (n-1) node?

88
Why does Qsort works well in practice?

• Intuitively, a real-life run of quicksort will
  produce a mix of bad and good splits
• Pretend for intuition that they alternate between
  best-case (n/2 n/2) and worst-case (n-1 1)
• What happens if we bad-split root node, then
  good-split the resulting size (n-1) node?
• We end up with three subarrays, size 1, (n-1)/2,
  (n-1)/2
• Combined cost of splits n n -1 2n -1 O(n)
• No worse than if we had good-split the root node!

89
Why does Qsort works well in practice?

• Intuitively, the O(n) cost of a bad split (or 2
  or 3 bad splits) can be absorbed into the O(n)
  cost of each good split
• Thus running time of alternating bad and good
  splits is still O(n lg n), with slightly higher
  constants
• How can we be more rigorous? well do average
  analysis of Qsort later while doing randomized
  algorithms.

90
Quicksort Vs Merge Sort

• Merge Sort takes O(n lg n) in the worst case
• Quick Sort takes O(n2) in the worst case
• So why would anybody use Qsort instead of merge
  sort?
• Because in practice, Qsort is quick as the worst
  case doesnt happen often.

91
Up Next

• Linear-Time Sorting Algorithms

92

• The End