Advanced Algorithm Design and Analysis

1
Advanced Algorithm Design and Analysis
Jiaheng Lu Renmin University of China
www.jiahenglu.net
2
Directed Acyclic Graphs

• A directed acyclic graph or DAG is a directed
  graph with no directed cycles

3
DFS and DAGs

• Argue that a directed graph G is acyclic iff a
  DFS of G yields no back edges
• Forward if G is acyclic, will be no back edges
• Trivial a back edge implies a cycle
• Backward if no back edges, G is acyclic
• Argue contrapositive G has a cycle ? ? a back
  edge
• Let v be the vertex on the cycle first
  discovered, and u be the predecessor of v on the
  cycle
• When v discovered, whole cycle is white
• Must visit everything reachable from v before
  returning from DFS-Visit()
• So path from u?v is yellow?yellow, thus (u, v) is
  a back edge

4
Topological Sort

• Topological sort of a DAG
• Linear ordering of all vertices in graph G such
  that vertex u comes before vertex v if edge (u,
  v) ? G
• Real-world example getting dressed

5
Getting Dressed
Underwear
Socks
Watch
Shoes
Pants
Shirt
Belt
Tie
Jacket
6
Getting Dressed
Underwear
Socks
Watch
Shoes
Pants
Shirt
Belt
Tie
Jacket
Socks
Underwear
Pants
Shoes
Watch
Shirt
Belt
Tie
Jacket
7
Topological Sort Algorithm

• Topological-Sort()
• Run DFS
• When a vertex is finished, output it
• Vertices are output in reverse topological order
• Time O(VE)
• Correctness Want to prove that (u,v) ? G ? u?f
  gt v?f

8
Correctness of Topological Sort

• Claim (u,v) ? G ? u?f gt v?f
• When (u,v) is explored, u is yellow
• v yellow ? (u,v) is back edge. Contradiction
  (Why?)
• v white ? v becomes descendent of u ? v?f lt u?f
  (since must finish v before backtracking and
  finishing u)
• v black ? v already finished ? v?f lt u?f

9
Minimum Spanning Tree

• Problem given a connected, undirected, weighted
  graph

6
4
5
9
14
2
10
15
3
8
10
Minimum Spanning Tree

• Problem given a connected, undirected, weighted
  graph, find a spanning tree using edges that
  minimize the total weight

6
4
5
9
14
2
10
15
3
8
11
Minimum Spanning Tree

• Which edges form the minimum spanning tree (MST)
  of the below graph?

A
6
4
5
9
H
B
C
14
2
10
15
G
E
D
3
8
F
12
Minimum Spanning Tree

• Answer

A
6
4
5
9
H
B
C
14
2
10
15
G
E
D
3
8
F
13
Minimum Spanning Tree

• MSTs satisfy the optimal substructure property
  an optimal tree is composed of optimal subtrees
• Let T be an MST of G with an edge (u,v) in the
  middle
• Removing (u,v) partitions T into two trees T1 and
  T2
• Claim T1 is an MST of G1 (V1,E1), and T2 is an
  MST of G2 (V2,E2) (Do V1 and V2
  share vertices? Why?)
• Proof w(T) w(u,v) w(T1) w(T2)(There cant
  be a better tree than T1 or T2, or T would be
  suboptimal)

14
Minimum Spanning Tree

• Thm
• Let T be MST of G, and let A ? T be subtree of T
• Let (u,v) be min-weight edge connecting A to V-A
• Then (u,v) ? T

15
Minimum Spanning Tree

• Thm
• Let T be MST of G, and let A ? T be subtree of T
• Let (u,v) be min-weight edge connecting A to V-A
• Then (u,v) ? T

16
Prims Algorithm

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

17
Prims Algorithm
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

14
2
10
15
3
8
Run on example graph
18
Prims Algorithm
?
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

?
?
?
14
2
10
15
?
?
?
3
8
?
Run on example graph
19
Prims Algorithm
?
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

?
?
?
14
2
10
15
r
0
?
?
3
8
?
Pick a start vertex r
20
Prims Algorithm
?
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

?
?
?
14
2
10
15
u
0
?
?
3
8
?
Red vertices have been removed from Q
21
Prims Algorithm
?
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

?
?
?
14
2
10
15
u
0
?
?
3
8
3
Red arrows indicate parent pointers
22
Prims Algorithm
?
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

14
?
?
14
2
10
15
u
0
?
?
3
8
3
23
Prims Algorithm
?
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

14
?
?
14
2
10
15
0
?
?
3
8
3
u
24
Prims Algorithm
?
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

14
?
?
14
2
10
15
0
8
?
3
8
3
u
25
Prims Algorithm
?
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

10
?
?
14
2
10
15
0
8
?
3
8
3
u
26
Prims Algorithm
?
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

10
?
?
14
2
10
15
0
8
?
3
8
3
u
27
Prims Algorithm
?
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

10
2
?
14
2
10
15
0
8
?
3
8
3
u
28
Prims Algorithm
?
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

10
2
?
14
2
10
15
0
8
15
3
8
3
u
29
Prims Algorithm
u
?
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

10
2
?
14
2
10
15
0
8
15
3
8
3
30
Prims Algorithm
u
?
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

10
2
9
14
2
10
15
0
8
15
3
8
3
31
Prims Algorithm
u
4
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

10
2
9
14
2
10
15
0
8
15
3
8
3
32
Prims Algorithm
u
4
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

5
2
9
14
2
10
15
0
8
15
3
8
3
33
Prims Algorithm
u
4
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

5
2
9
14
2
10
15
0
8
15
3
8
3
34
Prims Algorithm
u
4
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

5
2
9
14
2
10
15
0
8
15
3
8
3
35
Prims Algorithm
u
4
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

5
2
9
14
2
10
15
0
8
15
3
8
3
36
Prims Algorithm
4
6
4
9
5

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

5
2
9
14
u
2
10
15
0
8
15
3
8
3
37
Review Prims Algorithm

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

What is the hidden cost in this code?
38
Review Prims Algorithm

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• DecreaseKey(v, w(u,v))

39
Review Prims Algorithm

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• DecreaseKey(v, w(u,v))

How often is ExtractMin() called? How often is
DecreaseKey() called?
40
Review Prims Algorithm

• MST-Prim(G, w, r)
• Q VG
• for each u ? Q
• keyu ?
• keyr 0
• pr NULL
• while (Q not empty)
• u ExtractMin(Q)
• for each v ? Adju
• if (v ? Q and w(u,v) lt keyv)
• pv u
• keyv w(u,v)

What will be the running time?A Depends on
queue binary heap O(E lg V) Fibonacci heap
O(V lg V E)
41
Single-Source Shortest Path

• Problem given a weighted directed graph G, find
  the minimum-weight path from a given source
  vertex s to another vertex v
• Shortest-path minimum weight
• Weight of path is sum of edges
• E.g., a road map what is the shortest path from
  Chapel Hill to Charlottesville?

42
Binary Heap

• A special kind of binary tree. It has two
  properties that are not generally true for other
  trees
• Completeness
• The tree is complete, which means that nodes are
  added from top to bottom, left to right, without
  leaving any spaces. A binary tree is completely
  full if it is of height, h, and has 2h1-1 nodes.
  
• Heapness
• The item in the tree with the highest priority
  is at the top of the tree, and the same is true
  for every subtree.

43
Binary Heap

• Binary tree of height, h, is complete iff
• it is empty
• or
• its left subtree is complete of height h-1 and
  its right subtree is completely full of height
  h-2
• or
• its left subtree is completely full of height
  h-1 and its right subtree is complete of
• height h-1.

44
Binary Heap

• In simple terms
• A heap is a binary tree in which every parent is
  greater than its child(ren).
• A Reverse Heap is one in which the rule is every
  parent is less than the child(ren).

45
Binary Heap

• To build a heap, data is placed in the tree as it
  arrives.
• Algorithm
• add a new node at the next leaf position
• use this new node as the current position
• While new data is greater than that in the parent
  of the current node
• ? move the parent down to the current node
  
• ? make the parent (now vacant) the current
  node
• ? Place data in current node

46
Binary Heap

• New nodes are always added on the deepest level
• in an orderly way.
• The tree never becomes unbalanced.
• There is no particular relationship among the
  data items
• in the nodes on any given level, even the ones
  that have
• the same parent
• A heap is not a sorted structure. Can be regarded
  as
• partially ordered.
• A given set of data can be formed into many
  different
• heaps (depends on the order in which the data
  arrives.)

47
Binary Heap

• Example
• Data arrives to be heaped in the order
• 54, 87, 27, 67, 19, 31, 29, 18, 32, 56, 7, 12,
• 31

48
Binary Heap

• 54, 87, 27, 67, 19, 31, 29, 18, 32, 56, 7, 12, 31

54
49
Binary Heap

• 54, 87, 27, 67, 19, 31, 29, 18, 32, 56, 7, 12, 31

31
50
Binary Heap

• 54, 87, 27, 67, 19, 31, 29, 18, 32, 56, 7, 12, 31

18
32
etc
51
Binary Heap

• To delete an element from the heap
• Algorithm
• If node is the last logical node in the tree,
  simply delete it
• Else
• ? Replace the node with the last logical
  node in the tree
• ? Delete the last logical node from the tree
• ? Re-heapify

52
Binary Heap
Example Deleting the root node, T
53
Binary Heap
54
Binary Heap
55
Binary Heap
56

• Min-Max Heap
• Deaps
• Binomial Heaps
• Fibonacci Heaps

57
MIN-MAX Heaps (1/10)

• Definition
• A double-ended priority queue is a data structure
  that supports the following operations
• Insert an element with arbitrary key
• Delete an element with the largest key
• Delete an element with the smallest key
• Min heap or Max heap
• Only insertion and one of the two deletion
  operations are supported
• Min-Max heap
• Supports all of the operations just described.

58

• Definition
• A mix-max heap is a complete binary tree such
  that if it is not empty, each element has a field
  called key.
• Alternating levels of this tree are min levels
  and max levels, respectively.
• Let x be any node in a min-max heap. If x is on a
  min (max) level then the element in x has the
  minimum (maximum) key from amongall elements in
  the subtree with root x. We call this node a
  min (max) node.

59

• Insertion into a min-max heap (at a max level)
• If it is smaller/greater than its father (a
  min), then it must be smaller/greater than all
  max/min above. So simply check the
  min/max ancestors
• There exists a similar approach at a min level

60

• Following the nodes the max node i to the root
  and insert into its proper place

item 80
i
13
3
grandparent
3
0
define MAX_SIZE 100define FALSE 0define TRUE
1define SWAP(x,y,t) ((t)(x), (x)(y),
(y)(t))typedef struct int key/ other
fields /elementelement heapMAX_SIZE
1
80
2
3
4
5
6
7
8
9
10
11
12
13
40
61

• min_max_insert Insert item into the min-max heap

item.key
5
80
n
12
13
14
complexity O(log n)
parent
6
7
1
min
7
5
2
3
max
70
40
80
4
5
6
7
30
9
10
15
min
7
max
45
50
30
20
12
10
40
11
12
13
14
8
9
10
62

• Deletion of min element
• If we wish to delete the element with the
  smallest key, then this element is in the root.
• In general situation, we are to reinsert an
  element item into a min-max-heap, heap, whose
  root is empty.
• We consider the two cases
• The root has no children
• Item is to be inserted into the root.
• The root has at least one child.
• The smallest key in the min-max-heap is in one of
  the children or grandchildren of the root. We
  determine the node k has the smallest key.
• The following possibilities need to be considered

63

• item.key ? heapk.key
• No element in heap with key smaller than item.key
• Item may be inserted into the root.
• item.key ? heapk.key, k is a child of the root
• Since k is a max node, it has no descendants with
  key larger than heapk.key. Hence, node k has no
  descendants with key larger than item.key.
• heapk may be moved to the root and item
  inserted into node k.

64

• item.key ? heapk.key, k is a grandchild of
  the root
• In this case, heapk may be moved to the root,
  now heapk is seen as presently empty.
• Let parent be the parent of k.
• If item.key ? heapparent.key, then interchange
  them. This ensures that the max node parent
  contains the largest key in the sub-heap with
  root parent.
• At this point, we are faced with the problem of
  inserting item into the sub-heap with root k.
  Therefore, we repeat the above process.

65

• delete_min
• Delete the minimum element from the min-max heap

complexity O(log n)
n
12
11
i
1
5
last
5
k
5
11
parent
2
temp.key
x.key
12
1
0
7
7
9
2
3
70
40
4
5
6
7
30
9
10
15
12
45
50
30
20
12
8
9
10
11
12
66

• Deletion of max element
• Determine the children of the root which are
  located on max-level, and find the larger one
  (node) which is the largest one on the min-max
  heap
• We would consider the node as the root of a
  max-min heap
• There exist a similar approach (deletion of
  max element) as we mentioned above

max-min heap
67
Deaps(1/8)

• Definition
• The root contains no element
• The left subtree is a min-heap
• The right subtree is a max-heap
• Constraint between the two trees
• let i be any node in left subtree, j be the
  corresponding node in the right subtree.
• if j not exists, let j corresponds to parent of i
• i.key lt j.key

68
Deaps(2/8)

• i min_partner(n)
• j max_partner(n)
• if j gt heapsize j / 2

69
Deaps Insert(3/8)

• public void insert(int x)
• int i
• if (n 2)
• deap2 x return
• if (inMaxHeap(n))
• i minPartner(n)
• if (x lt deapi)
• deapn deapi
• minInsert(i, x)
• else maxInsert(n, x)

• else
• i maxPartner(n)
• if (x gt deapi)
• deapn deapi
• maxInsert(i, x)
• else minInsert(n, x)

70
Deaps Insert(3/8)

• public void insert(int x)
• int i
• if (n 2)
• deap2 x return
• if (inMaxHeap(n))
• i minPartner(n)
• if (x lt deapi)
• deapn deapi
• minInsert(i, x)
• else maxInsert(n, x)

71
Deaps Insert(3/8)

• http//www.csie.ntnu.edu.tw/swanky/ds/chap9.htm
• Insert and delete algorithm

72
Deaps(4/8)

• Insertion Into A Deap

73
Deaps(5/8)
74
Deaps(6/8)
75
Deaps delete min(7/8)

• public int deleteMin()
• int i, j, key deap2, x deapn--
• // move smaller child to i
• for (i 2 2i lt n deapi deapj, i
  j)
• j i 2
• if (j1 lt n (deapj gt deapj1)
  j

• // try to put x at leaf i
• j maxPartner(i)
• if (x gt deapj)
• deapi deapj
• maxInsert(j, x)
• else
• minInsert(i, x)
• return key

76
Deaps(8/8)
77
Binomial Heaps(1/10)

• Cost Amortization(????)
• actual cost of delete in Binomial Heap could be
  O(n), but insert and combine are O(1)
• cost amortization charge some cost of a heavy
  operation to lightweight operations
• amortized Binomial Heap delete is O(log2n)
• A tighter bound could be achieved for a sequence
  of operations
• actual cost of any sequence of i inserts, c
  combines, and dm delete in Binomial Heaps is
  O(icdmlogi)

78
Binomial Heaps(2/10)

• Definition of Binomial Heap
• Node degree, child ,left_link, right_link, data,
  parent
• roots are doubly linked
• a points to smallest root

79
Binomial Heaps(3/10)
80
Binomial Heaps(4/10)

• Insertion Into A Binomial Heaps
• make a new node into doubly linked circular list
  pointed at by a
• set a to the root with smallest key
• Combine two B-heaps a and b
• combine two doubly linked circular lists to one
• set a to the root with smallest key

81
Binomial Heaps(5/10)

• Deletion Of Min Element

82
Binomial Heaps(6/10)
83
Binomial Heaps(7/10)
84
Binomial Heaps(8/10)
85
Binomial Heaps(9/10)
86
Binomial Heaps(10/10)

• Trees in B-Heaps is Binomial tree
• B0 has exactly one node
• Bk, k gt 0, consists of a root with degree k and
  whose subtrees are B0, B1, , Bk-1
• Bk has exactly 2k nodes
• actual cost of a delete is O(logn s)
• s number of min-trees in a (original roots - 1)
  and y (children of the removed node)

87
Fibonacci Heaps(1/8)

• Definition
• delete, delete the element in a specified node
• decrease key
• This two operations are followed by cascading cut

88
Fibonacci Heaps(2/8)

• Deletion From An F-heap
• min or not min

89
Fibonacci Heaps(3/8)

• Decrease Key
• if not min, and smaller than parent, then delete

90
Fibonacci Heap(4/8)

• To prevent the amortized cost of delete min
  becomes O(n), each node can have only one child
  be deleted.
• If two children of x were deleted, then x must be
  cut and moved to the ring of roots.
• Using a flag (true of false) to indicate whether
  one of xs child has been cut

91
Fibonacci Heaps(5/8)

• Cascading Cut

92
Fibonacci Heaps(6/8)

• Lemma
• the ith child of any node x in a F-Heap has a
  degree of at least i 2, except when i1 the
  degree is 0
• Corollary
• Let Sk be the minimum possible number of
  descendants of a node of degree k, then S01,
  S12. From the lemma above, we got
• (2 comes from 1st
  child
• and root)

93
Fibonacci Heaps(7/8)

• Thats why the data structure is called Fibonacci
  Heap

94
Fibonacci Heaps(8/8)

• Application Of F-heaps