COSC 3101A - Design and Analysis of Algorithms 11

1
COSC 3101A - Design and Analysis of Algorithms11

• Minimum Spanning Trees
• Prims Algorithm / Kruskals Algorithm
• Single-Source Shortest Paths
• Dijkstras Algorithm / Bellman-Ford Algorithm

Many of these slides are taken from Simonas
Šaltenis, Aalborg University, simas_at_cs.auc.dk
2
Spanning Tree

• A spanning tree of G is a subgraph which
• is a tree
• contains all vertices of G

3
Minimum Spanning Trees

• Undirected, connected graph G (V,E)
• Weight function W E R (assigning cost or
  length or other values to edges)

• Spanning tree tree that connects all the
  vertices (above?)
• Minimum spanning tree tree that connects all the
  vertices and minimizes

4
Optimal Substructure
T2

• MST T
• Removing the edge (u,v) partitions T into T1 and
  T2
• We claim that T1 is the MST of G1(V1,E1), the
  subgraph of G induced by vertices in T1
• Also, T2 is the MST of G2

T1
5
Greedy Choice

• Greedy choice property locally optimal (greedy)
  choice yields a globally optimal solution
• Theorem
• Let G(V, E), and let S Í V and
• let (u,v) be min-weight edge in G connecting S to
  V S
• Then (u,v) Î T some MST of G

6
Greedy Choice (2)

• Proof
• suppose (u,v) Ï T
• look at path from u to v in T
• swap (x, y) the first edge on path from u to v
  in T that crosses from S to V S
• this improves T contradiction (T supposed to be
  MST)

V-S
S
y
x
u
v
7
Generic MST Algorithm

• Generic-MST(G, w)
• 1 AÆ // Contains edges that belong to a MST
• 2 while A does not form a spanning tree do
• 3 Find an edge (u,v) that is safe for A
• 4 AAÈ(u,v)
• 5 return A

Safe edge edge that does not destroy As
property
MoreSpecific-MST(G, w) 1 AÆ // Contains
edges that belong to a MST 2 while A does not
form a spanning tree do 3.1 Make a cut (S,
V-S) of G that respects A 3.2 Take the
min-weight edge (u,v) connecting S to V-S 4
AAÈ(u,v) 5 return A
8
Prims Algorithm

• Vertex based algorithm
• Grows one tree T, one vertex at a time
• A cloud covering the portion of T already
  computed
• Label the vertices v outside the cloud with
  keyv the minimum weigth of an edge connecting
  v to a vertex in the cloud, keyv , if no
  such edge exists

9
Prims Algorithm (2)

• MST-Prim(G,w,r)
• 01 Q VG // Q vertices out of T
• 02 for each u Î Q
• 03 keyu
• 04 keyr 0
• 05 pr NIL
• 06 while Q ¹ Æ do
• 07 u ExtractMin(Q) // making u part of T
• 08 for each v Î Adju do
• 09 if v Î Q and w(u,v) lt keyv then
• 10 pv u
• 11 keyv w(u,v)

updating keys
10
Example

• 0 ? ? ? ? ? ? ? ?
• Q a, b, c, d, e, f, g, h, i
• VA ?
• Extract-MIN(Q) ? a

key b 4 ? b a key h 8 ? h a
4 ? ? ? ? ? 8 ? Q b, c, d, e, f, g, h, i
VA a Extract-MIN(Q) ? b
11
Example
key c 8 ? c b key h 8 ? h a -
unchanged 8 ? ? ? ? 8 ? Q c, d, e, f,
g, h, i VA a, b Extract-MIN(Q) ? c
8
?
?
key d 7 ? d c key f 4 ? f c key
i 2 ? i c 7 ? 4 ? 8 2 Q d, e,
f, g, h, i VA a, b, c Extract-MIN(Q) ? i
7
2
4
12
Example
key h 7 ? h i key g 6 ? g i
7 ? 4 ? 8 Q d, e, f, g, h VA a, b, c,
i Extract-MIN(Q) ? f
key g 2 ? g f key d 7 ? d c
unchanged key e 10 ? e f 7 10 2 8
Q d, e, g, h VA a, b, c, i,
f Extract-MIN(Q) ? g
7
6
10
2
13
Example
key h 1 ? h g 7 10 1 Q d, e,
h VA a, b, c, i, f, g Extract-MIN(Q) ? h
7 10 Q d, e VA a, b, c, i, f,
g, h Extract-MIN(Q) ? d
1
14
Example
key e 9 ? e f 9 Q e VA
a, b, c, i, f, g, h, d Extract-MIN(Q) ? e Q ?
VA a, b, c, i, f, g, h, d, e
9
15
Priority Queues

• A priority queue is a data structure for
  maintaining a set S of elements, each with an
  associated value called key
• We need PQ to support the following operations
• BuildPQ(S) initializes PQ to contain elements
  of S
• ExtractMin(S) returns and removes the element of
  S with the smallest key
• ModifyKey(S,x,newkey) changes the key of x in S
• A binary heap can be used to implement a PQ
• BuildPQ O(n)
• ExtractMin and ModifyKey O(lg n)

16
PRIM( G, w, r)

• for each u ? VG
• do keyu ? 8
• pu ? NIL
• keyr ? 0
• Q ? VG
• while Q ? ?
• do u ? EXTRACT-MIN(Q)
• for each v ? Adju
• do if v ? Q and w(u, v) lt
  keyv
• then pv ? u
• keyv ?
  w(u, v) ? DECREASE-KEY(Q, v, w(u, v))

Total time O(VlgV ElgV) O(ElgV)
O(V) if Q is implemented as a min-heap
Min-heap operations O(VlgV)
Executed V times
Takes O(lgV)
Executed O(E) times
O(ElgV)
Constant
Takes O(lgV)
17
Prims Running Time

• Time VT(ExtractMin) O(E)T(ModifyKey)
• Time O(V lgV E lgV) O(E lgV)

Q T(ExtractMin) T(DecreaseKey) Total
array O(V) O(1) O(V 2)
binary heap O(lg V) O(lg V) O(E lgV )
Fibonacci heap O(lg V) O(1) amortized O(V lgV E )
18
Kruskal's Algorithm

• Edge based algorithm
• Add the edges one at a time, in increasing weight
  order
• The algorithm maintains A a forest of trees. An
  edge is accepted it if connects vertices of
  distinct trees
• We need a data structure that maintains a
  partition, i.e.,a collection of disjoint sets
• MakeSet(S,x) S S È x
• Union(Si,Sj) S S Si,Sj È Si È Sj
• FindSet(S, x) returns unique Si Î S, where x Î Si

19
Kruskal's Algorithm

• The algorithm adds the cheapest edge that
  connects two trees of the forest

MST-Kruskal(G,w) 01 A Æ 02 for each vertex v Î
VG do 03 Make-Set(v) 04 sort the edges of E
by non-decreasing weight w 05 for each edge (u,v)
Î E, in order by non-decreasing weight do 06 if
Find-Set(u) ¹ Find-Set(v) then 07 A A È
(u,v) 08 Union(u,v) 09 return A
20
Example
g, h, a, b, c, d, e, f, i g, h,
c, i, a, b, d, e, f g, h, f, c,
i, a, b, d, e g, h, f, c, i, a, b,
d, e g, h, f, c, i, a, b, d, e g, h,
f, c, i, a, b, d, e g, h, f, c, i, d,
a, b, e g, h, f, c, i, d, a, b, e g,
h, f, c, i, d, a, b, e g, h, f, c, i, d, a,
b, e g, h, f, c, i, d, a, b, e g, h, f, c,
i, d, a, b, e g, h, f, c, i, d, a, b, e g, h,
f, c, i, d, a, b, e

1. Add (h, g)
2. Add (c, i)
3. Add (g, f)
4. Add (a, b)
5. Add (c, f)
6. Ignore (i, g)
7. Add (c, d)
8. Ignore (i, h)
9. Add (a, h)
10. Ignore (b, c)
11. Add (d, e)
12. Ignore (e, f)
13. Ignore (b, h)
14. Ignore (d, f)

1 (h, g) 2 (c, i), (g, f) 4 (a, b), (c, f) 6
(i, g) 7 (c, d), (i, h)
8 (a, h), (b, c) 9 (d, e) 10 (e, f) 11 (b,
h) 14 (d, f)
a, b, c, d, e, f, g, h, i
21
Kruskal Example
22
Kruskal Example (2)
23
Kruskal Example (3)
24
Kruskal Example (4)
25
Disjoint Sets as Lists

• Each set a list of elements identified by the
  first element, all elements in the list point to
  the first element
• Union add a smaller list to a larger one
• FindSet O(1), Union(u,v) O(minC(u), C(v))

Æ
Æ
Æ
26
Kruskal Running Time

• Initialization O(V) time
• Sorting the edges Q(E lg E) Q(E lg V) (why?)
• O(E) calls to FindSet
• Union costs
• Let t(v) the number of times v is moved to a
  new cluster
• Each time a vertex is moved to a new cluster the
  size of the cluster containing the vertex at
  least doubles t(v) log V
• Total time spent doing Union
• Total time O(E lg V)

27
Shortest Path

• Generalize distance to weighted setting
• Digraph G (V,E) with weight function W E R
  (assigning real values to edges)
• Weight of path p v1 v2 vk is
• Shortest path a path of the minimum weight
• Applications
• static/dynamic network routing
• robot motion planning
• map/route generation in traffic

28
Shortest-Path Problems

• Shortest-Path problems
• Single-source (single-destination). Find a
  shortest path from a given source to each of the
  vertices
• Single-pair. Given two vertices, find a shortest
  path between them. Solution to single-source
  problem solves this problem efficiently, too.
• All-pairs. Find shortest-paths for every pair of
  vertices. Dynamic programming algorithm.
• Unweighted shortest-paths BFS.

29
Optimal Substructure

• Theorem subpaths of shortest paths are shortest
  paths
• Proof (cut and paste)
• if some subpath were not the shortest path, one
  could substitute the shorter subpath and create a
  shorter total path

30
Triangle Inequality

• Definition
• d(u,v) º weight of a shortest path from u to v
• Theorem
• d(u,v) d(u,x) d(x,v) for any x
• Proof
• shortest path u Î v is no longer than any other
  path u Î v in particular, the path
  concatenating the shortest path u Î x with the
  shortest path x Î v

31
Negative Weights and Cycles?

• Negative edges are OK, as long as there are no
  negative weight cycles (otherwise paths with
  arbitrary small lengths would be possible)
• Shortest-paths can have no cycles (otherwise we
  could improve them by removing cycles)
• Any shortest-path in graph G can be no longer
  than n 1 edges, where n is the number of
  vertices

32
Initialization

• Alg. INITIALIZE-SINGLE-SOURCE(V, s)
• for each v ? V
• do dv ? ?
• ?v ? NIL
• ds ? 0
• All the shortest-paths algorithms start with
  INITIALIZE-SINGLE-SOURCE

33
Relaxation

• Relaxing an edge (u, v) testing whether we can
  improve the shortest path to v found so far by
  going through u
• If dv gt du w(u, v)
• we can improve the shortest path to v
• ? update dv and ?v

After relaxation dv ? du w(u, v)
RELAX(u, v, w)
RELAX(u, v, w)
34
RELAX(u, v, w)

• if dv gt du w(u, v)
• then dv ? du w(u, v)
• ?v ? u
• All the single-source shortest-paths algorithms
• start by calling INIT-SINGLE-SOURCE
• then relax edges
• The algorithms differ in the order and how many
  times they relax each edge

35
Dijkstra's Algorithm

• Non-negative edge weights
• Greedy, similar to Prim's algorithm for MST
• Like breadth-first search (if all weights 1,
  one can simply use BFS)
• Use Q, priority queue keyed by dv (BFS used
  FIFO queue, here we use a PQ, which is re-ordered
  whenever d decreases)
• Basic idea
• maintain a set S of solved vertices
• at each step select "closest" vertex u, add it to
  S, and relax all edges from u

36
Dijkstras Pseudo Code

• Graph G, weight function w, root s

relaxing edges
37
Dijkstras Example
38
Dijkstras Example (2)

• Observe
• relaxation step (lines 10-11)
• setting dv updates Q (needs Decrease-Key)
• similar to Prim's MST algorithm

u
v
1
8
9
10
9
2
3
0
4
6
7
5
5
7
2
y
x
39
Dijkstras Correctness

• We will prove that whenever u is added to S, du
  d(s,u), i.e., that d is minimum, and that
  equality is maintained thereafter
• Proof
• Note that "v, dv ³ d(s,v)
• Let u be the first vertex picked such that
  shorter path than du, i.e., that Þ du gt
  d(s,u)
• We will show that the assumption of such a
  vertex leads to a contradiction

40
Dijkstra Correctness (2)

• Let y be the first vertex ÎV S on the actual
  shortest path from s to u, then it must be that
  dy d(s,y) because
• dx is set correctly for y's predecessor x ÎS on
  the shortest path (by choice of u as the first
  vertex for which d is set incorrectly)
• when the algorithm inserted x into S, it relaxed
  the edge (x,y), assigning dy the correct value

41
Dijkstra Correctness (3)

• But du gt dy Þ algorithm would have chosen y
  (from the PQ) to process next, not u Þ
  Contradiction
• Thus du d(s,u) at time of insertion of u into
  S, and Dijkstra's algorithm is correct

42
Dijkstras Running Time

• Extract-Min executed V time
• Decrease-Key executed E time
• Time V TExtract-Min E TDecrease-Key
• T depends on different Q implementations

Q T(Extract-Min) T(Decrease-Key)
array O(V) O(1) O(V 2)
binary heap O(lg V) O(lg V) O(E lg V)
Fibonacci heap O(lg V) O(1) O(V lgV E)
43
Bellman-Ford Algorithm

• Dijkstras doesnt work when there are negative
  edges
• Intuition we can not be greedy on the
  assumption that the lengths of paths will only
  increase in the future
• Bellman-Ford algorithm detects negative cycles
  (returns false) or returns the shortest path-tree
  

44
BELLMAN-FORD(G, w, s)

1. INITIALIZE-SINGLE-SOURCE(G, s)
2. for i ? 1 to VG - 1
3. do for each edge (u, v) ? EG
4. do RELAX(u, v, w)
5. for each edge (u, v) ? EG
6. do if dv gt du w(u, v)
7. then return FALSE
8. return TRUE

6
7
E (t, x), (t, y), (t, z), (x, t), (y, x), (y,
z), (z, x), (z, s), (s, t), (s, y)
45
Example
(t, x), (t, y), (t, z), (x, t), (y, x), (y, z),
(z, x), (z, s), (s, t), (s, y)
Pass 1
Pass 2
11
4
2
Pass 3
Pass 4
2
-2
46
BELLMAN-FORD(G, w, s)

• INITIALIZE-SINGLE-SOURCE(VG, s)
• for i ? 1 to VG - 1
• do for each edge (u, v) ? EG
• do RELAX(u, v, w)
• for each edge (u, v) ? EG
• do if dv gt du w(u, v)
• then return FALSE
• return TRUE
• Running time O(VE)

?(V)
O(V)
O(E)
O(E)
47
Correctness of Bellman-Ford

• Let di(s,u) denote the length of path from s to
  u, that is shortest among all paths, that contain
  at most i edges
• Prove by induction that du di(s,u) after the
  i-th iteration of Bellman-Ford
• Base case, trivial
• Inductive step (say du di-1(s,u))
• Either di(s,u) di-1(s,u)
• Or di(s,u) di-1(s,z) w(z,u)
• In an iteration we try to relax each edge ((z,u)
  also), so we will catch both cases, thus du
  di(s,u)

48
Correctness of Bellman-Ford

• After n-1 iterations, du dn-1(s,u), for each
  vertex u.
• If there is still some edge to relax in the
  graph, then there is a vertex u, such that
  dn(s,u) lt dn-1(s,u). But there are only n
  vertices in G we have a cycle, and it must be
  negative.
• Otherwise, du dn-1(s,u) d(s,u), for all u,
  since any shortest path will have at most n-1
  edges

49
Shortest-Path in DAGs

• Finding shortest paths in DAGs is much easier,
  because it is easy to find an order in which to
  do relaxations Topological sorting!

DAG-Shortest-Paths(G,w,s) 01 for each v Î VG 02
dv 03 ds 0 04 topologically sort
VG 05 for each vertex u, taken in topolog.
order do 06 for each vertex v Î Adju do 07
Relax(u,v,w)
50
Shortest-Paths in DAGs (2)

• Running time
• Q(VE) only one relaxation for each edge, V
  times faster than Bellman-Ford

51
Readings

• Chapters 23, 24