CS 450 Design

1
CS 450 Design Analysis of AlgorithmsNotes
12

• Fall 2009Sarah Mocas

2
Graph Algorithms

• Section VI
• Chapter 23 Minimum Spanning Trees
• Chapter 24 Shortest Path

3
Graph Algorithms
b
d
c
a
e
i
h
g
f
Graphs Nodes Edges
4
Minimum Spanning Trees

• Spanning Tree
• A tree (i.e., connected, acyclic graph) which
  contains all the vertices of the graph
• Minimum Spanning Tree (MST)
• Spanning tree with the minimum sum of weights
• Spanning forest
• If a graph is not connected, then there is a
  spanning tree for each connected component of the
  graph

5
Minimum Spanning Trees

• A spanning tree is a subgraph (tree) that
  connects all the vertices together.
• A minimum spanning tree (MST) is a spanning tree
  with weight less than or equal to the weight of
  every other spanning tree.

8
7
b
d
c
4
9
2
14
a
e
i
11
4
4
6
8
10
h
g
f
1
2
6
Minimum Spanning Trees
8
7
b
d
c
4
9
2
14
a
e
i
11
4
4
6
8
10
h
g
f
1
2
7
Applications of MST

• Find the least expensive way to connect a set of
  cities, terminals, computers, etc.

8
Example

• Problem
• A town has a set of houses
• and a set of roads
• A road connects 2 and only
• 2 houses
• A road connecting houses u and v has a repair
• cost w(u, v)
• Goal Repair enough (and no more) roads such
  that
• Everyone stays connected
• i.e., can reach every house from all other
  houses
• 2. Total repair cost is minimum

9
Minimum Spanning Trees

• A connected, undirected graph
• Vertices houses Edges roads
• A weight w(u, v) on each edge (u, v) ? E

• Find T ? E such that
• T connects all vertices
• w(T) S(u,v)?T w(u, v) is minimized

10
Properties of Minimum Spanning Trees

• Minimum spanning tree is not unique
• MST has no cycles see why
• We can take out an edge of a cycle, and still
  have the vertices connected while reducing the
  cost
• of edges in a MST is V - 1

11
Growing a MST Generic Approach

• Grow a set A of edges (initially empty)
• Incrementally add edges to A such that they would
  belong to a MST
• Idea add only safe edges

• An edge (u, v) is safe for A if and only if A ?
  (u, v) is also a subset of some MST

12
Generic MST algorithm

• A ? ?
• while A is not a spanning tree
• do find an edge (u, v) that is safe for A
• A ? A ? (u, v)
• return A
• How do we find safe edges?

13
Finding Safe Edges

• Look at edge (h, g)
• Is it safe for A initially?
• Later on
• Let S ? V be any set of vertices that includes h
  but not g (so that g is in V - S)
• In any MST, there has to be one edge (at least)
  that connects S with V - S
• Why not choose the edge with minimum weight
  (h,g)? Greedy strategy.

14
Definitions

• A cut (S, V - S) is a partition of vertices into
  disjoint sets S and V - S
• An edge crosses the cut (S, V - S) if one
  endpoint is in S and the other in V S

S?
?S
V- S ?
? V- S
15
Definitions (contd)

• A cut respects a set A of edges ? no edge
• in A crosses the cut
• An edge is a light edge crossing a cut ? its
  weight is minimum over all edges crossing the cut
• there can be gt 1 light edges crossing

16
Minimum Spanning Trees
A crossingedge
b
d
c
a
e
i
h
g
f
A Cut
17
Minimum Spanning Trees
Respectededges
b
d
c
a
e
i
h
g
f
A Cut
18
Theorem

• Let A be a subset of some MST (call it T) (S, V
  - S) be a cut that respects A, and (u, v) be a
  light edge crossing (S, V-S). Then (u, v) is safe
  for A.
• Proof
• Let T be an MST that includes A
• Case1 If T includes (u, v), then
• it would be safe for A done
• Case2 Suppose T does not
• include the edge (u, v)

Edges in A are shaded
19
Theorem - Proof

• Let T be a MST that includes A but not (u,v)
• We will construct another MST, T, for G
  containing (u,v), thus demonstrating the theorem
• T now includes A ? (u, v)

20
Theorem - Proof

• T contains a unique path p between u and v

• Path p must cross the cut (S, V - S) at least
  once let (x, y) be that edge
• Lets remove (x, y) ? breaks T into two components

• Adding (u, v) reconnects the components T T -
  (x, y) ? (u, v)

21
Theorem Proof (cont.)

• T T - (x, y) ? (u, v)
• Need to show T is an MST
• (u, v) is a light edge
• ? w(u, v) w(x, y)
• w(T)
• w(T) - w(x, y) w(u, v) w(T)

• Since T is a spanning tree w(T) w(T ) ? T
  must be an MST as well

22
Theorem Proof (cont.)

• Need to show that (u, v) is safe for A
• i.e., (u, v) can be a part of an MST
• A ? T since A ? T but (x, y) ? A since A
  respects the cut
• A ? (u, v) ? T
• Since T is an MST
• ? (u, v) is safe for A

23
Minimum Spanning Tree T
A
8
7
b
d
c
4
9
2
14
a
e
u
11
4
6
4
8
10
v
g
f
1
2

• u v are on opposite sides of the cut, there is
  at least one edge in T that crosses the cut on
  the path from u to v.
• Let (x,y) be any such edge.
• (x,y) is not in A since the cut respects A

24
Minimum Spanning Tree T
A
8
7
b
d
c
4
9
2
14
a
e
u
11
4
6
4
8
10
v
g
f
1
2
Removing (x,y) breaks the MST into two
parts Adding (u,v) heals it, forming a new MST
T Since (u,v) is a light edge w(T)
w(T) Since T is a MST, w(T) w(T) so T is a
MST
25
Prims Algorithm

• Begin with a single vertex - an arbitrary root
  VA a
• Add a light edge that crosses the cut
• Update the cut and repeat
• Repeat until the tree spans all vertices
• The edges in set A always form a single tree

26
Prims Algorithm
8
7
b
d
c
4
9
2
a
e
i
11
14
4
4
6
8
10
h
g
f
1
2

• Begin with a single vertex
• Add a light edge that crosses the cut
• Update the cut and repeat

27
Prims Algorithm
8
7
b
d
c
4
9
2
a
e
i
11
14
4
4
6
8
10
h
g
f
1
2

• Begin with a single vertex
• Add a light edge that crosses the cut
• Update the cut and repeat

28
Prims Algorithm
8
7
b
d
c
4
9
2
a
e
i
11
14
4
4
6
8
10
h
g
f
1
2

• Begin with a single vertex
• Add a light edge that crosses the cut
• Update the cut and repeat

29
How to Find Light Edges Quickly?

• Use a priority queue Q
• Contains vertices not yet
• included in the tree, i.e., (V VA)
• VA a, Q b, c, d, e, f, g, h, i
• We associate a key with each vertex v
• keyv minimum weight of any edge (u, v)
  connecting v to VA

w1
w2
a
Keya min(w1,w2)
30
How to Find Light Edges Quickly?

• After adding a new node to VA we update the
  weights of all the nodes adjacent to it
• so after adding a to the tree, kb4 and
  kh8
• Key of v is ? if v is not adjacent to any
  vertices in VA

31
Example

• 0 ? ? ? ? ? ? ? ?
• Q a, b, c, d, e, f, g, h, i
• VA ?
• Extract-MIN(Q) ? a

key b 4 ? b a key h 8 ? h a
4 ? ? ? ? ? 8 ? Q b, c, d, e, f, g, h, i
VA a Extract-MIN(Q) ? b
4
?
?
?
? v is the parent of v
8
7
b
c
d
4
9
?
2
?
a
e
i
11
14
4
6
7
8
10
h
g
f
2
1
?
?
?
8
32
Example
key c 8 ? c b key h 8 ? h a -
unchanged 8 ? ? ? ? 8 ? Q c, d, e, f,
g, h, i VA a, b Extract-MIN(Q) ? c
8
?
?
key d 7 ? d c key f 4 ? f c key
i 2 ? i c 7 ? 4 ? 8 2 Q
d, e, f, g, h, i VA a, b,
c Extract-MIN(Q) ? i
7
2
4
33
Example
key h 7 ? h i key g 6 ? g i
7 ? 4 6 8 Q d, e, f, g, h VA a, b, c,
i Extract-MIN(Q) ? f
key g 2 ? g f key d 7 ? d c
unchanged key e 10 ? e f 7 10 2 8
Q d, e, g, h VA a, b, c, i,
f Extract-MIN(Q) ? g
6
7
10
2
34
Example
key h 1 ? h g 7 10 1 Q d, e,
h VA a, b, c, i, f, g Extract-MIN(Q) ? h
1
7 10 Q d, e VA a, b, c, i, f,
g, h Extract-MIN(Q) ? d
35
Example
key e 9 ? e f 9 Q e VA
a, b, c, i, f, g, h, d Extract-MIN(Q) ? e Q
? VA a, b, c, i, f, g, h, d, e
9
36
PRIM(V, E, w, r)

• Q ? ?
• for each u ? V
• do keyu ? 8
• pu ? NIL
• INSERT(Q, u)
• keyr ? 0
• while Q ? ?
• do u ? EXTRACT-MIN(Q)
• for each v ? Adju
• do if v ? Q and w(u, v) lt keyv
• then pv ? u
• DECREASE-KEY(Q, v,
  w(u, v))

Total time O(VlgV ElgV) O(ElgV)
O(V) if Q is implemented as a min-heap
Min-heap operations O(VlgV)
Executed V times
Takes O(lgV)
Executed O(E) times total
O(ElgV)
Takes O(lgV)
37
Prims Algorithm

• Prims algorithm is a greedy algorithm
• Greedy algorithms find solutions based on a
  sequence of choices which are locally optimal
  at each step.
• Nevertheless, Prims greedy strategy produces a
  globally optimum solution!

38
What makes a greedy algorithm?

• Feasible
• Has to satisfy the problems constraints
• Locally Optimal
• The greedy part has to make the best local choice
  among all feasible choices available on that step
• If this local choice results in a global optimum
  then the problem has optimal substructure
• Irrevocable
• Once a choice is made it cant be un-done on
  subsequent steps of the algorithm
• Simple examples
• Playing chess by making best move without
  lookahead
• Giving fewest number of coins as change
• Simple and appealing, but doesnt always give the
  best solution

39
Does greedy work?

• The Traveling Salesman Problem
• A salesman wishes to visit k cities. The road
  between each pair of cities has a certain cost
  (distance). The salesman wants to visit all
  cities in the cheapest total cost.
• Repetition is not allowed - each city has to be
  visited exactly once
• The salesman must return home after the last city
  a trip is a cycle that goes through each vertex
  once
• Does greedy work?
• Travelling salesman problem - Wikipedia, the free
  encyclopedia

40
Traveling Salesman Problem
10
8
7
b
d
c
8
4
2
12
a
e
i
11
4
10
4
6
8
h
g
f
1
2
7
41
Traveling Salesman Problem
10
No known polynomial time solution!
8
7
b
d
c
8
4
2
12
a
e
i
11
4
10
4
6
8
h
g
f
1
2
7
Greedy doesnt work counter example
42
A different instance of the generic approach
(instance 2)
(instance 1)

• A is a forest containing connected components
• Initially, each component is a single vertex
• Any safe edge merges two components into one
• Each component is a tree

43
Kruskals Algorithm

• How is it different from Prims algorithm?
• Prims algorithm grows one tree all the time
• Kruskals algorithm grows
• multiple trees (i.e., a forest)
• Trees are merged together
• using safe edges
• Since an MST has exactly V - 1
• edges, after V - 1 merges,
• we would have only one component

44
Kruskals Algorithm

• Start with each vertex being its own component
• Repeatedly merge two components into one by
  choosing the light edge that connects them
• Which components to consider at each iteration?
• Scan the set of edges in monotonically increasing
  order by weight

45
Kruskals Algorithm
8
7
b
d
c
4
9
2
a
e
i
11
14
4
4
6
8
10
h
g
f
1
2

• Begin with a the lowest cost edge
• Add the lowest cost edge that grows the forest
• Repeat

46
Kruskals Algorithm
8
7
b
d
c
4
9
2
a
e
i
11
14
4
4
6
8
10
h
g
f
1
2

• Begin with a the lowest cost edge
• Add the lowest cost edge that grows the forest
• Repeat

47
Kruskals Algorithm
8
7
b
d
c
4
9
2
a
e
i
11
14
4
4
6
8
10
h
g
f
1
2

• Begin with a the lowest cost edge
• Add the lowest cost edge that grows the forest
• Repeat

48
Kruskals Algorithm
8
7
b
d
c
4
9
2
a
e
i
11
14
4
4
6
8
10
h
g
f
1
2

• Begin with a the lowest cost edge
• Add the lowest cost edge that grows the forest
• Repeat

49
Kruskals Algorithm
8
7
b
d
c
4
9
2
a
e
i
11
14
4
4
6
8
10
h
g
f
1
2

• Begin with a the lowest cost edge
• Add the lowest cost edge that grows the forest
• Repeat

50
KRUSKAL(V, E, w)

• A ? ?
• for each vertex v ? V
• do MAKE-SET(v)
• sort E into non-decreasing order by w
• for each (u, v) taken from the sorted list
• do if FIND-SET(u) ? FIND-SET(v)
• then A ? A ? (u, v)
• UNION(u, v)
• return A
• Running time O(V ElgE ElgV) O(ElgE)
  dependent on the implementation of the
  disjoint-set data structure

O(V)
O(ElgE)
O(E)
O(lgV)
51
Observations

• An optimal path to a destination node v that
  passes through x must contain the optimal path to
  x
• Additional data
• Dv distance of v from source (inf)
• Pv predecessor of v on the best path
• On to Shortest Path and Dijkstras Algorithm

52
Dijkstras Algorithm
8
7
b
d
c
4
9
2
a
e
i
11
14
4
4
6
8
10
h
g
f
1
2

• Want to find the best way to get from a single
  source to anywhere else
• Also a greedy algorithm

53
Dijkstras Algorithm
Make a greedy move and then update if new
shortest paths are found Called Relax
54
Shortest Path Problem

• Input
• Directed graph G (V, E)
• Weight function w E ? R
• Weight of path p ?v0, v1, . . . , vk?
• Shortest-path weight from u to v
• There might be multiple shortest paths from u to
  v

55
Negative-Weight Edges

• Negative-weight edges may form negative-weight
  cycles
• If such cycles are reachable from
• the source, then d(s, v) is not properly
• defined!
• Keep going around the cycle, and get w(s, v) -
  ? for all v on the cycle

56
Cycles

• Can shortest paths contain cycles?
• Negative-weight cycles
• Shortest path is not well defined
• Positive-weight cycles
• By removing the cycle, we can get a shorter path
• Zero-weight cycles
• No reason to use them
• Can remove them to obtain a path with same weight

No!
No!
57
Shortest-Paths Notation

• For each vertex v ? V
• d(s, v) shortest-path weight
• dv shortest-path weight estimate
• Initially, dv8
• dv?d(s,v) as algorithm progresses
• ?v predecessor of v on a shortest path from s
• If no predecessor, ?v NIL
• ? induces a tree - shortest-path tree

58
Dijkstras

• Dijkstra(G, w, s)
• Initialize ? and D / parent and
  weight /
• S ? nil
• Q ? VG / make the queue /
• while Q ! nil time O(V)
• u extract-min( Q )
• S ? S u / extend the path /
• for each vertex, v, adjacent to u
• if Dv gt Du w(u,v) Relax
• Dv ? Du w(u,v)
• ?v ? u

59
Relaxation Step

• Relaxing an edge (u, v) testing whether we can
  improve the shortest path to v found so far by
  going through u
• If dv gt du w(u, v)
• we can improve the shortest path to v
• ? dvduw(u,v)
• ? ?v ? u

After relaxation dv ? du w(u, v)
RELAX(u, v, w)
no change
60
Dijkstra (G, w, s)
Qlty,t,x,zgt
Sltsgt
Sltgt Qlts,t,x,z,ygt
t
x
1
?
?
10
9
2
4
3
6
0
s
s
7
5
?
?
2
y
z
61
(No Transcript)
62
All Pairs Shortest Paths

• Dont have a specific start and end point

63
Variants of Shortest Path

• Single-source shortest paths
• G (V, E) ? find a shortest path from a given
  source vertex s to each vertex v ? V
• Single-destination shortest paths
• Find a shortest path to a given destination
  vertex t from each vertex v
• Reversing the direction of each edge ?
  single-source
• Single-pair shortest path
• Find a shortest path from u to v for given
  vertices u and v
• All-pairs shortest-paths
• Find a shortest path from u to v for every pair
  of vertices u and v

64
Algorithms

• Bellman-Ford algorithm
• Negative weights are allowed
• Negative cycles reachable from the source are not
  allowed.
• Dijkstras algorithm
• Negative weights are not allowed
• Operations common in both algorithms
• Initialization
• Relaxation

65
Bellman-Ford Algorithm (contd)

• Idea
• Each edge is relaxed V1 times by making V-1
  passes over the whole edge set.
• To make sure that each edge is relaxed exactly
  V 1 times, it puts the edges in an
  unordered list and goes over the list V 1
  times.

(t, x), (t, y), (t, z), (x, t), (y, x), (y, z),
(z, x), (z, s), (s, t), (s, y)
66
BELLMAN-FORD(V, E, w, s)
Pass 1
6
7
E (t, x), (t, y), (t, z), (x, t), (y, x), (y,
z), (z, x), (z, s), (s, t), (s, y)
67
(t, x), (t, y), (t, z), (x, t), (y, x), (y, z),
(z, x), (z, s), (s, t), (s, y)
68
Floyd-Warshall Algorithm

• Consider a path between va and vfinish
• The path has zero or more intermediary vertices
  that are also on the path (vb, vc, etc)
• Now consider order vertices of a graph where the
  vertices take values v1, v2, etc

69
Shortest Paths

• Consider a subset of the vertices
• v1, v2, , vk
• Assume we restrict our shortest paths to use
  only these vertices (call this the vk problem)
• We can solve this problem recursively
• Either the shortest path doesnt contain vk, and
  so is the solution to the vk-1 problem
• Or, the shortest path goes through vk and so is
  the sum of va vk and vk vfinish

70
Alternatively

• d(i,j)k
• FW(W) W is a n by n matrix
• n ? rowsW
• D0 ? W
• for k ? 1 to n
• for i ? 1 to n
• for j ? 1 to n
• dijk min(d(i,j)k-1, d(i,k)k-1 d(k,j)k-1

wij If k 0 min(d(i,j)k-1, d(i,k)k-1
d(k,j)k-1 If k gt 1