CS575 Parallel Processing

1
CS575 Parallel Processing

• Lecture nine Graph Algorithms
• Wim Bohm, Colorado State University

Except as otherwise noted, the content of this
presentation is licensed under the Creative
Commons Attribution 2.5 license.
2
Definitions

• Graph G (V, E)
• V set of nodes or vertices
• E set of edges (pairs of nodes)
• In an undirected graph, edges are unordered pairs
  of nodes
• In a directed graph edges are ordered pairs of
  nodes
• Path sequence of nodes (v0..vn) s.t. ?i (vi
  ,vi1) is an edge
• Path length number of edges in the path
• Simple path all nodes distinct
• Cycle path with first and last node equal
• Acyclic graph graph without cycles

3
Definitions cont

• Adjacency
• Two nodes are adjacent if there is a path of
  length 1 between them
• Complete graph
• all nodes in the graph are adjacent
• Undirected graph is connected
• if for all nodes vi and vj there is a path from
  vi to vj
• Sub-graph
• G(V, E) is a sub-graph of G(V,E) if V?V and
  E? E
• Sub-graph of G induced by V
• has all the edges (u,v) ? E such that u ? V, v ?
  V
• Weighted graph
• edges have a weight (cost, length,..) associated
  with them

4
Representations

• Nodes structures with pointers to other nodes
• Good representation when nodes have limited
  degree
• Adjacency matrix
• Aij 1 if (vi,vj) is an edge, 0 otherwise
• Alternative Aij weight(vi,vj) or ? if no edge,
  Aii 0
• Good representation for dense graphs (many edges)
• Adjacency lists
• n lists (one for each node)
• Lists contain (representations of) the adjacent
  nodes
• Good representation for sparse graphs (few edges)

5
Minimal Spanning Tree (MST)

• Spanning tree of an undirected graph G
• A tree that is a sub-graph of G containing ALL
  vertices
• Minimal spanning tree of a weighted graph G
• Spanning tree with minimal total weight
• G must be a connected graph
• Applications
• Lowest cost set of roads connecting a set of
  towns
• Shortest cable connecting a set of computers

6
Prims Algorithm for MST

• Pick an arbitrary vertex
• Grow MST by choosing a new vertex v and edge e
• such that they are guaranteed to be in the final,
  correct MST
• Select least-cost (minimal) edge e(u,v) such that
  
• u is already in MST
• v is not in MST as yet
• Keep doing this until all vertices are in MST
• This is a GREEDY algorithm
• a locally optimizing strategy leading to a global
  optimum

7
Properties of any tree hence MST

• Path between two nodes a and b in MST is unique
• Cycles in MST
• there are no cycles
• If a and b are non-adjacent, adding the edge
  (a,b) creates a cycle
• Removing any edge on that cycle makes it a tree
  again

8
How greedy works for MST (sloppy!)

• Consider current stage MST M
• Add the least-cost edge to M to obtain the next
  stage M
• M will be minimal.
• Proof by contradiction
• Suppose we can create an MST by not taking the
  minimal cost edge
• Call the minimal edge is e, and the non-minimal
  edge taken e
• Build the rest of the spanning tree
• We can now make a lower cost spanning tree by
  removing e and adding e
• Hence the spanning tree with e in it was not
  minimal

9
Prims Algorithm Code Structure

• Pick vertex r
• Vt r Et // MST in
  construction
• dr 0
• ? v ? V if ((r,v) ? E)
• dv w(r,v) ev r
  
• else dv ?
• while Vt ! V
• Select vertex u from V-Vt with minimal du
  
• Vt Vt u Et Et (u,eu)
• ? v ? V-Vt if (w(u,v) lt dv)
  dvw(u,v)evu

10
Complexity of Prim

• while-loop executed n-1 times
• Loop-body O(n)
• Sequential complexity O(n2)
• while-loop is sequential in nature, because of
  the data dependencies in Vt, Et, d and e
• Inner ? loops can be parallelized

11
Parallel Implementation of Prims

• Data distribution
• Each PE has data for n/p vertices
• Adjacency matrix A is block striped (column-wise)
• d and e block striped
• PEs compute a local minimum ul
• Local minima accumulate to give global minimum in
  PE0
• PE0 broadcasts global minimum ug
• PE owning ug updates Vt ,Et
• All PEs update their partition of d and e using
  their columns of A

12
Single Source Shortest Path - SSSP

• Given a vertex s and weighted graph G, find the
  shortest distances from s to each vertex
• Dijkstras algorithm (very much like Prim)
• Vt s
• ? v ? V-Vt if ((s,v) ? E) lv
  w(s,v) else lv ?
• while Vt ! V
• Select vertex u from V-Vt with
  minimal lu
• Vt Vt u
• ? v ? V-Vt lv min(lv,
  luw(u,v))

13
Parallel SSSP

• Very similar to Prim
• Data distribution
• n/p vertices per PE
• Column distribute A
• block distribute l
• Find minimal ul locally
• Accumulate to obtain global minimum ug
• Broadcast global minimum ug
• Every PE updates its l block using its
  column-block of A

14
All pairs shortest paths - APSP

• Find length of shortest path between all vertex
  pairs
• nn distance matrix D Dij is shortest distance
  for vi ? vj
• Algorithms
• Dijkstras APSP
• Dynamic programming approach
• Matrix multiplication based APSP
• Floyds APSP

15
Dijkstras APSP

• for all nodes v in G do Dijkstra SSSP
• Sequential O(n3)
• Two possible approaches for parallelization
• Source-partitioned Partition nodes over n PEs,
  all nodes get A
• Do Sequential Dijkstra algorithm at each PE.
• No communication required, Tp O(n2)
• Work W O(n3), Speedup O(n), Efficiency O(1)
• Source-parallel Partition nodes and A over p PEs
• Do Parallel Dijkstras algorithm for each node
• More parallelism
• No perfect speedup

16
Dynamic Programming approach

• Formulate the problem in a recursive fashion
• Reverse this formulation to create a BOTTOM UP
  solution
• Use solutions for smaller problems to create
  solutions for larger ones
• There can be multiple recursive formulations
• Recurrence on path length (Matrix Multiply
  formulation)
• Recurrence on node set (Floyds algorithm)

17
Matrix multiplication based APSP

• Terms used
• Pijk minimal length path from vi to vj using at
  most k edges
• dijk length of the path Pijk
• Dk matrix that stores dijk s
• Recursion based on path length
• Two possibilities Pijk can have k edges or lt k
  edges
• lt k edges Pijk Pijk-1
• k edges
• Consider vertex vm such that edge(vm, vj) belongs
  to Pijk
• Hence path from vi to vm contains (k-1) edges
• Pijk Pimk-1 (vm, vj)
• dijk mindijk-1, mini?m?n(dimk-1 w(vm,vj))
• mini?m?n(dimk-1 w(vm,vj))
• dij1 Aij

18
Matrix multiplication (cont.)

• Implementation
• Take matrix multiply algorithm
• Replace sum reduction by min reduction
• Replace point-wise multiply by point-wise sum
• And you have one APSP step .....
• Solution D Dn-1 Ak, where k is the next
  power of two from n-1
• Any parallel matrix multiply algorithm can be
  used
• Complexity O(n3.log(n))
• This algorithm is not as good as simple parallel
  Dijkstra !!

19
Floyds APSP

• Terms used
• Vk v1, v2, ..... , vk
• Pijk minimal length path from vi to vj passing
  through nodes in Vk
• dijk length of the path Pijk
• Recursion based on node sets
• Two possibilities vk in Pijk or not
• vk not in Pijk Pijk Pijk-1
  and dijk dijk-1
• vk in Pijk Pijk Pikk-1 Pkjk-1 and
  dijk dikk-1 dkjk-1
• dijk min(dijk-1, dikk-1 dkjk-1) for k gt
  0
• w(vi, vj)
  for k 0
• Solution D Dn

20
Floyds APSP (Sequential)

• D0 A
• for k 1 to n
• for i 1 to n
• for j 1 to n
• Dijk min(Dijk-1 ,
  Dikk-1 Dkjk-1)
• O(n3) sequential time complexity
• O(n2) space complexity

21
Floyd Parallel

• Checkerboard partitioning
• Iteration k Broadcast k-th row and k-th column
  of D
• for k 1 to n
• each PE having a segment of row k
  of Dk-1 broadcast it in its column
• each PE having a segment of column
  k of Dk-1 broadcasts it in its row
• each PE waits to receive the
  needed segments of Dk-1
• each PE computes its part of Dk
• Note that this algorithm can be pipelined

22
Transitive Closure

• Given graph G(V,E)
• Transitive closure G(V,E)
• E (v1,v2) ? path from v1 to v2 in G
• Connectivity matrix A
• Aij 1 if (vi,vj) in E or i j
• ? otherwise
• Computing G, A
• Assign weight 1 to all edges in G
• Then use any APSP
• Aij 1 if dij gt 0 or i j
• ? otherwise.
• OR use Floyds algorithm replacing min by or and
  sum by and