ESI 6912: Dynamic Programming

1
ESI 6912Dynamic Programming

• Shortest Path problems

2
Shortest path problems

• Any (deterministic, finite horizon) dynamic
  programming problem can be viewed as a shortest
  path problem in an acyclic network.

3
Acyclic networks

• Set of nodes S
• Set of directed arcs T ? S ? S
• For all arcs i lt j (why?, how?)

4
Acyclic networks

• Arc length tij ??, for all (i,j )?T
• Path (i1,i2,,in) such that (ik,ik1)?T
  (k1,,n-1)
• Path length
• Goal find the
• Shortest path
• Longest path
• In acyclic networks these are equivalent!

5
Shortest path problem

• Consider the following street network
• For simplicity, we have assumed that all streets
  in the reverse direction do not make sense.
• Goal find the shortest route from node 1 to node
  8.

7
6
6
4
2
3
1
4
2
1
8
2
10
3
2
5
7
5
3
4
4
6
Shortest path problem

• This problem can be solved using
• Linear programming algorithm
• Minimum cost network flow algorithm
• Dijkstras algorithm
• Special purpose algorithms that use the (acyclic)
  structure of the network dynamic programming

7
Shortest path problem

• Structure
• Actions at any node, choose the next node to
  travel to
• Goal travel from node 1 to node 8 while
  traversing minimum distance
• Note that
• given that we are at any node,
• the remaining problem is to find the shortest
  path from that node to node 8.

8
Dynamic Programming approach

• The Dynamic Programming approach is to actually
  solve many problems at once
• All shortest path problems from node i to node 8
• The optimal solutions to these problems are
  strongly related to each other
• We say that the shortest path problem from node 1
  to node 8 is embedded in this larger class of
  optimization problems.

9
Shortest path problem

• Define
• fi shortest path length from node i to node 8
• We wish to find f1
• We can determine f8 trivially f8 0
• How can we relate the other values?

10
Shortest path problem

• From node i, we can go to any of the nodes j such
  that (i,j)?T
• Thus fi ? tij fj for (i,j)?T
• Since there are no other options
• Recall that i lt j for all (i,j)?T, so we can
  solve these equations by recursive fixing

11
Dynamic Programming terminology

• i state
• fi optimal value function
• 
  recurrence relation
• f8 0 boundary condition
• We may also keep track of the optimal first
  decision, say
• pi optimal first node to visit after i
• This function is called the optimal policy
  function

12
Shortest path tree

• The optimal policy function, can graphically be
  represented by the so-called shortest path tree
• This tree gives the solutions to the entire class
  of optimization problems

9
3
7
6
4
2
3
1
4
2
0
1
8
10
2
5
9
7
5
3
5
4
5
13
Principle of Optimality

• The basic principle underlying dynamic
  programming is the principle of optimality
• An optimal policy has the property that, from any
  state (node), the remaining decisions are optimal
  with respect to that starting state
• There exists a policy that is optimal for every
  state
• Any path contained in an optimal path is also
  optimal
• The latter characterization may not be true if
  the recurrence relation takes on a different form

14
Reaching

• We can obtain another dynamic programming
  formulation by noting
• to go from node 1 to j, we must pass through one
  of the immediate predecessors of node j
• Define
• vj shortest path length from node 1 to node j
• We wish to find v8, and trivially v1 0
• Recurrence relation

15
Recursive fixing vs. reaching

• A so-called backward recursion is solved by
  recursive fixing.
• A so-called forward recursion is solved by
  reaching.
• Which type of recursion is more suitable is
  problem-dependent.

16
Running Time

• The recursive fixing and reaching procedures to
  solve an acyclic shortest path problem consist
  of
• One addition
• One comparison
• per arc