Chapter 8: Dynamic Programming

1
Chapter 8 Dynamic Programming

• Deterministic Dynamic Programming
• Recursion
• Principle of Optimality
• Stochastic Dynamic Programming
• Applications
• Games
• shortest paths
• capacity expansion
• knapsack
• many, many more.
• Overview
• Transforms a complex optimization problem into a
  sequence of simpler ones.
• Usually begins at the end and works backwards
• Can handle a wide range of problems
• Relies on recursion, and on the principle of
  optimality
• Developed by Richard Bellman

2
Deterministic Dynamic Programming

• Elements of a DP model
• Stages one solves decision problems one stage
  at a time. Stages often can be thought of as
  time in most instances.
• Alternatives at each stage options to be
  selected
• States The description of the smaller
  subproblems is often referred to as the state
  (si) information links stages together
• Recursive function (forward/backward)
• Express the relationship between fi(si) and
  fi1(si1)
• Express si1 as a function of si
• Principle of Optimality
• Whatever the current state and decision, the
  remaining decisions must constitute an optimal
  policy regardless of the policy adopted in
  previous stages
• E.g. Whatever node j is selected, the remaining
  path from j to the end is the shortest path
  starting at j.

3
Deterministic Dynamic Programming

• Problem formulation Consider state variables,
  si, decision variables, xi, stage contribution,
  Ci(si, xi), a objective function,
  fi(xn,,xn-i,sn), and a transformation function,
  ti(xi, si)
• Solution procedure solve recursively the
  following

4
Example 1

• Shortest-route problem select the shortest route
  between 2 nodes
• Elements of the DP model
• Stage layer of nodes
• State node si at stage i
• Alternatives are represented by the routes from
  node si at stage i to other nodes si1 at stage
  i1
• Recursive function
• Difference between stages (stage rewards)
  distance d(si-1,si)
• Relationship between stages
• Relationship between states of stages d(si-1,si)
• Starting stage f0(s0) 0 (or fn(sn) 0 if
  backward recursion)

5
Stochastic Dynamic Programming

• Problem formulation Let S is the number of
  possible states at stage n1. The system goes to
  state i (i 1,2,,S) with probability pi given
  state xn and decision dn at stage n. If the
  system goes to state i, Ci(xi,di) is the
  contribution of state i at stage n to the
  objective function
• SDP ? decision tree
• Recursive function identify optimal policy for
  stage n, given the optimal policy for stage n1
  is available

6
Stochastic Dynamic Programming
7
Example 2

• An individual wishes to invest up to C 10000
  in the stock market over the next n 4 years.
  The investment plan calls for buying the stock at
  the start of the year and selling it at the end
  of the same year. Accumulated money may then be
  reinvested (in the whole or part) at the start of
  the following year. The degree of risk in the
  investment is represented by expressing the
  return probabilistically. A study of the market
  shows that the return on investment is affected
  by S (favorable or unfavorable) market conditions
  and that condition i yields a return Ci with
  probability pi, i 1,2,,S. How should the
  amount C be invested to realize the highest
  accumulation at the end of the n years? If S 3
  represents for the chance to double, break even
  and lose amount of money, respectively we have
  the following information p1 0.4, p2 0.2, p3
  0.4 and C1 1 C2 0, C3 -1

8
Example 2

• Let xi amount of funds available for investment
  at the start of year i (x1 C)
• yiamount actually invested at the start of year
  i (yi xi)
• Stage i is represented by year i
• The alternatives at stage i are given by yi
• The state at stage i is given by xi
• Recursive function
• Given that market condition k occurs with
  probability pk and fn1(xn1) xn1 because no
  investment occurs after year n. we have
• For market condition k