Title: Practical Dynamic Programming
1University of Houston Economics Department
Practical Dynamic Programming in Ljungqvist
Sargent (2004) Presented by Edson Silveira
Sobrinho for Dynamic Macro class
2MotivationFirst approachSecond
approach Conclusions
The curse of dimensionality
- Chapter 3 presents some computational methods for
Dynamic programs of the form - Subject to
- given x.
- The Euler Equation helps in a variety of cases.
- One can also start with a Vo and iterate until
convergence. - However, it may be impossible to solve
analytically. Then we have to adopt some
numerical approximations.
2
3Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
- MotivationFirst approachSecond approach
- Conclusions
- First approach for obtaining numerical
approximations - The first method replaces the original problem
with another problem by forcing the state vector
to live on a finite and discrete grid of points. - Then applies discrete-state dynamic programming
to this problem. - Downside is that it requires a small number of
discrete states. - Guess what? Markov chains!
3
4Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
- MotivationFirst approachSecond approach
- Conclusions
- Set up an infinitely lived household likes to
consume one good, which it can acquire by using
labor income (wst) or accumulated savings (at). - st is endowment of labor at time t. si is
endowment of labor in state i. - m-state Markov chain with transition matrix P.
Say m 2, where si 0 (if unemployed) or si 1
(if employed). - The wage w is fixed over time.
- The household can choose to hold a single asset
in discrete amount at where at ? A is a grid a1
lt a2 lt lt an. - The asset bears a gross rate of return r fixed
over time.
4
5- MotivationFirst approachSecond approach
- Conclusions
Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
- Household maximization problem (Eq. 4.2.1)
- Subject to
- Assume that
5
6- MotivationFirst approachSecond approach
- Conclusions
Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
- Bellman equation
- Which is a matrix m x n (2 states of employment x
n choices of a). - Let a' and s' indicate next period's asset and
state. - v (a,s) is the optimal value of the objective
function, starting from asset and employment
state (a, s). - A solution of this problem is a value function
v(a, s) that satisfies the Bellman Equation and
an associated policy function a' g(a, s)
mapping this periods (a, s) pair into an optimal
choice of assets to carry into next period. -
6
7- MotivationFirst approachSecond approach
- Conclusions
Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
- For discrete-state space of small size, it is
easy to solve the Bellman equation numerically by
manipulating matrices. Here is how to write a
computer program - Define two n x 1 vectors vj, j1,2 , i.e. one
vector for each employment state. Each row takes
the value of v(i) for a given asset i. - Let 1 be the n x 1 vector consisting entirely of
ones
7
8- MotivationFirst approachSecond approach
- Conclusions
- Discretization of state space
- Discrete-state dynamic programming
- Application of Howard Improvement algorithm
- Numerical implementation
- Define two n x n matrices Rj whose (i, h) element
is (Eq. 4.3.0) - This is the current period utility conditional on
current asset i and next period asset h, for a
given employment state j. - That is, all the possible utilities that arise
from combinations of choices of assets for this
period and the next one.
8
9- MotivationFirst approachSecond approach
- Conclusions
- Discretization of state space
- Discrete-state dynamic programming
- Application of Howard Improvement algorithm
- Numerical implementation
- Define an operator T (v1, v2) that maps a pair
of vectors v1, v2 into a pair of vectors tv1,
tv2 (Eq. 4.3.1) - LHS is a column vector n x 1
- The term in the Max operator is n x n Matrix.
- Intuition each element brings the lifetime
utility for a given pair of asset choices for
current and next period. For each row (say
current asset), the agent chooses the column
(next period's asset) that allows max lifetime
utility. Current consumption comes from the
budget constraint in R1. - Similarly, for unemployed state
9
10- MotivationFirst approachSecond approach
- Conclusions
- Discretization of state space
- Discrete-state dynamic programming
- Application of Howard Improvement algorithm
- Numerical implementation
The production dynamics
- Compact notation (Eq. 4.3.2)
- Then the Bellman Equation can be viewed as
- And can be solved as
-
10
11- MotivationFirst approachSecond approach
- Conclusions
Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
- Back in Chapter 3 we saw the Howard Improvement
Algorithm. - The idea was to Pick a feasible policy, u
h0(x), and compute the value associated with
operating forever with that policy. Next generate
a new policy u hj1(x) that solves the
two-period problem. Then iterate. - Now let there be a predetermined class M of (N
N) stochastic matrices P, which are the objects
of choice. - I think that we can associate each stochastic
matrix P to a pair (a,a'). That's why we have n x
n matrices P. So choosing P is like choosing
(a,a').
11
12- MotivationFirst approachSecond approach
- Conclusions
Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
- We can put the Bellman equation in the following
form (Eq. 4.4.3) - For B T I , find Pn1 such that (Eq. 4.4.4)
- Intuition Pick a feasible policy function and
use it to iterate. I believe the advantage of the
algorithm is that it is easier to compute
(compare Eq 4.3.0 with 4.5.2)
12
13- MotivationFirst approachSecond approach
- Conclusions
Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
- Eq. 4.5.1
- Eq. 4.5.2
- Step 1. For an initial feasible policy function
g j (k, j) for j 1, form the rh matrices using
an equation similar to (4.5.1), then use equation
(4.5.2) to evaluate the vectors of values vj1,
vj2 implied by using that policy forever. - Step 2. Use vj1, vj2 as the terminal value
vectors in equation (4.3.2), and perform one step
on the Bellman equation to find a new policy
function gj1(k, s) for j 1 2. Use this
policy function, update j , and repeat step 1. - Step 3. Iterate.
13
14MotivationThe modelImplications for saving and
growth Conclusions
- MotivationFirst approachSecond approach
- Conclusions
Polynomial approximations
- Second approach for obtaining numerical
approximations - Judd (1998) describes a method for iterating on
the Bellman equation using a polynomial to
approximate the value function and a numerical
optimizer to perform the optimization at each
iteration. - We describe this method in the context of the
Bellman equation for a particular problem. - Model of optimal unemployment insurance as in
chapter 19.
14
15MotivationThe modelImplications for saving and
growth Conclusions
- MotivationFirst approachSecond approach
- Conclusions
Polynomial approximations
- A planner wants to provide incentives to an
unemployed worker to search for a new job while
also partially insuring the worker against bad
luck in the search process. - The planner seeks to deliver discounted expected
utility V to an unemployed worker at minimum cost
while providing proper incentives to search for
work. - Hopenhayn and Nicolini show that the minimum cost
C(V) satisfies the Bellman equation (Eq. 4.7.1).
15
16MotivationThe modelImplications for saving and
growth Conclusions
- MotivationFirst approachSecond approach
- Conclusions
Polynomial approximations
- The method uses a polynomial to approximate the i
th iterate Ci(V) of C(V). This polynomial is
stored on the computer in terms of n 1
coefficients. - Then at each iteration, the Bellman equation is
to be solved at a small number m gt n 1 values
of V . - This procedure gives values of the i th iterate
of the value function Ci(V) at those particular V
s. - Then we interpolate to fill in the continuous
function Ci(V). - The minimum problem on the right side of equation
(4.7.1) becomes a numerical minimizer (doable
with simple algorithms in Matlab or Gauss).
16
17MotivationThe modelImplications for saving and
growth Conclusions
- MotivationFirst approachSecond approach
- Conclusions
Polynomial approximations
- Algorithm summary
- Choose upper and lower bounds for Vu Vu, Vu
required to assure participation and a positive
search effort, computed in chapter 19. - Choose a degree n for the approximator, a
Chebyshev polynomial, and a number m gt n 1 of
nodes. - Generate the m zeros of the Chebyshev polynomial
on the set 1,-1 , given by (4.7.6). - By a change of scale, transform the zis to
corresponding points Vlu in - Vu, Vu.
17
18MotivationThe modelImplications for saving and
growth Conclusions
- MotivationFirst approachSecond approach
- Conclusions
Polynomial approximations
- Choose initial values of the n1 coefficients.
Use these coefficients to define the function
Ci(Vu) for iteration number i 0. - Compute the function
- For each point Vlu , use a numerical minimization
program to find Ci1(Vlu ). - Compute new values of the coefficients by using
least squares. Return to step 5 and iterate to
convergence.
18
19- MotivationFirst approachSecond approach
- Conclusions
- This chapter has described two of three standard
methods for approximating solutions of dynamic
programs numerically discretizing the state
space and using polynomials to approximate the
value function. - The next chapter describes the third method
making the problem have a quadratic return
function and linear transition law. - This will make solving a dynamic program easy by
exploiting stochastic linear difference equations.
19