Practical Dynamic Programming

1 / 19
About This Presentation
Title:

Practical Dynamic Programming

Description:

University of Houston Economics Department Practical Dynamic Programming in Ljungqvist Sargent (2004) Presented by Edson Silveira Sobrinho for Dynamic Macro class – PowerPoint PPT presentation

Number of Views:1
Avg rating:3.0/5.0
Slides: 20
Provided by: EdsonSilv
Learn more at: http://www.uh.edu

less

Transcript and Presenter's Notes

Title: Practical Dynamic Programming


1
University of Houston Economics Department
Practical Dynamic Programming in Ljungqvist
Sargent (2004) Presented by Edson Silveira
Sobrinho for Dynamic Macro class
2
MotivationFirst approachSecond
approach Conclusions
The curse of dimensionality
  • Chapter 3 presents some computational methods for
    Dynamic programs of the form
  • Subject to
  • given x.
  • The Euler Equation helps in a variety of cases.
  • One can also start with a Vo and iterate until
    convergence.
  • However, it may be impossible to solve
    analytically. Then we have to adopt some
    numerical approximations.

2
3
Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
  • MotivationFirst approachSecond approach
  • Conclusions
  • First approach for obtaining numerical
    approximations
  • The first method replaces the original problem
    with another problem by forcing the state vector
    to live on a finite and discrete grid of points.
  • Then applies discrete-state dynamic programming
    to this problem.
  • Downside is that it requires a small number of
    discrete states.
  • Guess what? Markov chains!

3
4
Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
  • MotivationFirst approachSecond approach
  • Conclusions
  • Set up an infinitely lived household likes to
    consume one good, which it can acquire by using
    labor income (wst) or accumulated savings (at).
  • st is endowment of labor at time t. si is
    endowment of labor in state i.
  • m-state Markov chain with transition matrix P.
    Say m 2, where si 0 (if unemployed) or si 1
    (if employed).
  • The wage w is fixed over time.
  • The household can choose to hold a single asset
    in discrete amount at where at ? A is a grid a1
    lt a2 lt lt an.
  • The asset bears a gross rate of return r fixed
    over time.

4
5
  • MotivationFirst approachSecond approach
  • Conclusions

Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
  • Household maximization problem (Eq. 4.2.1)
  • Subject to
  • Assume that

5
6
  • MotivationFirst approachSecond approach
  • Conclusions

Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
  • Bellman equation
  • Which is a matrix m x n (2 states of employment x
    n choices of a).
  • Let a' and s' indicate next period's asset and
    state.
  • v (a,s) is the optimal value of the objective
    function, starting from asset and employment
    state (a, s).
  • A solution of this problem is a value function
    v(a, s) that satisfies the Bellman Equation and
    an associated policy function a' g(a, s)
    mapping this periods (a, s) pair into an optimal
    choice of assets to carry into next period.

6
7
  • MotivationFirst approachSecond approach
  • Conclusions

Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
  • For discrete-state space of small size, it is
    easy to solve the Bellman equation numerically by
    manipulating matrices. Here is how to write a
    computer program
  • Define two n x 1 vectors vj, j1,2 , i.e. one
    vector for each employment state. Each row takes
    the value of v(i) for a given asset i.
  • Let 1 be the n x 1 vector consisting entirely of
    ones

7
8
  • MotivationFirst approachSecond approach
  • Conclusions
  • Discretization of state space
  • Discrete-state dynamic programming
  • Application of Howard Improvement algorithm
  • Numerical implementation
  • Define two n x n matrices Rj whose (i, h) element
    is (Eq. 4.3.0)
  • This is the current period utility conditional on
    current asset i and next period asset h, for a
    given employment state j.
  • That is, all the possible utilities that arise
    from combinations of choices of assets for this
    period and the next one.

8
9
  • MotivationFirst approachSecond approach
  • Conclusions
  • Discretization of state space
  • Discrete-state dynamic programming
  • Application of Howard Improvement algorithm
  • Numerical implementation
  • Define an operator T (v1, v2) that maps a pair
    of vectors v1, v2 into a pair of vectors tv1,
    tv2 (Eq. 4.3.1)
  • LHS is a column vector n x 1
  • The term in the Max operator is n x n Matrix.
  • Intuition each element brings the lifetime
    utility for a given pair of asset choices for
    current and next period. For each row (say
    current asset), the agent chooses the column
    (next period's asset) that allows max lifetime
    utility. Current consumption comes from the
    budget constraint in R1.
  • Similarly, for unemployed state

9
10
  • MotivationFirst approachSecond approach
  • Conclusions
  • Discretization of state space
  • Discrete-state dynamic programming
  • Application of Howard Improvement algorithm
  • Numerical implementation

The production dynamics
  • Compact notation (Eq. 4.3.2)
  • Then the Bellman Equation can be viewed as
  • And can be solved as

10
11
  • MotivationFirst approachSecond approach
  • Conclusions

Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
  • Back in Chapter 3 we saw the Howard Improvement
    Algorithm.
  • The idea was to Pick a feasible policy, u
    h0(x), and compute the value associated with
    operating forever with that policy. Next generate
    a new policy u hj1(x) that solves the
    two-period problem. Then iterate.
  • Now let there be a predetermined class M of (N
    N) stochastic matrices P, which are the objects
    of choice.
  • I think that we can associate each stochastic
    matrix P to a pair (a,a'). That's why we have n x
    n matrices P. So choosing P is like choosing
    (a,a').

11
12
  • MotivationFirst approachSecond approach
  • Conclusions

Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
  • We can put the Bellman equation in the following
    form (Eq. 4.4.3)
  • For B T I , find Pn1 such that (Eq. 4.4.4)
  • Intuition Pick a feasible policy function and
    use it to iterate. I believe the advantage of the
    algorithm is that it is easier to compute
    (compare Eq 4.3.0 with 4.5.2)

12
13
  • MotivationFirst approachSecond approach
  • Conclusions

Discretization of state space Discrete-state
dynamic programming Application of Howard
Improvement algorithm Numerical implementation
  • Eq. 4.5.1
  • Eq. 4.5.2
  • Step 1. For an initial feasible policy function
    g j (k, j) for j 1, form the rh matrices using
    an equation similar to (4.5.1), then use equation
    (4.5.2) to evaluate the vectors of values vj1,
    vj2 implied by using that policy forever.
  • Step 2. Use vj1, vj2 as the terminal value
    vectors in equation (4.3.2), and perform one step
    on the Bellman equation to find a new policy
    function gj1(k, s) for j 1 2. Use this
    policy function, update j , and repeat step 1.
  • Step 3. Iterate.

13
14
MotivationThe modelImplications for saving and
growth Conclusions
  • MotivationFirst approachSecond approach
  • Conclusions

Polynomial approximations
  • Second approach for obtaining numerical
    approximations
  • Judd (1998) describes a method for iterating on
    the Bellman equation using a polynomial to
    approximate the value function and a numerical
    optimizer to perform the optimization at each
    iteration.
  • We describe this method in the context of the
    Bellman equation for a particular problem.
  • Model of optimal unemployment insurance as in
    chapter 19.

14
15
MotivationThe modelImplications for saving and
growth Conclusions
  • MotivationFirst approachSecond approach
  • Conclusions

Polynomial approximations
  • A planner wants to provide incentives to an
    unemployed worker to search for a new job while
    also partially insuring the worker against bad
    luck in the search process.
  • The planner seeks to deliver discounted expected
    utility V to an unemployed worker at minimum cost
    while providing proper incentives to search for
    work.
  • Hopenhayn and Nicolini show that the minimum cost
    C(V) satisfies the Bellman equation (Eq. 4.7.1).

15
16
MotivationThe modelImplications for saving and
growth Conclusions
  • MotivationFirst approachSecond approach
  • Conclusions

Polynomial approximations
  • The method uses a polynomial to approximate the i
    th iterate Ci(V) of C(V). This polynomial is
    stored on the computer in terms of n 1
    coefficients.
  • Then at each iteration, the Bellman equation is
    to be solved at a small number m gt n 1 values
    of V .
  • This procedure gives values of the i th iterate
    of the value function Ci(V) at those particular V
    s.
  • Then we interpolate to fill in the continuous
    function Ci(V).
  • The minimum problem on the right side of equation
    (4.7.1) becomes a numerical minimizer (doable
    with simple algorithms in Matlab or Gauss).

16
17
MotivationThe modelImplications for saving and
growth Conclusions
  • MotivationFirst approachSecond approach
  • Conclusions

Polynomial approximations
  • Algorithm summary
  • Choose upper and lower bounds for Vu Vu, Vu
    required to assure participation and a positive
    search effort, computed in chapter 19.
  • Choose a degree n for the approximator, a
    Chebyshev polynomial, and a number m gt n 1 of
    nodes.
  • Generate the m zeros of the Chebyshev polynomial
    on the set 1,-1 , given by (4.7.6).
  • By a change of scale, transform the zis to
    corresponding points Vlu in
  • Vu, Vu.

17
18
MotivationThe modelImplications for saving and
growth Conclusions
  • MotivationFirst approachSecond approach
  • Conclusions

Polynomial approximations
  • Choose initial values of the n1 coefficients.
    Use these coefficients to define the function
    Ci(Vu) for iteration number i 0.
  • Compute the function
  • For each point Vlu , use a numerical minimization
    program to find Ci1(Vlu ).
  • Compute new values of the coefficients by using
    least squares. Return to step 5 and iterate to
    convergence.

18
19
  • MotivationFirst approachSecond approach
  • Conclusions
  • This chapter has described two of three standard
    methods for approximating solutions of dynamic
    programs numerically discretizing the state
    space and using polynomials to approximate the
    value function.
  • The next chapter describes the third method
    making the problem have a quadratic return
    function and linear transition law.
  • This will make solving a dynamic program easy by
    exploiting stochastic linear difference equations.

19
Write a Comment
User Comments (0)