Practical Dynamic Programming in Ljungqvist – Sargent (2004) Presented by Edson Silveira Sobrinho for Dynamic Macro class University of Houston Economics Department

Chapter 3 presents some computational methods for Dynamic programs of the form: Subject to: given x. The Euler Equation helps in a variety of cases. One can also start with a Vo and iterate until convergence. However, it may be impossible to solve analytically. Then we have to adopt some numerical approximations. Motivation First approach Second approach Conclusions The curse of dimensionality 2

First approach for obtaining numerical approximations: The first method replaces the original problem with another problem by forcing the state vector to live on a finite and discrete grid of points. Then applies discrete-state dynamic programming to this problem. Downside is that it requires a small number of discrete states. Guess what? Markov chains! 3 Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation Motivation First approach Second approach Conclusions

Set up: an infinitely lived household likes to consume one good, which it can acquire by using labor income (ws t ) or accumulated savings (a t ). s t is endowment of labor at time t. s i is endowment of labor in state i. m-state Markov chain with transition matrix P. Say m = 2, where s i = 0 (if unemployed) or s i = 1 (if employed). The wage w is fixed over time. The household can choose to hold a single asset in discrete amount a t where a t Є A is a grid [ a 1 < a 2 < … < a n ]. The asset bears a gross rate of return r fixed over time. 4 Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation Motivation First approach Second approach Conclusions

Household maximization problem (Eq ): Subject to : Assume that: 5 Motivation First approach Second approach Conclusions Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation

Bellman equation: Which is a matrix m x n (2 states of employment x n choices of a). Let a' and s' indicate next period's asset and state. v (a,s) is the optimal value of the objective function, starting from asset and employment state (a, s). A solution of this problem is a value function v (a, s) that satisfies the Bellman Equation and an associated policy function a' = g(a, s) mapping this period’s (a, s) pair into an optimal choice of assets to carry into next period. 6 Motivation First approach Second approach Conclusions Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation

For discrete-state space of small size, it is easy to solve the Bellman equation numerically by manipulating matrices. Here is how to write a computer program Define two n x 1 vectors v j, j=1,2, i.e. one vector for each employment state. Each row takes the value of v(i) for a given asset i. Let 1 be the n x 1 vector consisting entirely of ones 7 Motivation First approach Second approach Conclusions Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation

Define two n x n matrices Rj whose (i, h) element is (Eq ): This is the current period utility conditional on current asset i and next period asset h, for a given employment state j. That is, all the possible utilities that arise from combinations of choices of assets for this period and the next one. 8 Motivation First approach Second approach Conclusions Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation

Define an operator T ([v1, v2]) that maps a pair of vectors [v1, v2] into a pair of vectors [tv1, tv2] (Eq ) : LHS is a column vector n x 1 The term in the Max operator is n x n Matrix. Intuition: each element brings the lifetime utility for a given pair of asset choices for current and next period. For each row (say current asset), the agent chooses the column (next period's asset) that allows max lifetime utility. Current consumption comes from the budget constraint in R 1. Similarly, for unemployed state: 9 Motivation First approach Second approach Conclusions Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation

Compact notation (Eq ): Then the Bellman Equation can be viewed as: And can be solved as: The production dynamics 10 Motivation First approach Second approach Conclusions Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation

Back in Chapter 3 we saw the Howard Improvement Algorithm. The idea was to Pick a feasible policy, u = h 0 (x), and compute the value associated with operating forever with that policy. Next generate a new policy u = h j+1 (x) that solves the two-period problem. Then iterate. Now let there be a predetermined class M of (N × N) stochastic matrices P, which are the objects of choice. I think that we can associate each stochastic matrix P to a pair (a,a'). That's why we have n x n matrices P. So choosing P is like choosing (a,a'). 11 Motivation First approach Second approach Conclusions Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation

We can put the Bellman equation in the following form (Eq ): For B = T – I, find P n+1 such that (Eq ): Intuition: Pick a feasible policy function and use it to iterate. I believe the advantage of the algorithm is that it is easier to compute (compare Eq with 4.5.2): 12 Motivation First approach Second approach Conclusions Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation

Eq Eq Step 1. For an initial feasible policy function g j (k, j) for j = 1, form the r h matrices using an equation similar to (4.5.1), then use equation (4.5.2) to evaluate the vectors of values [v j 1, v j 2 ] implied by using that policy forever. Step 2. Use [v j 1, v j 2 ] as the terminal value vectors in equation (4.3.2), and perform one step on the Bellman equation to find a new policy function gj+1(k, s) for j + 1 = 2. Use this policy function, update j, and repeat step 1. Step 3. Iterate. 13 Motivation First approach Second approach Conclusions Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation

Second approach for obtaining numerical approximations: Judd (1998) describes a method for iterating on the Bellman equation using a polynomial to approximate the value function and a numerical optimizer to perform the optimization at each iteration. We describe this method in the context of the Bellman equation for a particular problem. Model of optimal unemployment insurance as in chapter Motivation The model Implications for saving and growth Conclusions Motivation First approach Second approach Conclusions Polynomial approximations

A planner wants to provide incentives to an unemployed worker to search for a new job while also partially insuring the worker against bad luck in the search process. The planner seeks to deliver discounted expected utility V to an unemployed worker at minimum cost while providing proper incentives to search for work. Hopenhayn and Nicolini show that the minimum cost C(V) satisfies the Bellman equation (Eq ). 15 Motivation The model Implications for saving and growth Conclusions Motivation First approach Second approach Conclusions Polynomial approximations

The method uses a polynomial to approximate the i th iterate C i (V) of C(V). This polynomial is stored on the computer in terms of n + 1 coefficients. Then at each iteration, the Bellman equation is to be solved at a small number m >= n + 1 values of V. This procedure gives values of the i th iterate of the value function C i (V) at those particular V ’s. Then we interpolate to fill in the continuous function Ci(V). The minimum problem on the right side of equation (4.7.1) becomes a numerical minimizer (doable with simple algorithms in Matlab or Gauss). 16 Motivation The model Implications for saving and growth Conclusions Motivation First approach Second approach Conclusions Polynomial approximations

Algorithm summary: 1.Choose upper and lower bounds for V u [Vu, Vu] required to assure participation and a positive search effort, computed in chapter Choose a degree n for the approximator, a Chebyshev polynomial, and a number m >= n + 1 of nodes. 1.Generate the m zeros of the Chebyshev polynomial on the set [1,−1], given by (4.7.6). 1.By a change of scale, transform the z i ’s to corresponding points V l u in [Vu, Vu]. 17 Motivation The model Implications for saving and growth Conclusions Motivation First approach Second approach Conclusions Polynomial approximations

5.Choose initial values of the n+1 coefficients. Use these coefficients to define the function Ci(Vu) for iteration number i = 0. 6.Compute the function 5.For each point V l u, use a numerical minimization program to find C i+1 (V l u ). 6.Compute new values of the coefficients by using “least squares”. Return to step 5 and iterate to convergence. 18 Motivation The model Implications for saving and growth Conclusions Motivation First approach Second approach Conclusions Polynomial approximations

This chapter has described two of three standard methods for approximating solutions of dynamic programs numerically: discretizing the state space and using polynomials to approximate the value function. The next chapter describes the third method: making the problem have a quadratic return function and linear transition law. This will make solving a dynamic program easy by exploiting stochastic linear difference equations. 19 Motivation First approach Second approach Conclusions