1. MIT and James Orlin © 2003 1 Dynamic Programming 2 –Review –More examples

2. MIT and James Orlin © 2003 2 Match game example Suppose that there are 20 matches on a table, and the person who picks up the last match wins. At each alternating turn, my opponent or I can pick up 1, 2 or 6 matches. Assuming that I go first, how can I be sure of winning the game?

3. MIT and James Orlin © 2003 3 Determining the strategy using DP n = number of matches left (n is the state/stage) g(n) = 1 if you can force a win at n matches. g(n) = 0 otherwise g(n) = optimal value function. At each state/stage you can make one of three decisions: take 1, 2 or 6 matches. g(1) = g(2) = g(6) = 1 (boundary conditions) g(3) = 0; g(4) = g(5) = 1. The recursion: g(n) = 1 if g(n-1) = 0 or g(n-2) = 0 or g(n-6) = 0; g(n) = 0 otherwise. Equivalently, g(n) = 1 – min (g(n-1), g(n-2), g(n-6)).

4. MIT and James Orlin © 2003 4 12345678910 11121314151617181920 12345678910 11121314151617181920 Winning and Losing Positions

5. MIT and James Orlin © 2003 5 Principle of Optimality Any optimal policy has the property that whatever the current state and decision, the remaining decisions must constitute an optimal policy with regard to the state resulting from the current decision. Whatever node j is selected, the remaining path from j to the end is the shortest path starting at j.

6. MIT and James Orlin © 2003 6 Capital Budgeting, again Investment budget = $14,000

7. MIT and James Orlin © 2003 7 Stages, and the DP recursion Solve the knapsack problem as a sequence of problems At stage k we consider whether to add item k or not to the knapsack. Let f(k,B) be the best NPV limited to stocks 1, 2, …, k only and using a budget of at most B. Knapsack problem via DP

8. MIT and James Orlin © 2003 8 Solving the Stockco Problem At stage 1 compute for each budget from 0 to $14,000 compute the best NPV while limited to stock 1. At stage 2 compute for each budget from 0 to $14,000 compute the best NPV while limited to stocks 1 and 2. At stage 3 compute for each budget from 0 to $14,000 compute the best NPV while limited to stocks 1, 2 and 3.

9. MIT and James Orlin © 2003 9 Capital budgeting in general Max  j=1..n c j x j s.t  j=1..n a j x j  b x binary Let f(k, B) = Max  j=1..k c j x j s.t  j=1..k a j x j  B x binary

10. MIT and James Orlin © 2003 10 The recursion f(0,0) = 0; f(0,k) is undefined for k > 0 f(k, B) = min ( f(k-1, B), f(k-1, B-a k ) + c k ) either item k is included, or it is not The optimum solution to the original problem is max { f(n, B) : 0  B  b }. Note: we solve the capital budgeting problem for all right hand sides less than b.

11. MIT and James Orlin © 2003 11 The Knapsack Problem Is there a feasible solution to 31 x + 35 y + 37 z = 422 x, y, z are non-negative integers. let f(1, j) = T if there is a solution to 31 x = j, f(1, j) = F if there is no solution to 31 x = j. where x is a non-negative integer let f(2, j) = T if there is a solution to 31 x + 35 y = j, x and y are non-negative integer

12. MIT and James Orlin © 2003 12 The Knapsack Problem Is there a feasible solution to 31 x + 35 y + 37 z = 422 x, y, z are non-negative integers. let f(3,j) = T if there is a solution to 31 x + 35 y + 37 z = j, x, y, and z are non-negative integer Note: there are three stages to this problem.

13. MIT and James Orlin © 2003 13 The recursion for the knapsack problem Is there a feasible solution to 31 x = 422 x non-negative integer? f(0,0) = T, f(0,k) = F for k > 0 f(1,k) = T if f(0,k) = T or f(1, k-31) = T for k = 31 to 422 f(1,k) = F otherwise. Does this really work?

14. MIT and James Orlin © 2003 14 The recursion for the knapsack problem Is there a feasible solution to 31 x + 35 y = 422 x, y are non-negative integers. f(2,k) = T if f(1,k) = T or f(2, k-35) = T for k = 0 to 422 f(2,k) = F otherwise

15. MIT and James Orlin © 2003 15 The recursion for the knapsack problem Is there a feasible solution to 31 x + 35 y + 37 z = 422 x, y, z are non-negative integers? f(3,k) = T if f(2,k) = T or f(3, k-37) = T f(3,k) = F otherwise

16. MIT and James Orlin © 2003 16 The DP again, without stages Is there a feasible solution to 31 x + 35 y + 37 z = 422 x, y, z are non-negative integers. let g(j) = T if there is a solution to 31 x + 35 y + 37 z = j, x, y, and z are non-negative integer Then g(0) = T. For j > 0, g(j) is true if i.g(j-31) is true or ii.g(j-35) is true or iii. g(j-37) is true

17. MIT and James Orlin © 2003 17 Optimal Capacity Expansion: What is the least cost way of building plants? YearCum. DemandCost per plant in $millions 2002154 2003256 2004458 2005657 2006755 2007852 Cost of $15 million in any year in which a plant is built. At most 3 plants a year can be built

18. MIT and James Orlin © 2003 18 Cost of Building plants

19. MIT and James Orlin © 2003 19 Step 1. For each year, identify those table entries that are feasible, taking into account the lower bounds and the fact that at most 3 plants can be built per year. (Y, k) is a state denoting having k plants with Y years to go.

20. MIT and James Orlin © 2003 20 Step 2. f(Y, k) = remaining cost of capacity expansion starting with k plants with Y years to go. Fill in the entries for 0 years to go. Then fill in the entries for 1 year to go. Then fill in entries for 2 years to go, and continue.

21. MIT and James Orlin © 2003 21 Determining state and stage Stage: Y = years to go State: number of plants f(j, Y) = optimum cost of capacity expansion starting with j plants with Y years to go We want to compute f(0, 6). What is f(j, 0) for different j?

22. MIT and James Orlin © 2003 22 The recursion Let g(Y, k) be the cost of building k plants with Y years to go. Let l(Y) be the minimum number of plants needed with Y years to go. Let M(Y) be the maximum number of plants that can be built with Y years to go. Let f(Y,k) be the remaining cost of capacity expansion starting with k plants with Y years to go. What is f(0,k)? Can you compute f(Y,k) assuming that f(Y-1, j) is already computed for each j?

23. MIT and James Orlin © 2003 23 Summary for Dynamic Programming Recursion Principle of optimality states and stages and decisions useful in a wide range of situations –games –shortest paths –capacity expansion –more will be presented in next lecture.

24. MIT and James Orlin © 2003 24 Dynamic Programming Review Next lecture: DP under uncertainty. Applications –meeting a financial goal (a very simplified version) –getting a plane through enemy territory