1 1 Slide © 2000 South-Western College Publishing/ITP Slides Prepared by JOHN LOUCKS

2 2 Slide Chapter 18 Dynamic Programming n Dynamic Programming Overview n Dynamic Programming Notation n Backwards Recursion n 3 Applications of Dynamic Programming n A Production and Inventory Control Problem

3 3 Slide Dynamic Programming n Dynamic programming (DP) is an approach to problem solving which permits decomposing of the original problem into a series of several smaller subproblems. n To successfully apply DP, the original problem must be viewed as a multistage decision problem. Defining the stages is sometimes obvious, but at other times this requires subtle reasoning.

4 4 Slide Dynamic Programming n The power of DP is that one need solve only a small portion of all subproblems. This is due to Bellman's principle of optimality. It states that regardless of what decisions were made at previous stages, if the decision to be made at stage n is to be part of an overall optimal solution, then the decision made at stage n must be optimal for all remaining stages.

5 5 Slide Dynamic Programming Notation n At each stage, n, of the dynamic program, there is a state variable, x n, and an optimal decision variable, d n. n For each value of x n and d n at stage n, there is a return function value, r n ( x n, d n ). n The output of the process at stage n is x n- 1, the state variable for stage n- 1. It is calculated by a stage transformation function, t n ( x n, d n ). n The optimal value function, f n ( x n ), is the cumulative return starting at stage n in state x n and proceeding to stage 1 under an optimal policy (strategy).

6 6 Slide Backwards Recursion n Generally, a dynamic programming problem is solved by starting at the final stage and working backwards to the initial stage. This is called backwards recursion. n The following recursion relation can be used to operationalize the principle of optimality: f n ( x n ) = MAX { r n ( x n, d n ) + f n -1 ( t n ( x n, d n ))} f n ( x n ) = MAX { r n ( x n, d n ) + f n -1 ( t n ( x n, d n ))} d n d n n A problem is solved beginning at stage 0 with the boundary condition f 0 ( x 0 ) = 0, and working backwards to the last stage, N.

7 7 Slide Three Applications of Dynamic Programming Problems n Shortest Route Problem In solving a shortest route problem using dynamic programming, one should consider the network as a series of stages with a unique subset of nodes corresponding to each stage. The state variables correspond to the different nodes at each stage.

8 8 Slide Three Applications of Dynamic Programming Problems n Knapsack or Cargo Loading Problem The knapsack problem seeks to determine the optimal number of each of N items (which must not be fractional) to select in order to maximize profit subject to an overall capacity constraint. In solving a knapsack problem using dynamic programming, the stages correspond to the different items being placed into a knapsack. The state variables correspond to the capacity available at the stage.

9 9 Slide Three Applications of Dynamic Programming Problems n Production and Inventory Control Problems In production and inventory control problems, the stages correspond to time periods and the state variables generally will refer to the amounts of inventory on hand at the beginning of each stage.

10 Slide Example: Dicom Corporation n Production and Inventory Control Problem Dicom Corporation wishes to determine a production schedule for its new Model 44/12 virtual memory computer. Because of differences in parts availability and spare production capacity, the cost of producing the machines will vary from month to month. These costs and other data are shown on the next slide. The holding cost for each unsold machine still in inventory at the end of the month is $500,000. Corporate policy dictates that the maximum number of machines allowed in inventory at the end of any month is 8. Determine an optimal 4 month production schedule for the Dicom Model 44/12.

11 Slide Example: Dicom Corporation The following table gives these costs together with the sales demand over the next 4 months as well as the maximum possible production level per month. Cost of Maximum Cost of Maximum Production Production Sales Production Production Sales Per Machine Level Demand Per Machine Level Demand Month (in $100,000's) for Month (in Units) Month (in $100,000's) for Month (in Units) August August September September October October November November

12 Slide Example: Dicom Corporation n 4-Stage Dynamic Programming Problem Working backwards, let stage 1 correspond to November, stage 2 to October, etc. The following data (costs are in $100,000's) can be inferred: Product. Holding Product. Holding Product. Storage Cost Cost Product. Storage Cost Cost Month Dem. Capacity Capacity Per Unit Per unit ( n ) D n P n W n C n H n ( n ) D n P n W n C n H n

13 Slide Example: Dicom Corporation n State Variable Defined x n = number of computers in inventory at the beginning of month n. x n = number of computers in inventory at the beginning of month n. x 4 = 0 (Since the computer is new, there will be no inventory at the start of August.) x 4 = 0 (Since the computer is new, there will be no inventory at the start of August.) n Decision Variable Defined d n = production quantity for month n. d n = production quantity for month n. n Stage Transformation Function Then the stage transformations for months 0 through 4 can be defined by: (Previous month's inventory) + (production this month) - (demand this month), or Then the stage transformations for months 0 through 4 can be defined by: (Previous month's inventory) + (production this month) - (demand this month), or x n- 1 = x n + d n - D n x n- 1 = x n + d n - D n

14 Slide Example: Dicom Corporation n Return Function r n ( x n, d n ) = sum of the production and holding costs for month n. The production cost is the production cost per unit multiplied by the number of units produced ( d n ). The holding cost is the ending inventory for the month multiplied by the holding cost per unit. Hence, r n ( x n, d n ) = C n d n + H n ( x n + d n - D n ) This gives: r 1 ( x 1, d 1 ) = 50 d x This gives: r 1 ( x 1, d 1 ) = 50 d x r 2 ( x 2, d 2 ) = 31 d x r 2 ( x 2, d 2 ) = 31 d x r 3 ( x 3, d 3 ) = 23 d x r 3 ( x 3, d 3 ) = 23 d x r 4 ( x 4, d 4 ) = 37 d x r 4 ( x 4, d 4 ) = 37 d x

15 Slide Dicom Corporation n Restrictions on x n and d n Since backordering is not allowed, we must be able to meet the sales demand. That is, for month n :Since backordering is not allowed, we must be able to meet the sales demand. That is, for month n : x n + d n > D n (1) x n + d n > D n (1) Because there is a maximum storage of W n at each stage n, the total inventory at the end of any month cannot exceed W n. Hence, for each month it must be true that x n + d n - D n < W n, orBecause there is a maximum storage of W n at each stage n, the total inventory at the end of any month cannot exceed W n. Hence, for each month it must be true that x n + d n - D n < W n, or x n + d n < W n + D n (2) x n + d n < W n + D n (2) The amount produced in any given month cannot exceed the production capacity for that month, orThe amount produced in any given month cannot exceed the production capacity for that month, or d n < P n (3) d n < P n (3)

16 Slide Dicom Corporation n Optimal Value Function f n ( x n ) = optimal return (minimal cost) for stages 1 through n given one starts stage n with x n computers in inventory f n ( x n ) = MIN { r n ( x n, d n ) + f n -1 ( x n -1 )} f n ( x n ) = MIN { r n ( x n, d n ) + f n -1 ( x n -1 )} d n d n where the d n is constrained by restrictions (1), (2), and (3).

17 Slide Dicom Corporation n Stage 1 Starting at stage 1 (November), since f 0 ( x 0 ) = 0 is a boundary condition, then, f 1 ( x 1 ) = MIN r 1 ( x 1, d 1 ) f 1 ( x 1 ) = MIN r 1 ( x 1, d 1 ) d 1 d 1 Using the expression for r 1 ( x 1, d 1 ) and restrictions (1), (2), and (3), Using the expression for r 1 ( x 1, d 1 ) and restrictions (1), (2), and (3), f 1 ( x 1 ) = MIN 5 x d f 1 ( x 1 ) = MIN 5 x d s.t. x 1 + d 1 > 3 (1) s.t. x 1 + d 1 > 3 (1) x 1 + d 1 < 11 (2) x 1 + d 1 < 11 (2) d 1 < 5 (3) d 1 < 5 (3) and, d 1 > 0 and, d 1 > 0

18 Slide Example: Dicom Corporation n Tabulated Values for 50 d 1 +5 x d 1 d 1 x d 1 * f 1 ( x 1 ) x d 1 * f 1 ( x 1 ) Note that having x 1 > 3 would result in Dicom having computers in inventory at the end of November which is undesirable. Note that having x 1 > 3 would result in Dicom having computers in inventory at the end of November which is undesirable.

19 Slide Example: Dicom Corporation n Stage 2 f 2 ( x 2 ) = MIN 5 x d f 1 ( x 1 ) f 2 ( x 2 ) = MIN 5 x d f 1 ( x 1 ) d 2 d 2 Given the restrictions, the subproblem is: f 2 ( x 2 ) = MIN 5 x d f 1 ( x 2 + d 2 - 5) f 2 ( x 2 ) = MIN 5 x d f 1 ( x 2 + d 2 - 5) s.t. x 2 + d 2 > 5 (1) s.t. x 2 + d 2 > 5 (1) x 2 + d 2 < 13 (2) x 2 + d 2 < 13 (2) d 2 < 4 (3) d 2 < 4 (3) and d 2 > 0 and d 2 > 0

20 Slide Example: Dicom Corporation n Tabulated Values for 5 x d f 1 ( x 1 ) d 2 d 2 x d 2 * f 2 ( x 2 ) x 2 + d 2 *-5 = x 1 x d 2 * f 2 ( x 2 ) x 2 + d 2 *-5 = x Note that x 2 = 0 is infeasible.

21 Slide Example: Dicom Corporation n Stage 3 f 3 ( x 3 ) = MIN 5 x d f 2 ( x 2 ) d 3 f 3 ( x 3 ) = MIN 5 x d f 2 ( x 2 ) d 3 Given the restrictions, the subproblem is: f 3 ( x 3 ) = MIN 5 x d f 2 ( x 3 + d 3 - 1) f 3 ( x 3 ) = MIN 5 x d f 2 ( x 3 + d 3 - 1) s.t. x 3 + d 3 > 1 (1) s.t. x 3 + d 3 > 1 (1) x 3 + d 3 < 9 (2) x 3 + d 3 < 9 (2) d 3 < 3 (3) d 3 < 3 (3) and d 3 > 0 and d 3 > 0

22 Slide Dicom Corporation n Tabulated Values for 5 x d f 2 ( x 3 + d 3 -1) d 3 d 3 x d 3 * f 3 ( x 3 ) x 3 + d 3 *-1 = x 2 x d 3 * f 3 ( x 3 ) x 3 + d 3 *-1 = x

23 Slide Dicom Corporation n Stage 4 f 4 ( x 4 ) = MIN 5 x d f 3 ( x 3 ) f 4 ( x 4 ) = MIN 5 x d f 3 ( x 3 ) d 4 d 4 Given the restrictions, the subproblem is: Given the restrictions, the subproblem is: f 4 ( x 4 ) = MIN 5 x d f 3 ( x 4 + d 4 - 2) f 4 ( x 4 ) = MIN 5 x d f 3 ( x 4 + d 4 - 2) s.t. x 4 + d 4 > 2 (1) s.t. x 4 + d 4 > 2 (1) x 4 + d 4 < 10 (2) x 4 + d 4 < 10 (2) d 4 < 4 (3) d 4 < 4 (3) and d 4 > 0 and d 4 > 0

24 Slide Dicom Corporation n Tabulated Values for 5 x d f 3 ( x 4 + d 4 - 2) Since August starts with x 4 = 0 inventory on hand, compute table only for x 4 = 0. d 4 d 4 x d 4 * f 4 ( x 4 ) x 4 + d 4 *-2 = x 3 x d 4 * f 4 ( x 4 ) x 4 + d 4 *-2 = x

25 Slide Dicom Corporation n Solution Summary Working backwards through the tables, one can determine the optimal solution that gives the minimum cost of $327 x 100,000 = $32,700,000. Inventory On- Inventory On- Hand Beginning Hand Beginning Stage Month Produce ( d n *) Next Month x n -1 Stage Month Produce ( d n *) Next Month x n -1 4 August August September September October October November November 2 0

26 Slide The End of Chapter 18