1 OR II GSLM 52800

2 Outline  introduction to discrete-time Markov Chain introduction to discrete-time Markov Chain  problem statement  long-term average cost pre unit time  solving the MDP by linear programming

3 States of a Machine StateCondition 0Good as new 1Operable – minor deterioration 2Operable – major deterioration 3Inoperable – output of unacceptable quality

4 Transition of States State /81/16 103/41/8 2001/

5 Possible Actions DecisionActionRelevant States 1Do nothing0, 1, 2 2 Overhaul (return to state 1) 2 3 Replace (return to state 0) 1, 2, 3

6 Problem  adopting different collections of actions leading to different long-term average cost per unit time  problem: to find the policy that minimizes the long-term average cost per unit time

7 Costs of Problem  cost of defective items  state 0: 0; state 1: 1000; state 2: 3000  cost of replacing the machine = 4000  cost of losing production in machine replacement = 2000  cost of overhauling (at state 2) = 2000

8 Policy R d : Always Replace When State  0  half of the time at state 0, with cost 0  half of the time at other states, all with cost 6000, because of machine replacement  average cost per unit time = /16 7/ /16

9 Long-Term Average Cost of a Positive, Irreducible Discrete-time Markov Chain  a positive, irreducible discrete-time Markov chain with M+1 states, 0, …, M  only M of the balance eqt plus the normalization eqt

10 Policy R a : Replace at Failure but Otherwise Do Nothing /16 7/8 1 1/2 1 1/16 3/4 1/8 1/2

11 Policy R b : Replace in State 3, and Overhaul in State /16 7/ /16 3/4 1/8 1

12 Policy R c : Replace in States 2 and /16 7/ /16 3/4 1/8 1 1

13 Problem R b, i.e., replacing in State 3 and overhauling in State 2  in this case the minimum-cost policy is R b, i.e., replacing in State 3 and overhauling in State 2  question: Is there any efficient way to find the minimum cost policy if there are many states and different types of actions?  impossible to check all possible cases

14 Linear Programming Approach for an MDP  let  D ik be the probability of adopting decision k at state i   i be the stationary probability of state i  y ik = P(state i and decision k)  C ik = the cost of adopting decision k at state i

15 Linear Programming Approach for an MDP

16 Linear Programming Approach for an MDP at optimal, D ik = 0 or 1, i.e., a deterministic policy is used

17 Linear Programming Approach for an MDP  actions possibly to adopt at state  0: do nothing (i.e., k = 1)  1: do nothing or replace (i.e., k = 1 or 3)  2: do nothing, overhaul, or replace (i.e., k = 1, 2, or 3)  3: replace (i.e., k = 3)  variables: y 01, y 11, y 13, y 21, y 22, y 23, and y 33

18 Linear Programming Approach for an MDP Stat e /81/16 103/41/8 2001/

19 Linear Programming Approach for an MDP  solving, y 01 = 2/21, y 11 = 5/7, y 13 = 0, y 21 = 0, y 22 = 2/21, y 23 = 0, y 33 = 2/21  optimal policy  at state 0: do nothing  state 1: do nothing  state 2: overhaul  state 3: replace