1. INFM 718A / LBSC 705 Information For Decision Making Lecture 9

2. Outline Markov chains –“Gambler’s ruin” –Expected time of an event –Steady-state probabilities

3. Markov Chains A Markov chain is a process that evolves from state to state at random. The probabilities of moving to a state are determined solely by the current state. Hence, Markov chains are “memoryless” processes.

4. Example: Gambler’s Ruin p. 444 in the textbook: You have $1. You play a game of luck, which can end in one of two states: you have $6, or you have $0. You bet $1 in each round, and earn $1 if you win, and lose your dollar if you lose the round. P (win a round) =.6; P (lose a round) =.4

5. Probability of Reaching a State What is the probability of reaching $6, if you start with i dollars? First let’s analyze the situation. Then we will use Excel/Solver to calculate the w(i)s. (Please refer to the solution on pp. 444-446 as necessary.)

6. Gambler’s Ruin $0$1$2$3$4$5$6 P= 0.60.6 P= 0.40.4 1 1

7. Gambler’s Ruin w(i) = P (state 6 will be rached | state I is occupied now) w(i) = 0.6 * w(i+1) + 0.4 * w(i-1) w(0) = 0 w(6) = 1 w(1) = 0.6 * w(2) + 0.4 * w(0) w(2) = 0.6 * w(3) + 0.4 * w(1) w(3) = 0.6 * w(4) + 0.4 * w(2) w(4) = 0.6 * w(5) + 0.4 * w(3) w(5) = 0.6 * w(6) + 0.4 * w(4)

8. Gambler’s Ruin w(0) = 0 w(6) = 1 - 0.6 * w(2) + w(1) - 0.4 * w(0) = 0 - 0.6 * w(3) + w(2) - 0.4 * w(1) = 0 - 0.6 * w(4) + w(3) - 0.4 * w(2) = 0 - 0.6 * w(5) + w(4) - 0.4 * w(3) = 0 - 0.6 * w(6) + w(5) - 0.4 * w(4) = 0

9. Expected Time of an Event Please refer to Problem B on pp. 447-449. 1/2 1/3 4 + T(B) 1 + T(C) 2 5 + T(C) 4 + T(A) 3 5 + T(B) 1 + T(A)

10. Expected Time of an Event  (A) = E[T(A)],  (B) = E[T(B)],  (C) = E[T(C)]  (A) = (1/2)*[4 + E[T(B)]] + (1/2)*[1 + E[T(C)]] = (1/2)*[4 +  (B)] + (1/2)*[1 +  (C)]  (B) = (1/3)*(2) + (1/3)*[4 +  (A)] + (1/3)*[5 +  (C)]  (B) = (1/3)*(3) + (1/3)*[1 +  (A)] + (1/3)*[5 +  (B)]

11. Expected Time of an Event  (A) - (1/2)*  (B) - (1/2)*  (C) = 5/2 - (1/3)*  (A) +  (B) - (1/3)*  (C) = 11/3 - (1/3)*  (A) - (1/3)*  (B) +  (C) = 9/3 Solve in Excel for  (A),  (B), and  (C)

12. Steady-state Probabilities What are the probabilities of being in each state over the long run (i.e., when those probabilities become steady.) –Using multiple transitions (refer to pp. 450- 453). –Using direct calculation (refer to p. 454). –Using “flux” (refer to pp. 455-456).