Solutions Markov Chains 1 1) Given the following one-step transition matrices of a Markov chain, determine the classes of the Markov chain and whether they are recurrent. a. b. 1 2 3 All states communicate, all states recurrent 1 2 3 Communicate {0}, {1, 2}, {3} states 0, 1, 2 recurrent state 3 transient

Problems Markov Chains 2 2) Suppose we have 3 pints of blood on hand at the end of the day. At start, we get another pint making 4 pints total. During the period we have a demand for 0, 1, 2, or 3 pints of blood with the following distribution. x 0 1 2 3 P(x) .4 .3 .2 .1 Assume we receive 1 pint of blood each day at the start of the day. We can hold no more than 5 pints of blood at the end of any given day. Excess blood over 5 pints will be discarded. Construct a one-step transition probability matrix to model this process as a Markov chain. Soln: Let S = the pints of blood at the end of each day. Since, we can hold more than 5, S = 0, 1, 2, 3, 4, 5 Pints available at start of day = S+1 S Day Start Event Probability End S 5 6 0 demand 0.4 5 threw one away 5 6 1 0.3 5 5 6 2 0.2 4 5 6 3 0.1 3 4 5 0 0.4 5 4 5 1 0.3 4 etc.

Problems Markov Chains 3 Soln: Let S = the pints of blood at the end of each day. Since, we can hold more than 5, S = 0, 1, 2, 3, 4, 5 Pints available at start of day = S+1 S Day Start Event Probability End S 5 6 0 demand 0.4 5 threw one away 5 6 1 0.3 5 5 6 2 0.2 4 5 6 3 0.1 3 4 5 0 0.4 5 4 5 1 0.3 4 etc.

Solutions Markov Chains 4 Louise Ciccone, a dealer in luxury cars, faces the following weekly demand distribution. Demand 0 1 2 Prob. .1 .5 .4 She adopts the policy of placing an order for 3 cars whenever the inventory level drops to 2 or fewer cars at the end of a week. Assume that the order is placed just after taking inventory. If a customer arrives and there is no car available, the sale is lost. Show the transition matrix for the Markov chain that describes the inventory level at the end of each week if the order takes one week to arrive. Soln S= inventory level at week’s end = 1, 2, 3, 4, 5 note: we can have at most 5 cars in the lot, 2 cars at week’s end followed by no sales followed by the arrival of 3 cars at the end of the week. note: we could start with 0 cars, but once an arrival of 3 cars comes we will never get back to 0 cars since we never sell more than 2 and we get below 2 at time t, 3 arrive at t+1

Solutions Markov Chains 4 Soln cont. State Event End State Probability 5 sell 0 5 .1 5 sell 1 4 .5 5 sell 2 3 .4 4 sell 0 4 .1 4 sell 1 3 .5 4 sell 2 2 .4 3 sell 0 3 .1 3 sell 1 2 .5 3 sell 2 1 .4 2 3 arrive at week’s end sell 0 5 .1 sell 1 4 .5 sell 2 3 .4 1 3 arrive at week’s end sell 0 4 .1 sell 1 3 .5+.4 since demand of 1 or 2 deletes inventory during the week

Solutions Markov Chains 8 Soln cont.