1. IENG 362
Markov Chains

2. Motivation; Inventory
Consider (S,s) inventory system:

3. Motivation; Inventory
Consider (S,s) inventory system:
order at s=0
Dt = demand in per. t
Xt = inv. on hand at t

4. Motivation; Inventory
Consider (S,s) inventory system:
order at s=0
Dt = demand in per. t
Xt = inv. on hand at t
Inventory = 3, Dt =2, Inventory End = 1

5. Motivation; Inventory
Consider (S,s) inventory system:
order at s=0
Dt = demand in per. t
Xt = inv. on hand at t
Inventory = 2, Dt =3, Inventory End = 0

6. Motivation; Inventory
Consider (S,s) inventory system:
order at s=0
Dt = demand in per. t
Xt = inv. on hand at t
Inventory = 0, Dt =2, Inventory End = 3 - 2

7. Motivation; Inventory
Consider (S,s) inventory system:
order at s=0
Dt = demand in per. t
Xt = inv. on hand at t
max (Xt - Dt , 0) , Xt > 1
Xt+1 =
max (3 - Dt , 0) , Xt = 0

8. Inventory Example Recall, Xt = 0, 1, 2, 3 Let
Pij = P{End Inv. = j | Start Inv. = i)
= P{Xt+1= j | Xt =i)

9. Events and Probabilities
Current Next
State Events Probability State
S=3 Demand = P33 S=3
Demand = P32 S=2
Demand = P31 S=1
Demand = P30 S=0
Demand > P30 S=0
S=2 Demand = P22 S=2
Demand = P21 S=1
Demand = P20 S=0
Demand > P20 S=0

10. Inventory Example p p p p p p p p P = p p p p p p p p
P =transition matrix showing probabilities from
state i to state j. p p p p 00 01 02 03 p p p p P = 10 11 12 13 p p p p 20 21 22 23 p p p p 30 31 32 33
Pij = P{Xt+1=j | Xt = i)

11. Computing Transition Prob.X D t 10 1 = + { | }

12. Computing Transition Prob.X D t 10 1 = + { | }
Let’s suppose that demand follows a Poisson
distribution with l =1.

13. Computing Transition Prob.X D t 01 1 = + { | }
Let’s suppose that demand follows a Poisson
distribution with l =1. P D x e t { } ! . = - l 1
36788

14. Computing Transition Prob.X D t 10 1 = + { | } P D x e t { } ! = - 1 p P D e x
t+1 10 1 2 = + - å { } ! .

15. Computing Transition Prob.X D t 10 1 = + { | } P D x e t { } ! = - 1 p 10 P D e
t+1 1 0!
632 = - { } .

16. Computing Transition Prob.X D t 20 1 2 = - + { | }

17. Computing Transition Prob.e 1 0!
264 = - + p P X D t 20 2 { | } [ ! ] .

18. Computing Transition Prob.
Class Exercise:
Compute the following transition probabilities
a. p21 d. p00
b. p22 e. p01
c. p23

19. Inventory Example
Transition Matrix P =
080
184
368
632
264 .

20. Example; Weather Let State 0 = dry 1 = rain
p00 = P{dry today | dry yesterday} = 0.7
p01 = P{rain today | dry yesterday} = 0.3
p10 = P{dry today | rain yesterday} = 0.5
p11 = P{rain today | rain yesterday} = 0.5

21. Example; Weather
Let State
0 = dry
1 = rain P = 7 3 5 .

22. Example; Weather II Let State 0 = dry today and yesterday
1 = dry today and rain yesterday
2 = rain today and dry yesterday
3 = rain today and yesterday
p00 = P{dry today & yesterday | dry yesterday & day before}
= 0.9

23. Example; Weather Let State 0 = dry today and yesterday
1 = dry today and rain yesterday
2 = rain today and dry yesterday
3 = rain today and yesterday
p01 = P{dry today & rain yesterday | dry yesterday & day before}
= not possible

24. Example; Weather
State
0 = dry today and yesterday
1 = dry today and rain yesterday
2 = rain today and dry yesterday
3 = rain today and yesterday . 9 . . 1 . . 6 . . 4 . P = . . 5 . . 5 . . 3 . . 7

25. Example; Gambling
Gambler bets $1 with each play. He wins $1 with probability p and loses $1 with probability 1-p. Game ends when he wins $3 or goes broke.

26. Example; Gambling
Gambler bets $1 with each play. He wins $1 with probability p and loses $1 with probability 1-p. Game ends when he wins $3 or goes broke.
Let,
Xt = money on hand = 0, 1, 2, 3

27. Example; Gambling
Gambler bets $1 with each play. He wins $1 with probability p and loses $1 with probability 1-p. Game ends when he wins $3 or goes broke.
Let,
Xt = money on hand = 0, 1, 2, 3 P p = - 1

28. Classification of States
Communicate
If Pij(n) > 0 for some n, i,j communicate
For gambler’s ruin, P p = - 1

29. Classification of States
Communicate
If Pij(n) > 0 for some n, i,j communicate
Properties,
1. Any state communicates with itself.
2. If i communicates with j, j communicates with i
3. If i comm. with j, j comm. with k, then i comm. with k.

30. Classification of States
Irreducible
all states communicate.
Inventory example, P =
080
184
368
632
264 . 1 2 3

31. Classification of States
Irreducible
all states communicate.
Gambler’s ruin, P p = - 1 1 2 3

32. Classification of States
Absorbing
If the one-step transition probability equals 1.0
pii = 1 P p = - 1 2 3

33. Period
Gambler’s ruin
It is possible to enter state 1 at times t = 2, 4, 6, . . .
period = 2 P p = - 1 2 3

34. Period
The period of state I is defined to be the integer t (t > 1) such that Pii(n) = 0 for all values of n other than
t, 2t, 3t, . . .
State with period = 1 is aperiodic