1. IENG 362
Markov Chains

2. Steady State Conditions
Proposition: For any irreducible ergodic M.C., lim Pij(n) exists and is independent of i.
lim ( ) n ij j P =
> p j i ij M P = å p j M = å p 1

3. Steady State Conditions
Heuristic p j n i M ij P X = å { } | ( )

4. Steady State Conditions
Now, P ij n ik k M kj ( ) + = å 1

5. Steady State Conditions
Now, P ij n ik k M kj ( ) + = å 1
Taking the limit as n p j k M kj P = å

6. å Example; Inventory p P = p 080 632 264 08 184 368 = + . P = 080 184
080
184
368
632
264 . p j k M kj P = å p 1 2 3
080
632
264 08
184
368 = + .

7. Example; Inventory p 1 2 3
080
632
264 08
184
368 = + . 1 2 3 = + p

8. Example; Inventory p 368 1 - = . ( ) p 582 = . p 080 632 264 08 1841 2 3
080
632
264 08
184
368 = + .
Choose p0 = 1 p 3
368 1 - = . ( ) p 3
582 = .

9. Example; Inventory p 080 632 264 08 184 368 = + . p 368 1 582 - = + .1 2 3
080
632
264 08
184
368 = + .
p0 = 1, p3 = 0.582 p 2
368 1
582 - = + . ( ) p 2
9212 = .

10. Example; Inventory p 080 632 264 08 184 368 = + . p 368 184 582 - = +1 2 3
080
632
264 08
184
368 = + .
p0 = 1, p3 = 0.582, p2 = p 1
368
184
582 - = + . (
) (.9212) ) p 1
1.0 = .

11. Example; Inventory p 1 2 3
000
921
582
503 = å .

12. å Example; Inventory p 000 921 582 503 = . p 503 285 263 166 / . Þ = 11 2 3
000
921
582
503 = å . p 1 2 3
503
285
263
166 / . Þ =

13. Example; Inventory p 1 2 3
285
263
166 . = j i ij M P = å p 1

14. Example; Inventory å p 285 263 166 . = P p 1 lim P = > p P . 2861 2 3
285
263
166 . = j i ij M P = å p 1 P ( ) . 16
286
263
166 =
lim ( ) n ij j P =
> p

15. Special Cases lim P d = m p
Irreducible positive recurrent periodic Markov Chain, period = d
lim ( ) n jj nd j P d = m p

16. Special Cases Ex: 1 = P If we start in state 0, P does not exist , .00 1 2 3 4 ( ) , .
lim =
Ex: 1 P = 1 1 1

17. Special Cases Ex: lim P = 1 p = 1 2 p 1 = P However, 1 1 1 ( ) n jj j= 2 1 p j = 1 2 p
Ex: 1 P = 1 1 1

18. { Cost Functions in M.C. C X ( ) , = 2 1 4 6 3
Suppose C(Xt) = cost of being in state Xt. In our inventory example, suppose it costs $2 / unit for each time period inventory is carried over. Then C X t ( ) , = 2 1 4 6 3 {

19. å Cost Functions ] [ E n C X ( ) 1
In general, the expected cost over all states and n time periods then is given by E n C X t ( ) 1 = å ] [

20. Cost Functions
Consider the first 4 time periods, with some arbitrary path. For the path given, 3 2 1
t = t = t = t = 3
Cost C X = + ( ) 1 2 3 6 4

21. Cost Functions
But the probability of incurring this cost is the probability associated with this path; i.e. P
Cost C X j { } ( ) = + 14 3 2 32 20 1 02 3 2 1
t = t = t = t = 3

22. Cost Functions
The expected cost is the cost associated with each path x the probability associated with each path. E
Cost P X k i C M [ ] { | } ( ) = å 1 3 2 1
t = t = t = t = 3

23. Cost Functions
The expected cost is the cost associated with each path x the probability associated with each path. E
Cost P X k i C M [ ] { | } ( ) = + å 1 2 3 4 3 2 1
t = t = t = t = 3

24. Cost Functions
But we know how to consider all possible paths to a state. Consider
Cost x
Prob P C X i = 2 ( ) 3 2 1 E
Cost P C X j t ij [ ] ( ) = å 2 3
t = t = t = t = 3

25. Cost Functions
But we know how to consider all possible paths to a state. Consider E n C X t [ ( ) ] 1 = å 3 2 1 n P C X j ij t M ( ) 1 = å = å 1 n P C X ij t j M ( )
t = t = t = t = 3

26. Cost Functions
The long run average cost then may be given by 1 n å E [
Cost ] =
lim E [ C ( X = j ) ] n t n j = 1 n P C X j M ij t ( ) = å 1 M å = p C ( j ) j j =

27. { Example; Inventory C X ( ) , = 2 1 4 6 3
Suppose C(Xt) = cost of being in state Xt. In our inventory example, suppose it costs $2 / unit for each time period inventory is carried over. Then C X t ( ) , = 2 1 4 6 3 {
t = 1
t = 

28. Example; Inventory
Suppose C(Xt) = cost of being in state Xt. In our inventory example, suppose it costs $2 / unit for each time period inventory is carried over. Then E
Cost [ ] C j M ( ) = å p
t =  2 4 6
t = 1

29. { Complex Costs Inventory; Xt-1 C X D ( , ) = + < £ > ³ 25 10 3
Suppose an order for x units is made at a cost of $ x. For unsatisfied demand, cost of a lost sale is $50.
If (s,S) = (0,3) inventory policy is used,
Xt-1 C X D t ( , ) - = +
<
> 1 25 10 3 50 {

30. Complex Costs E
Costs n P C X D ij t [ ] ( , )
lim = - å 1

31. Complex Costs å å lim E Costs n P C X D [ ] ( , ) = 1 k j ( ) = pij t [ ] ( , )
lim = - å 1
a miracle occurs k j M ( ) = å p

32. Example; Inventory å k E C D P ( ) [ , ] { } . 55 50 4 100 5 56 = + E
Cost [ ] k j M ( ) = å p
t =  2 4 6
p3 = .166
p2 = .264
p1 = .285
p0 = .285 k E C D P t ( ) [ , ] { } . 55 50 4
100 5 56 = +
t = 1

33. Example; Inventory å k E C D ( ) [ , ] 1 = = + 50 2 100 3 150 4 18 P D
Cost [ ] k j M ( ) = å p k E C D t ( ) [ , ] 1 =
t =  2 4 6
p3 = .166
p2 = .264
p1 = .285
p0 = .285 = + - 50 2
100 3
150 4 18 1 P D e t { } . ! 56
t = 1

34. Example; Inventory å k E C D ( ) [ , ] 2 = = + 50 3 100 4 150 5 5.2 P
Cost [ ] k j M ( ) = å p k E C D t ( ) [ , ] 2 =
t =  2 4 6
p3 = .166
p2 = .264
p1 = .285
p0 = .285 = + - 50 3
100 4
150 5
5.2 1 P D e t { } . !
18.4 56
t = 1

35. Example; Inventory å k E C D ( ) [ , ] 3 = = + 50 4 100 5 150 6 1.2 P
Cost [ ] k j M ( ) = å p k E C D t ( ) [ , ] 3 =
t =  2 4 6
p3 = .166
p2 = .264
p1 = .285
p0 = .285 = + - 50 4
100 5
150 6
1.2 1 P D e t { } . !
5.2
18.4 56
t = 1

36. Example; Inventory å = + 56 285 18 4 5 2 264 1 166 70 ( . ) $22 E Cost[ ] k j M ( ) = å p
1.2
t =  2 4 6
p3 = .166
p2 = .264
p1 = .285
p0 = .285 = + 56
285 18 4 5 2
264 1
166 70 ( . )
$22
5.2
18.4 56
t = 1