1. IENG 362
Markov Chains

2. Cost Functions in M.C.
Suppose C(Xt) = cost of being in state Xt. In our inventory example, suppose it costs $2 / unit for each time period inventory is carried over. Then C X t ( ) , = 2 1 4 6 3 {

3. å Cost Functions ] [ E n C X ( ) 1
In general, the expected cost over all states and n time periods then is given by E n C X t ( ) 1 = å ] [

4. Cost Functions
Consider the first 4 time periods, with some arbitrary path. For the path given, 3 2 1
t = t = t = t = 3
Cost C X = + ( ) 1 2 3 6 4

5. Cost Functions
But the probability of incurring this cost is the probability associated with this path; i.e. P
Cost C X j { } ( ) = + 14 3 2 32 20 1 02 3 2 1
t = t = t = t = 3

6. Cost Functions
The expected cost is the cost associated with each path x the probability associated with each path. E
Cost P X k i C M [ ] { | } ( ) = å 1 3 2 1
t = t = t = t = 3

7. Cost Functions
The expected cost is the cost associated with each path x the probability associated with each path. E
Cost P X k i C M [ ] { | } ( ) = + å 1 2 3 4 3 2 1
t = t = t = t = 3

8. Cost Functions
But we know how to consider all possible paths to a state. Consider
Cost x
Prob P C X i = 2 ( ) 3 2 1 E
Cost P C X j t ij [ ] ( ) = å 2 3
t = t = t = t = 3

9. Cost Functions
But we know how to consider all possible paths to a state. Consider E n C X t [ ( ) ] 1 = å 3 2 1 n P C X j ij t M ( ) 1 = å = å 1 n P C X ij t j M ( )
t = t = t = t = 3

10. Cost Functions
The long run average cost then may be given by 1 n å E [
Cost ] =
lim E [ C ( X = j ) ] n t n j = 1 n P C X j M ij t ( ) = å 1 M å = p C ( j ) j j =

11. Example; Inventory
Suppose C(Xt) = cost of being in state Xt. In our inventory example, suppose it costs $2 / unit for each time period inventory is carried over. Then C X t ( ) , = 2 1 4 6 3 {
t = 1
t = 

12. Example; Inventory
Suppose C(Xt) = cost of being in state Xt. In our inventory example, suppose it costs $2 / unit for each time period inventory is carried over. Then E
Cost [ ] C j M ( ) = å p
t =  2 4 6
t = 1

13. { Complex Costs Inventory; Xt-1 C X D ( , ) = + < £ > ³ 25 10 3
Suppose an order for x units is made at a cost of $ x. For unsatisfied demand, cost of a lost sale is $50.
If (s,S) = (0,3) inventory policy is used,
Xt-1 C X D t ( , ) - = +
<
> 1 25 10 3 50 {

14. Complex Costs E
Costs n P C X D ij t [ ] ( , )
lim = - å 1

15. Complex Costs å å lim E Costs n P C X D [ ] ( , ) = 1 k j ( ) = pij t [ ] ( , )
lim = - å 1
a miracle occurs k j M ( ) = å p

16. Example; Inventory å k E C D P ( ) [ , ] { } . 55 50 4 100 5 56 = + E
Cost [ ] k j M ( ) = å p
t =  2 4 6
p3 = .166
p2 = .264
p1 = .285
p0 = .285 k E C D P t ( ) [ , ] { } . 55 50 4
100 5 56 = +
t = 1

17. Example; Inventory å k E C D ( ) [ , ] 1 = = + 50 2 100 3 150 4 18 P D
Cost [ ] k j M ( ) = å p k E C D t ( ) [ , ] 1 =
t =  2 4 6
p3 = .166
p2 = .264
p1 = .285
p0 = .285 = + - 50 2
100 3
150 4 18 1 P D e t { } . ! 56
t = 1

18. Example; Inventory å k E C D ( ) [ , ] 2 = = + 50 3 100 4 150 5 5.2 P
Cost [ ] k j M ( ) = å p k E C D t ( ) [ , ] 2 =
t =  2 4 6
p3 = .166
p2 = .264
p1 = .285
p0 = .285 = + - 50 3
100 4
150 5
5.2 1 P D e t { } . !
18.4 56
t = 1

19. Example; Inventory å k E C D ( ) [ , ] 3 = = + 50 4 100 5 150 6 1.2 P
Cost [ ] k j M ( ) = å p k E C D t ( ) [ , ] 3 =
t =  2 4 6
p3 = .166
p2 = .264
p1 = .285
p0 = .285 = + - 50 4
100 5
150 6
1.2 1 P D e t { } . !
5.2
18.4 56
t = 1

20. Example; Inventory å = + 56 285 18 4 5 2 264 1 166 70 ( . ) $22 E Cost[ ] k j M ( ) = å p
1.2
t =  2 4 6
p3 = .166
p2 = .264
p1 = .285
p0 = .285 = + 56
285 18 4 5 2
264 1
166 70 ( . )
$22
5.2
18.4 56
t = 1