1. IENG 362
Markov Chains

2. Inventory Example
Transition Matrix P =
080
184
368
632
264 .

3. 2-step transitions
Suppose we start with 3 in inventory. What is the probability we will have 1 in inventory after 2 time periods (2 steps)? P X t { | } + = 2 1 3

4. 2-step transitions
Suppose we start with 3 in inventory. What is the probability we will have 1 in inventory after 2 time periods (2 steps)? P X t { | } + = 2 1 3 = + P X t { | } 2 1 3

5. 2-step transitions
Suppose we start with 3 in inventory. What is the probability we will have 1 in inventory after 2 time periods (2 steps)? P X t { | } + = 2 1 3 = + P X t { | } 1 3 2

6. 2-step transitions P X { | } = 1 3 = p31p33 + p21p32 + p11p31 + p01p30
= p31p p21p p11p31 + p01p30

7. 2-step transitions
Suppose we start with 3 in inventory. What is the probability we will have 1 in inventory after 2 time periods (2 steps)? P X t { | } + = 2 1 3
= p31p p21p p11p31 + p01p30

8. Chapman-Kolmogorov Eqs.X t { | } + = 2 1 3
= p31p p21p p11p31 + p01p30 p k 31 2 3 1 ( ) = å

9. Chapman-Kolmogorov Eqs.31 2 3 1 ( ) = å
In general, p ij n ik m k M kj ( ) = - å

10. Aside; Matrix Multiplication00 01 02 03 10 11 12 13 20 21 22 23 30 31 32 33 x
a31 = p31p p21p p11p31 + p01p30

11. Aside; Matrix Multiplication00 01 02 03 10 11 12 13 20 21 22 23 30 31 32 33 x
a31 = p31p p21p p11p31 + p01p30 = P X t { | } + = 2 1 3

12. Chapman-Kolmogorov Eqs.ij n ik m k M kj ( ) = - å
In matrix form, P n ( ) =