IENG 362 Markov Chains

Inventory Example Transition Matrix P = 080 184 368 632 264 .

2-step transitions Suppose we start with 3 in inventory. What is the probability we will have 1 in inventory after 2 time periods (2 steps)? 3 1 2 t t+1 t+2 P X t { | } + = 2 1 3

2-step transitions ; found by finding probability of all paths from state 3 to state 1 P X t { | } + = 2 1 3 3 1 2 t t+1 t+2

2-step transitions P X t { | } + = 2 1 3 = + P X t { | } 2 1 3

2-step transitions P X t { | } + = 2 1 3 = + P X t { | } 1 3 2

2-step transitions P X { | } = 1 3 = p31p33 + p21p32 + p11p31 + p01p30 = p31p33 + p21p32 + p11p31 + p01p30

2-step transitions Suppose we start with 3 in inventory. What is the probability we will have 1 in inventory after 2 time periods (2 steps)? P X t { | } + = 2 1 3 = p31p33 + p21p32 + p11p31 + p01p30

Chapman-Kolmogorov Eqs. X t { | } + = 2 1 3 = p31p33 + p21p32 + p11p31 + p01p30 p k 31 2 3 1 ( ) = å

Chapman-Kolmogorov Eqs. 31 2 3 1 ( ) = å In general, p ij n ik m k M kj ( ) = - å

Aside; Matrix Multiplication 00 01 02 03 10 11 12 13 20 21 22 23 30 31 32 33 x a31 = p31p33 + p21p32 + p11p31 + p01p30

Aside; Matrix Multiplication 00 01 02 03 10 11 12 13 20 21 22 23 30 31 32 33 x a31 = p31p33 + p21p32 + p11p31 + p01p30 = P X t { | } + = 2 1 3

Chapman-Kolmogorov Eqs. ij n ik m k M kj ( ) = - å In matrix form, P n ( ) =