Chapman-Kolmogorov Equations 𝑃 0𝑗 (𝑛−𝑚) 𝑃 𝑖0 (𝑚) 𝑃 𝑖𝑗 (𝑛) = 𝑃 𝑖0 (𝑚) 𝑃 0𝑗 (𝑛−𝑚) + 𝑃 𝑖1 (𝑚) 𝑃 1𝑗 (𝑛−𝑚) +…+ 𝑃 𝑖𝑚 (𝑚) 𝑃 𝑚𝑗 (𝑛−𝑚) = 𝑘=0 𝑀 𝑃 𝑖𝑘 (𝑚) 𝑃 𝑘𝑗 (𝑛−𝑚) ex： (3 states) 𝑃 12 (2) = 𝑃 10 𝑃 12 + 𝑃 11 𝑃 12 + 𝑃 12 𝑃 22 𝑃 00 𝑃 01 𝑃 02 𝑃 10 𝑃 11 𝑃 12 𝑃 20 𝑃 21 𝑃 22 𝑃 00 𝑃 01 𝑃 02 𝑃 10 𝑃 11 𝑃 12 𝑃 20 𝑃 21 𝑃 22 = ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗

Chapman-Kolmogorov Equations (continued)

n-step Transition Matrices for the Weather Example

n-step Transition Matrices for the Weather Example (continued)

n-step Transition Matrices for the Inventory Example

Unconditional State Probabilities 𝑃 𝑋 2 =3 =𝑃 𝑋 0 =0 P 03 (2) +𝑃 𝑋 0 =1 P 13 (2) +𝑃 𝑋 0 =2 P 23 (2) +𝑃 𝑋 0 =3 P 33 (2) =1 ‧ P 33 (2) = 0.165

Classification of States of a Markov Chain Accessible => ｉ →　ｊ 2 → 3 3→2

Classification of States of a Markov Chain (continued)

Classification of States of a Markov Chain (continued) I ↔ j , j ↔ k , I ↔ k Weather 0,1 inventory{ 0,1,2,3} (在2 states後) Gambling 1,2 0 3

Recurrent States and Transient States Gambling ： 3(X) ↗ 2 ↘ 1

Recurrent States and Transient States (continued) Gambling：state 0,3

Recurrent States and Transient States (continued)

Recurrent States and Transient States (continued) 0 1 2 3 4 1 2 3 4

Periodicity Properties $2->$1,$1->$2 Stock ↓↑

Periodicity Properties (continued) Ergodic =Recurrent + aperiopdic