1. Discrete-time markov chain (continuation)

2. CHAPMAN-KOLMOGOROV EQUATIONS IN MATRIX FORM
Starting from the transition matrix 𝐏, we have 𝐏 (𝟐) =𝐏𝐏= 𝐏 𝟐 𝐏 (𝟑) =𝐏 𝐏 (𝟐) = 𝐏 𝟑 𝐏 (𝟒) =𝐏 𝐏 (𝟑) = 𝐏 𝟒 In general, 𝐏 (𝒏) =𝐏 𝐏 (𝒏−𝟏) = 𝐏 𝒏

3. CHAPMAN-KOLMOGOROV EQUATIONS IN MATRIX FORM
Recall our example with 𝐏= 𝟎 𝟎.𝟕 𝟎 𝟎.𝟑 𝟎.𝟐 𝟎 𝟎.𝟖 𝟎 𝟎.𝟗 𝟎 𝟎.𝟏 𝟎 𝟎 𝟎.𝟎𝟓 𝟎 𝟎.𝟗𝟓 Refer to the MS Excel file.

4. Unconditional state probabilities
If we start with 𝑿 𝟎 =𝟐, what are the probabilities 𝑷 𝑿 𝟏𝟎 =𝟏 , 𝑷 𝑿 𝟏𝟎 =𝟐 , 𝑷 𝑿 𝟏𝟎 =𝟑 and 𝑷{ 𝑿 𝟏𝟎 =𝟒}? After 10 time periods

5. Unconditional state probabilities
After 10 matrix multiplications: 𝐏 (𝟏𝟎) ≈ 𝟎.𝟏𝟐 𝟎.𝟏𝟕 𝟎.𝟏𝟐 𝟎.𝟔𝟎 𝟎.𝟏𝟔 𝟎.𝟏𝟐 𝟎.𝟏𝟕 𝟎.𝟓𝟓 𝟎.𝟏𝟗 𝟎.𝟏𝟑 𝟎.𝟏𝟏 𝟎.𝟓𝟔 𝟎.𝟎𝟗 𝟎.𝟏𝟎 𝟎.𝟎𝟖 𝟎.𝟕𝟑

6. Unconditional state probabilities
If we start with 𝑿 𝟎 =𝟐, what are the probabilities 𝑷 𝑿 𝟏𝟎 =𝟏 , 𝑷 𝑿 𝟏𝟎 =𝟐 , 𝑷 𝑿 𝟏𝟎 =𝟑 and 𝑷{ 𝑿 𝟏𝟎 =𝟒}? 𝟎 𝟏 𝟎 𝟎 𝟎.𝟏𝟐 𝟎.𝟏𝟕 𝟎.𝟏𝟐 𝟎.𝟔𝟎 𝟎.𝟏𝟔 𝟎.𝟏𝟐 𝟎.𝟏𝟕 𝟎.𝟓𝟓 𝟎.𝟏𝟗 𝟎.𝟏𝟑 𝟎.𝟏𝟏 𝟎.𝟓𝟔 𝟎.𝟎𝟗 𝟎.𝟏𝟎 𝟎.𝟎𝟖 𝟎.𝟕𝟑

7. Unconditional state probabilities
If we start with 𝑿 𝟎 =𝟐, the probabilities are 𝑷 𝑿 𝟏𝟎 =𝟏 =𝟎.𝟏𝟔, 𝑷 𝑿 𝟏𝟎 =𝟐 =𝟎.𝟏𝟐, 𝑷 𝑿 𝟏𝟎 =𝟑 =𝟎.𝟏𝟕 and 𝑷 𝑿 𝟏𝟎 =𝟒 =𝟎.𝟓𝟓.

8. STEADY-STATE PROBABILITIES
After 50 matrix multiplications: 𝐏 (𝟓𝟎) ≈ 𝟎.𝟏𝟏 𝟎.𝟏𝟏 𝟎.𝟏𝟎 𝟎.𝟔𝟕 𝟎.𝟏𝟏 𝟎.𝟏𝟏 𝟎.𝟏𝟎 𝟎.𝟔𝟕 𝟎.𝟏𝟏 𝟎.𝟏𝟏 𝟎.𝟏𝟎 𝟎.𝟔𝟕 𝟎.𝟏𝟏 𝟎.𝟏𝟏 𝟎.𝟏𝟎 𝟎.𝟔𝟕 What is the meaning of this?

9. STEADY-STATE PROBABILITIES
𝒑 𝒊𝟏 (𝟓𝟎) ≈𝟎.𝟏𝟏 𝒑 𝒊𝟐 (𝟓𝟎) ≈𝟎.𝟏𝟏 𝒑 𝒊𝟑 (𝟓𝟎) ≈𝟎.𝟏𝟎 𝒑 𝒊𝟒 (𝟓𝟎) ≈𝟎.𝟔𝟕 This is called the steady-state probabilities of the Markov Chain.

10. STEADY-STATE PROBABILITIES
𝒑 𝒊𝟏 (𝟓𝟎) ≈𝟎.𝟏𝟏
𝒑 𝒊𝟐 (𝟓𝟎) ≈𝟎.𝟏𝟏
𝒑 𝒊𝟑 (𝟓𝟎) ≈𝟎.𝟏𝟎
𝒑 𝒊𝟒 (𝟓𝟎) ≈𝟎.𝟔𝟕
Note: Do not be confused with steady-state probabilities and stationary transition probabilities.

11. STEADY-STATE PROBABILITIES
𝒑 𝒊𝟏 (𝟓𝟎) ≈𝟎.𝟏𝟏 𝒑 𝒊𝟐 (𝟓𝟎) ≈𝟎.𝟏𝟏 𝒑 𝒊𝟑 (𝟓𝟎) ≈𝟎.𝟏𝟎 𝒑 𝒊𝟒 (𝟓𝟎) ≈𝟎.𝟔𝟕 Can we derive them directly without doing too many matrix multiplications?

12. STEADY-STATE PROBABILITIES
In some cases, we can use the concept of fixed- point iteration. 𝛑=𝛑𝐏 The steady-state probabilities: 𝛑= 𝜋 𝟎 𝜋 𝟏 𝜋 𝟐 … 𝜋 𝑴 .

13. STEADY-STATE PROBABILITIES
Recall again our example:
𝜋 𝟏 𝜋 𝟐 𝜋 𝟑 𝜋 𝟒 = 𝜋 𝟏 𝜋 𝟐 𝜋 𝟑 𝜋 𝟒 𝟎 𝟎.𝟕 𝟎 𝟎.𝟑 𝟎.𝟐 𝟎 𝟎.𝟖 𝟎 𝟎.𝟗 𝟎 𝟎.𝟏 𝟎 𝟎 𝟎.𝟎𝟓 𝟎 𝟎.𝟗𝟓
We also include in our equations:
𝜋 𝟏 + 𝜋 𝟐 +𝜋 𝟑 + 𝜋 𝟒 =𝟏

14. STEADY-STATE PROBABILITIES
Solving it will result in: 𝜋 𝟏 ≈𝟎.𝟏𝟏𝟐𝟓 𝜋 𝟐 ≈𝟎.𝟏𝟏𝟐𝟓 𝜋 𝟑 ≈𝟎.𝟏 𝜋 𝟒 ≈𝟎.𝟔𝟕𝟓