1. CS723 - Probability and Stochastic Processes

2. Lecture No. 38

3. In Previous Lecture
1-step transition probabilities and associated stochastic matrix P
n-step transition probabilities as entries in nth power of matrix P
Probability distribution of Xn from initial distribution of X0
Unique convergence of n-step transition probability matrix with all non-zero entries

4. Markov Chains
Properties of n-step transition probability matrix

5. Markov Chains
If all entries in Pm = Pm are non-zero, all entries in Pn=Pn , for n ≥ m, will also be non-zero

6. Markov Chains
State y can be reached from state x in n steps if Pn(x,y) > 0

7. Markov Chains
n-step reachability

8. Markov Chains

9. Markov Chains

10. Markov Chains

11. Markov Chains

12. Markov Chains

13. Markov Chains
All examples represent recurrent chains but only first has a convergent stochastic matrix
The second and third examples never have a stochastic matrix with all non-zero entries
A stochastic matrix with all non-zero entries may converge under a set of easily verifiable conditions

14. Markov Chains If Pn converges to a unique matrix P∞
Is true for any initial distribution
Hence,  is left-side eigenvector of P with eigenvalue of 1

15. Markov Chains For convergence,  = P∞ implies
that  is a left eigenvector of P∞ with an eigenvalue of 1

16. Markov Chains
Eigenvalues: 1, -0.1

17. Markov Chains
Eigenvalues: 1, 0.57,

18. Markov Chains
Eigenvalues: 1, 0.5, -0.5, -1

19. Three eigenvalues with magnitude = 1
Markov Chains
Three eigenvalues with magnitude = 1

20. Markov Chains
Finding Pn = Pn using diagonalization of transition probability matrix P

21. Markov Chains
Higher powers of D

22. Markov Chains
Higher powers of D

23. Markov Chains