CS723 - Probability and Stochastic Processes

Lecture No. 38

In Previous Lecture 1-step transition probabilities and associated stochastic matrix P n-step transition probabilities as entries in nth power of matrix P Probability distribution of Xn from initial distribution of X0 Unique convergence of n-step transition probability matrix with all non-zero entries

Markov Chains Properties of n-step transition probability matrix

Markov Chains If all entries in Pm = Pm are non-zero, all entries in Pn=Pn , for n ≥ m, will also be non-zero

Markov Chains State y can be reached from state x in n steps if Pn(x,y) > 0

Markov Chains n-step reachability

Markov Chains

Markov Chains

Markov Chains

Markov Chains

Markov Chains

Markov Chains All examples represent recurrent chains but only first has a convergent stochastic matrix The second and third examples never have a stochastic matrix with all non-zero entries A stochastic matrix with all non-zero entries may converge under a set of easily verifiable conditions

Markov Chains If Pn converges to a unique matrix P∞ Is true for any initial distribution Hence,  is left-side eigenvector of P with eigenvalue of 1

Markov Chains For convergence,  = P∞ implies that  is a left eigenvector of P∞ with an eigenvalue of 1

Markov Chains Eigenvalues: 1, -0.1

Markov Chains Eigenvalues: 1, 0.57, - 0.07

Markov Chains Eigenvalues: 1, 0.5, -0.5, -1

Three eigenvalues with magnitude = 1 Markov Chains Three eigenvalues with magnitude = 1

Markov Chains Finding Pn = Pn using diagonalization of transition probability matrix P

Markov Chains Higher powers of D

Markov Chains Higher powers of D

Markov Chains