CS723 - Probability and Stochastic Processes

Lecture No. 43

In Previous Lecture Partitioning of stochastic matrix P into sub-matrices PR and Q and convergence of sub-matrices Analysis of transient or short-term behavior of transient Markov chains using indicator functions Average number of pre-absorption visits to a transient state j as (I , j) entry of (I - Q) -1

Markov Chains States 0 and 1 are recurrent states 4

Markov Chains 5

Markov Chains 6

Markov Chains 7

Markov Chains 8

Markov Chains 9

Markov Chains 10

Markov Chains 11

Markov Chains T1 is the first return times of a recurrent states j T2 is the second return times of a recurrent states j

Markov Chains From laws of large numbers There will be k visits to state j in a total of kE[ T ] steps but the ratio of these quantities is (j)

Markov Chains 14

Markov Chains 15

Markov Chains 16

Markov Chains 17

Markov Chains Markov chains with countably infinite sample space Transition probability matrix is of infinite size with entries Sum of a ‘row’ gives 1

Markov Chains Example: random walk on natural numbers with reflection at 1

Markov Chains Example: a queue of customers served by one operator; queue is observed after every second 20

Markov Chains 21

Markov Chains Example: a random walk on 2-D grid 22

Markov Chains A Markov chains with infinite state space is recurrent if every state is visited infinitely often 23