1. CS723 - Probability and Stochastic Processes

2. Lecture No. 43

3. In Previous Lecture
Partitioning of stochastic matrix P into sub-matrices PR and Q and convergence of sub-matrices
Analysis of transient or short-term behavior of transient Markov chains using indicator functions
Average number of pre-absorption visits to a transient state j as (I , j) entry of (I - Q) -1

4. Markov Chains
States 0 and 1 are recurrent states 4

5. Markov Chains 5

6. Markov Chains 6

7. Markov Chains 7

8. Markov Chains 8

9. Markov Chains 9

10. Markov Chains 10

11. Markov Chains 11

12. Markov Chains T1 is the first return times of a recurrent states j
T2 is the second return times of a recurrent states j

13. Markov Chains From laws of large numbers
There will be k visits to state j in a total of kE[ T ] steps but the ratio of these quantities is (j)

14. Markov Chains 14

15. Markov Chains 15

16. Markov Chains 16

17. Markov Chains 17

18. Markov Chains Markov chains with countably infinite sample space
Transition probability matrix is of infinite size with entries
Sum of a ‘row’ gives 1

19. Markov Chains
Example: random walk on natural numbers with reflection at 1

20. Markov Chains
Example: a queue of customers served by one operator; queue is observed after every second 20

21. Markov Chains 21

22. Markov Chains
Example: a random walk on 2-D grid 22

23. Markov Chains
A Markov chains with infinite state space is recurrent if every state is visited infinitely often 23