1. CS723 - Probability and Stochastic Processes

2. Lecture No. 45

3. In Previous Lecture
Finished analysis of Markov chains with finite and countable infinite state space
Random walks on discretized spaces of 1, 2, and higher dimensions
Started working with continuous-time Markov chains
Analysis of arrival process using Poisson distribution

4. Cont-time Markov Chains
Continuous-time Markov chains can change their state at any time 4

5. Cont-time Markov Chains
Arrival process of customers 5

6. Cont-time Markov Chains
Customers arrive one at a time
Average rate of arrival remains constant at  customers/second
Probability of arrival in t, a short interval of time, is t
Customers arrive independent of each other 6

7. Cont-time Markov Chains
Prob. of k customers in [0,t]
Arrival time of k-th customer
Inter-arrival times Tk=Yk-Yk-1 are i.i.d. 7

8. Cont-time Markov Chains
Rate of change of probability value
Probability of state change from k-1 8

9. Cont-time Markov Chains
Incremental time model of a continuous-time Markov chain and the transition probabilities 9

10. Cont-time Markov Chains
Probability of being in state k at t +t 10

11. Cont-time Markov Chains
Probability of being in state k at t +t 11

12. Cont-time Markov Chains
Solving for P0(t) using P0(0)=1
Solving for Pk(t) 12

13. Cont-time Markov Chains
Birth and death type processes 13

14. Cont-time Markov Chains
Population increases or decreases by no more than 1
Birth rate k may depend upon the current size of population
Death rate k may depend upon the current population size
Births and deaths occur independent of each other 14

15. Cont-time Markov Chains
Incremental time model of a birth and death type process 15

16. Cont-time Markov Chains
Probability of state change from k 16

17. Cont-time Markov Chains
Arrival process of customers with
n = and n=0 17

18. Cont-time Markov Chains
Customers in a single queue with
n = and n=  18

19. Cont-time Markov Chains
General birth and death process with possible extinction 0 = 0 = 0 19

20. Cont-time Markov Chains
A continuous-time Markov chain is transient if
M/M/1 queue is transient if  < 
M/M/k queue is transient if k < 
M/M/∞ queue is never transient 20

21. Cont-time Markov Chains
Stationary probability distribution of a continuous-time Markov chain
Rate of change of Pn(t) should be 0 21

22. Cont-time Markov Chains
Recursive solution gives
Since (n)=1 22

23. Cont-time Markov Chains
Stationary probability distribution of M/M/1 queue
Expected length of the queue 23