1. Continuous Time Markov Chains and Basic Queueing Theory
EE384X Review 4
Winter 2004

2. Review: DTMC
pij is the transition probability from i to j over one time slot
The time spent in a state is geometrically distributed
Result of the Markov (memoryless) property
When there is a jump from state i, it goes to state j with probability

3. Continuous Time Versionj k
qij
qik
qij is the transition rate from state i to state j

4. CTMC
Upon entering state i, a random timer Tij»Exp(qij) is started for each potential transition i!j
These timers are independent of each other
Recall that Exponential distribution is memoryless
When the first timer expires, the MC makes the corresponding transition
Let Ti be the time spent in state i, and qi=åj¹i qij, then Ti » Exp(qi)
When there is a transition, the probability of jumping to state j is qij /qi

5. Definitions
{X(t):t¸0} is a continuous time Markov chain if P{X(s+t)=j | X(u); u·s} = P{X(s+t)=j | X(s)}
Similar to Discrete Time MCs, Continuous Time MCs have stationary distribution p
Exists when Markov chain is positive recurrent and irreducible

6. Stationary Distribution
Balance equations:
Transition rates in and out of state i are equal
Define matrix transition rate Q = (qij) with qii= -qi , then p Q = 0, where p is a row vector
Together with åi p(i) = 1, can solve for p

7. Queueing Theory Notation
A/S/s/k
A is the arrival process, e.g., Geometric, Poisson, Deterministic
S is the service distribution, e.g., Geometric, Exponential, Deterministic
s is the number of servers, e.g., 1, N, 1
k is the buffer size (if k is absent, then k = 1)
E.g., Geom/M/1, M/M/1, M/D/1, M/M/1

8. M/M/1 Queue Arrivals are Poisson with rate l1 2 3 l m
Arrivals are Poisson with rate l
Inter-arrival times are exp(l)
Services are exponential with rate m
These are also transition rates for the Markov chain
This looks very similar to Geom/Geom/1 queue, but different

9. Solving M/M/1 Queue We have pi l = pi+1 m
Let r = l/m, then pi = pi-1 r = p0 ri
If r < 1, the stationary distribution exists: pi = (1 - r) ri
Average Queue size:

10. M/M/1 Queue
NQ is the queue size, excluding the one in service:

11. M/M/1 Queue Customer arrival process is Poisson(l)1 2 3 l m 2m 3m 4m
Customer arrival process is Poisson(l)
All customers are served in parallel »exp(m)
Departure rate proportional to # of customers

12. Solving M/M/1 Queue
We have pi-1 l = pi i m
Let r = l/m, then
Thus

13. M/M/1 Queue
The queue size distribution of the M/M/1 queue is Poisson(r)
Therefore the average queue size is E(Q)=r
What’s the condition for the queue to be recurrent?