1. Introduction to Stochastic Models GSLM 54100

2. Outline
continuous-time Markov chain 2

3. Continuous-Time Markov Chain
continuous-time analog of discrete-time Markov chain
objective: to deduce the long-term average cost per unit time
Discrete -
time
Markov chain { X n }
Continuous
time Markov chain { ( t )}
State
Markov
property
Stationary
transition
Discrete
Discrete P ( X n +1 = j | i , - 1
, …, )
" P ( X
t+s
) = j | t i , r x
), 0
< ) = ),
" s
(> 0) P ( X n +1 = j | i
) = p ij ,
" P ( X
t+s
) = j | t i p ij s )
" ,
(> 0) 3

4. Differences Between CTMC & DTMC
duration of a state
DTMC: one period
the consecutive duration staying in a state ~ Geo
CTMC: any positive real number
the consecutive duration staying in a state ~ exp
cost calculation
DTMC: cost per visit = cost per unit time at state
CTMC: cost per visit  cost per unit time at state 4

5. Cost Calculation in CTMC
ci = cost per unit at state i
long-term average cost per unit time
ci = cost per visit of state i
long-term average cost per unit time 5

6. Sojourn Times of CTMC
sojourn time: the time that a CTMC stays at a state in a visit
fact:
result of the Markov property
time
excellent
good
fair
bad 6

7. Markov Property of CTMC
P(X(t+s) = j |X(t) = i, X(r) = x(r) for 0  r < t) = P(X(t+s) = j | X(t) = i) for all i, j and for all t, s > 0
 given present (i.e., X(t)), the future (i.e., X(t+s)) and the past (i.e., X(r) = x(r) for 0  r < t) are independent
present
future
X(t) = i
X(t+s) = ? t
t+s
known states, X(r) = x(r), 0  r < t
past 7

8. Intuitive Justification of i.i.d. Exp Sojourn Times
Ti: the sojourn time of state i
just entering state i at t = 0
P(Ti > t) = P(Ti > t|X(0) = i) being a function of i
suppose X(r) = i for 0  r  t
P(Ti > t+s|X(r) = i, 0  r  t) = P(Ti > t+s|Ti > t)
Markov property  P(Ti > t+s|X(r) = i, 0  r  s) = P(Ti > t|X(0) = i) = P(Ti > t)
P(Ti > t+s|Ti > t) = P(Ti > t)  Ti ~ exp(qi) 8

9. Definition of a CTMC
{X(t)}, a continuous-time, discrete-state stochastic process is a CTMC if  
(a). sojourn times at state i, Ti ~ exp(qi), independent of everything else, 0 < qi < 
(b). whenever leaving state i, next visiting state j with probability pij > 0, j pij = 1
for the time being, assume pii = 0 for all i, because of the way that we define sojourn times 9

10. Procedure to Define a CTMC
defining three kinds of quantities
the states (and the state space)
qi for all i (where Ti ~ exp(qi)) and
pij for all i, j (where next visiting j with probability pij upon leaving i ) 10

11. Example 8.4.2 one machine, one repairman
working times of machine ~ i.i.d. exp()
repairing times of machine ~ i.i.d. exp()
working
repairing 11

12. state 1: one exp() and one exp() competing
Example 8.4.3
two machines, one repairman
working times of all machines ~ i.i.d. exp()
repairing times of all machine ~ i.i.d. exp()
state N(t) = number of working machines 1 2
N(t) t
state 2: two exp() competing
exp(2)
exp(+)
exp(+)
state 1: one exp() and one exp() competing
exp()
exp() 12
state 0: one exp() competing

13. Another Way to Define a CTMC
last example revealing that in a CTMC, events with exponential times to occur competing to be the first one
defining three kinds of quantities
the states (and the state space)
qij for all i, j (where Tij ~ exp(qij) is the duration of the activity that leads to state change from i to j)
qi, where Ti = min(Tij, j  i) ~ exp(qi) and qi = j qij 13

14. Two Ways to Define a CTMC
{qi, {pij}}  { Tij ~ exp(qij), Ti = min(Tij, j  i)}
Tij ~ exp(qij)
Ti = min(Tij, j  i)
qi = j qij,
pij = P(Ti = min(Tij, j  i)) = qij/qi  
occurrence of events following Poisson of rate qi
random partitioning by pij
time to change to state j,
Ti1 ~ exp(qi pi1), …
Tij ~ exp(qi pij)
qi, {pij} 
Ti ~ exp(qi) 14

15. Transition Diagram Example 8.4.2: N(t) = # of working machine at t 1   2 1 2   15

16. Birth and Death Processes
a CTMC such that qi,j = 0 for |i-j|  1
birth rate at state i: i
death rate at state i: i 2 1 2 1 1 0 … n 2 n
n1 3
n+1 n 16

17. Standard M/M/1 Queue
Poisson arrivals of rate , exponential service of rate , single server, infinite buffer 2 1   … n 17

18. M/M/1 Queue with Finite Buffer
Poisson arrivals of rate , exponential service of rate , single server, finite buffer of at most N customers in system 2 1   … N
N1 18

19. Standard M/M/c Queue
Poisson arrivals of rate , exponential service of rate , c servers, infinite buffer 2 1   … 3
c+1 c 2
c1
(c-1) c 19

20. M/M/c Queue with Finite Buffer
Poisson arrivals of rate , exponential service of rate , c servers, finite buffer of size N, N  c 2 1   …
c1
(c-1) 2 3 c c
c+1
N1 N 20

21. M/M/ Queue
Poisson arrivals of rate , exponential service of rate ,  number of servers, infinite buffer 2 1   … 2 3 3 4 4 5 21

22. Finite-Source Queue with Exponential Servers
K machines, i.i.d. working times ~ exp()
R repairmen, i.i.d. repairing times ~ exp() 2 1  K …
R1
(R-1) 2 3 R R
R+1
N1 2 N 
(K-1)
(K-2)
(K-R+2)
(K-R+1)
(K-R)
(K-R-1) 22

23. Limiting Behavior of a Positive Irreducible DTMC
an irreducible DTMC {Xn} is positive  there exists a unique nonnegative solution to
j: stationary (steady-state) distribution of {Xn} 23

24. Limiting Behavior of a Positive Irreducible DTMC
j = fraction of time at state j
j = fraction of expected time at state j
average cost
cj for each visit at state j
random i.i.d. Cj for each visit at state j
for aperiodic chain: 24

25. Limiting Behavior of a Positive Irreducible CTMC
if {pj} satisfy
cj = cost/time at j
dj = cost/visit at j 25

26. Intuitive Derivation of Expressions for Average Cost
for cj, cost per unit time
T: a large positive number
time at state j  pjT
cost induced by state j  cjpjT
total cost  j cjpjT
cost per unit time = j cjpj 26

27. Intuitive Derivation of Expressions for Average Cost
for dj, cost per visit
T: a large positive number
time at state j  pjT
mean time in state j per visit = 1/qj
approximate number of visits in pjT  qjpjT
cost induced by state j  cjqjpjT
total cost  j cjqjpj
cost per unit time = j cjqjpj 27

28. Examples
Example 8.6.1
stationary distributions of the queueing systems
M/M/1
M/M/c
M/M/
M/M/1 with finite buffer
M/M/c with finite buffer
finite-source queue with exponential servers 28

29. Expected Time Taken to Visit a State
0 = E(time taken to visit state 2 for the first time|X(0) = 0)
define 1 = E(time taken to visit state 2 for the first time|X(0) = 0) 2 1   2 29

30. Exercise
The arrivals of potential customers (cars) to a two-mechanic garage follow a Poisson process of rate . The garage has space for two cars; cars that find the garage full go away and never return. At any moment, a car service requires only one mechanic. 30

31. Exercise
(a). Suppose that service times are i.i.d. exponential random duration of rate , and that the service and the arrival processes are independent.
(i). Model the problem as a CTMC. You may draw out the transition diagram of the chain after you have defined the state and state space.
(ii). The chain should be positive. Find its stationary distribution.
(iii). Each served car pays a lump sum of $c for its service, and the garage pays each mechanic $r per unit time. Find the long-term net revenue rate ($ per unit time) of the garage.
(iv). In addition to (iii), suppose that the garage pays back a served car $h for each unit time that the car stays in the garage. (Woo! A car can actually earn some money if its service is longer than c/h.) Find the long- term net revenue rate of the garage in this case. 31

32. Exercise
Model the following parts as CTMCs. Each part should be modeled independently from others and from part (a). For each part, first define the state and state space and then draw its transition diagram.
(b). There are two. Any service completed by the first mechanic is of exp(1) and that by the second mechanic of exp(2). When a car comes, if only one mechanics is idle, the mechanic picks up the job; if both mechanics are idle, the car is assigned to the mechanic who has taken longer rest since last service completion. 32

33. Exercise
(c). There are two types of service, A and B, provided by the garage. For any of the two mechanics, a type A service ~ exp(A) while type B ~ exp(B). Each car needs only one type of service: type A with probability p, 0 < p < 1, independent of everything else, and type B otherwise. The service times are independent from each other and from the arrival process. 33