1. Industrial Engineering Dep
Lecture 2: Algorithmic Methods for transient analysis of continuous time Markov Chains
Dr. Ahmad Al Hanbali
Industrial Engineering Dep
University of Twente

2. Lecture 2: transient analysis of continuous time Markov chains
This Lecture deals with continuous time Markov processes as opposed to discrete time Markov chains in Lecture 1
Objectives:
Find equilibrium distribution
Find transient probabilities
Matrix decomposition
Uniformization method
Find Transient measures
Lecture 2: transient analysis of continuous time Markov chains

3. Background (1)
Let {𝑋(𝑡): 𝑡 ≥ 0} denote a continuous time stochastic process of state space {0,1,…,𝑁}
𝑋(𝑡) is a Markov chain if the conditional transition probability, for every 𝑡, 𝑠≥ 0 and 𝑗
𝑃(𝑋(𝑠+𝑡)=𝑗 | 𝑋(𝑢); 𝑢≤ 𝑠 )=𝑃(𝑋(𝑠+𝑡)=𝑗 | 𝑋(𝑠) )
𝑋(𝑡) is homogeneous (or stationary) if
𝑃(𝑋(𝑠+𝑡)=𝑗 | 𝑋(𝑠)=𝑖 ) = 𝑃(𝑋(𝑡)=𝑗 | 𝑋(0)=𝑖) = 𝑝𝑖𝑗(𝑡)
𝑋(𝑡) is irreducible if all states can communicate
Lecture 2: transient analysis of continuous time Markov chains

4. Background (2)
Define (infinitesimal) transition rate from state i to j of a Markov process
𝑞𝑖𝑗 = lim 𝑡→0 𝑝𝑖𝑗 𝑡 𝑡 , 𝑖 ≠ 𝑗
Let {𝑇𝑛 : 𝑛=0,1,.. } denote epochs of transition of CTMC then for 𝑛≥ 0 (by convention 𝑇0=0)
𝑃 𝑇𝑛− 𝑇 𝑛−1 ≤𝑥 | 𝑋( 𝑇 𝑛−1 )=𝑖, 𝑋(𝑇𝑛)=𝑗 =1−exp⁡(−𝑎𝑖𝑥),
where 𝑎𝑖 (= ∑𝑗≠𝑖 𝑞𝑖𝑗 ) is the total outgoing rate of state 𝑖
𝑎 𝑖 = lim 𝑡→0 1−𝑝𝑖𝑖 𝑡 𝑡
Lecture 2: transient analysis of continuous time Markov chains

5. Lecture 2: transient analysis of continuous time Markov chains
Background (3)
Let 𝑞𝑖𝑖 =−𝑎𝑖. The matrix 𝑄 = [𝑞𝑖𝑗]0≤𝑖,𝑗≤𝑁 is called the generator of the continuous time Markov chain (CTMC). Note: ∑𝑗 𝑞𝑖𝑗 = 0
Let 𝑝=(𝑝0,…,𝑝𝑁) equilibrium probabilities. The equilibrium equations of CTMC gives
𝑝 𝑖 𝑗≠𝑖 𝑞 𝑖𝑗 = 𝑗≠𝑖 𝑝 𝑗 𝑞 𝑗𝑖 ,
in matrix equation 𝑝𝑄=0, 𝑝𝑒=1
Idea: take advantage of Methods developed for discrete time Markov chains (in Lecture 1)
Lecture 2: transient analysis of continuous time Markov chains

6. An equivalent discrete time Markov chain
Equilibrium distribution 𝑝 can be obtained from an equivalent Markov chain via an elementary transformation. Let ∆ be real number such that
0<∆≤ min 𝑖 (−1/𝑞𝑖𝑖) , and 𝑃=𝐼+∆𝑄
𝑃 is a stochastic matrix, i.e., its entries are between 0 and 1, and its rows sum to 1. Further,
𝑝𝑃=𝑝 ⟺ 𝑝𝑄=0
The Markov chain with transition probability 𝑃 is a discretization of the Markov process of 𝑄 with time step ∆
Lecture 2: transient analysis of continuous time Markov chains

7. Uniformization of CTMC
To have same mean sojourn time in all states per visit the uniformization of CTMC introduces fictitious transitions from states to themselves
Let 0<∆≤ 𝑚𝑖𝑛𝑖(−1/𝑞𝑖𝑖), introduce a fictitious transition from state 𝑖 to itself with rate (𝑞𝑖𝑖+1/∆). This yields:
Equilibrium distribution of Q doesn't change
Outgoing rate from state i becomes (𝑞𝑖𝑖+1/∆−𝑞𝑖𝑖=1/∆) same for all states
Equilibrium distribution of the uniformized Markov process of Q is same as the Markov chain of transition matrix P(=I+∆Q) embedded at epoch of transitions (jumps)
The transitions of the uniformized process take place according to a Poisson process with rate ∆
Lecture 2: transient analysis of continuous time Markov chains

8. Equilibrium distribution
All methods developed for solving the equilibrium equation for discrete time Markov chain can be applied to the uniformized Markov chain of transition matrix P
Lecture 2: transient analysis of continuous time Markov chains

9. Transient Behavior of CTMC
Kolmogorov's equations are needed for the transient analysis. Let define the transient probability
𝑝𝑖𝑗(𝑡) = 𝑃(𝑋(𝑡)=𝑗 | 𝑋(0)=𝑖)
Then, for 0 ≤ 𝑠 < 𝑡
𝑝𝑖𝑗(𝑡) =∑𝑘 𝑝𝑖𝑘(𝑠) 𝑝𝑘𝑗(𝑡−𝑠)
Kolmogorov's equations are set of differential equations for 𝑝𝑖𝑗(𝑡)
Lecture 2: transient analysis of continuous time Markov chains

10. Lecture 2: transient analysis of continuous time Markov chains
Background (1)
Let 𝑃(𝑡) the matrix of (𝑖,𝑗) entries 𝑝𝑖𝑗(𝑡)
Kolmogorov's forward equations are derived by letting s approaches t from below (backward equations)
𝑃 ′ 𝑡 =𝑃 𝑡 𝑄,𝑃 0 =𝐼
Hence, 𝑃(𝑡)=𝑃 0 𝑛≥0 𝑄𝑡 𝑛 𝑛! = exp 𝑄𝑡
Truncating the infinite sum is inefficient since 𝑄 has positive and negative elements
Lecture 2: transient analysis of continuous time Markov chains

11. Matrix decomposition method
Let 𝑙𝑖, 𝑖 = 0,…,𝑁, be the (𝑁+1) eigenvalues of 𝑄
Let 𝑦𝑖 and 𝑥𝑖 be the left and right eigenvectors corresponding to 𝑙𝑖, such that 𝑦𝑖 𝑥𝑖 = 1 and 𝑦𝑖 𝑥𝑗 = 0 for 𝑖≠𝑗.
The matrix then reads
𝑄=𝑋𝐿 𝑋 −1 ,
where 𝑋 −1 is the matrix whose rows are 𝑦𝑖, 𝐿 is the diagonal matrix of entries 𝑙𝑖, and 𝑋 is the matrix whose columns are 𝑥𝑖
Lecture 2: transient analysis of continuous time Markov chains

12. Matrix decomposition method (cnt'd)
The transient probability matrix then reads
𝑃(𝑡)= 𝑛≥0 𝑄𝑡 𝑛 𝑛! =𝑋.𝑒𝑥𝑝⁡(𝐿𝑡). 𝑋 −1 =∑𝑖 𝑥𝑖. 𝑒𝑥𝑝 𝑙 𝑖 𝑡 .𝑦𝑖
What is the interpretation of 𝑃(∞)?
What conditions li should satisfy when t tends infinity ?
Disadvantage of matrix decomposition?
Due to Eingenvalues Gershgorin theorem all eigenvalues of Q have a non-positive real part.
Lecture 2: transient analysis of continuous time Markov chains

13. Uniformization method
Let 0<∆≤ min 𝑖 (−1/𝑞𝑖𝑖) and 𝑃=𝐼+∆𝑄. Conditioning on 𝑌, number of transitions in (0,𝑡) which is Poisson distributed with mean 𝑡/Δ, gives
𝑃 𝑡 = 𝑛≥0 𝑃 𝑌=𝑛 𝑃 𝑛 = 𝑛≥0 exp −𝑡/Δ 𝑡/Δ 𝑛 𝑛! 𝑃 𝑛
Truncating the latter sum on the first 𝐾 terms gives a good approximation, 𝐾=max⁡{20, 𝑡/∆+5 𝑡/∆ }
It is better to take the largest possible value of ∆= min 𝑖 (−1/𝑞𝑖𝑖)
Lecture 2: transient analysis of continuous time Markov chains

14. Lecture 2: transient analysis of continuous time Markov chains
Occupancy Time: mean
Occupancy time of a state is the sojourn time in that state during (0,𝑇). Note that depends on state at time 0
Let 𝑚𝑖𝑗(𝑇) denote the mean occupancy time in state 𝑗 during (0,𝑇) given initial state 𝑖. Then,
𝑚 𝑖𝑗 𝑇 =𝐸 𝑡=0 𝑇 1 𝑋 𝑡 =𝑗|𝑋 0 =𝑖 𝑑𝑡 = 𝑡=0 𝑇 𝐸 1 𝑋 𝑡 =𝑗|𝑋 0 =𝑖 𝑑𝑡 = 𝑡=0 𝑇 𝑝 𝑖𝑗 𝑡 𝑑𝑡
In matrix equation,
𝑀 𝑇 = 𝑚 𝑖𝑗 (𝑇) = 𝑡=0 𝑇 𝑝 𝑖𝑗 𝑡 𝑑𝑡 = 𝑡=0 𝑇 𝑃 𝑡 𝑑𝑡
Lecture 2: transient analysis of continuous time Markov chains

15. Mean occupancy time (cnt'd)
Using the uniformized process (∆,P) then,
𝑀 𝑇 = 𝑡=0 𝑇 𝑛≥0 exp −𝑡/Δ 𝑡/Δ 𝑛 𝑛! 𝑃 𝑛 𝑑𝑡 = 𝑛≥0 𝑡=0 𝑇 exp −𝑡/Δ 𝑡/Δ 𝑛 𝑛! 𝑑𝑡 𝑃 𝑛 .
Note 𝑡=0 𝑇 exp −𝑡/Δ 𝑡/Δ 𝑛 𝑛! 𝑑𝑡=Δ 1−𝑃 𝑌≤𝑛 , where Y is a Poisson random variable of mean 𝑡/Δ
We find that
𝑀 𝑇 =Δ 𝑛≥0 1−𝑃 𝑌≤𝑛 𝑃 𝑛 .
Note 𝑛≥0 𝑃 𝑛 does not converge so do not split up the latter sum to compute M(T).
Lecture 2: transient analysis of continuous time Markov chains

16. Cumulative distribution of occupancy time
Let 𝑂(𝑇) denote the total sojourn time during [0,T] in a subset of states, 𝛺 𝑜 . Then, for 0≤𝑥<𝑇
𝑃 𝑂 𝑇 ≤𝑥 = 𝑛=0 ∞ 𝑒 −𝑇/Δ 𝑇/Δ 𝑛 𝑛! 𝑘=0 𝑛 𝛼 𝑛,𝑘 𝑗=𝑘 𝑛 𝑛 𝑗 𝑥 𝑇 𝑗 1− 𝑥 𝑇 𝑛−𝑗 ,
𝑃 𝑂 𝑇 =𝑇 = 𝑛=0 ∞ 𝑒 −𝑇/Δ 𝑇/Δ 𝑛 𝑛! 𝛼 𝑛,𝑛+1 ,
where 𝛼(𝑛,𝑘) is the probability that uniformized process visits 𝑘 times Ω𝑜 during [0,𝑇] given that it makes 𝑛 transitions.
Proof, see for details Tijms 2003:
Condition on Poisson number of transitions of the uniformized process to be n
Occupancy time is smaller than x if uniformized process will visit k times 𝛺 𝑜 out of the n visits and at least k of these transitions happens before x. The former probability is 𝛼 𝑛,𝑘 and latter is function of a binomial distribution
𝛼 𝑛,𝑘 can be computed recursively. Note they are dependent on the initial position of the chain at time 0.
Lecture 2: transient analysis of continuous time Markov chains

17. Moments of occupancy time
Proposition: The m-th moment of O(T) is given by:
𝐸 𝑂 𝑇 𝑚 𝑇 𝑚 = 𝑛=0 ∞ 𝑒 −𝑇/Δ 𝑇/Δ 𝑛 𝑛+𝑚 ! 𝑘=1 𝑛+1 𝛼 𝑛,𝑘 𝑙=𝑘 𝑘+𝑚−1 𝑙
Proposition: Given that the chain starts in equilibrium the second moment of the occupancy time in the subset Ω 0 during [0,T] gives
𝐸 𝑂 𝑇 𝑇 2 = 𝑛=1 ∞ 𝑒 −𝑇/Δ 𝑇/Δ 𝑛 𝑛+2 ! 𝑝 0 𝑖=1 𝑛 𝑛−𝑖+1 𝑃 𝑖 𝑒 0 + 𝑙∈ Ω 0 𝑝 𝑙 𝑒 −𝑇/Δ +𝑇/Δ−1 𝑇/Δ ,
where 𝑝𝑖 is the steady state probability of the Markov chain in state 𝑖, 𝑝0 is the column vector with i-th entry equal to 𝑝𝑖 if 𝑖∈ Ω 𝑜 and zero otherwise, and 𝑒0 is the column vector with i-th entry equal to 1 if 𝑖∈ Ω 𝑜 and zero otherwise.
For proofs see: A. Al Hanbali, M.C. van der Heijden. Interval Availability Analysis of a Two-echelon, Multi-Item System. European Journal of Operational Research (EJOR), vol. 228, issue 3, , 2013
Lecture 2: transient analysis of continuous time Markov chains

18. References
V.G. Kulkarni. Modeling, analysis, design, and control of stochastic systems. Springer, New York, 1999
Tijms, H. C. A first course in stochastic models. New York: Wiley, 2003