1. EE255/CPS226 Discrete Time Markov Chain (DTMC)
Dept. of Electrical & Computer engineering
Duke University
11/30/2018

2. Discrete Time Markov Chain
Markov process: dynamic evolution is such that future state depends only on the present (past is irrelevant).
Markov Chain  Discrete state (or sample) space.
DTMC : time (index) is also discrete i.e. system is observed only at discrete intervals of time.
X0, X1, .., Xn, .. :observed state (of a particular ensemble} member (of the sample space) at discrete times, t0, t1,..,tn, ..
{X0, X1, .., Xn , ..} describes the states of a DTMC
Xn = j  system state at time step n is j. Then for a DTMC,
P(Xn = in| X0 = i0, X1 = i1, …, Xn-1 = in-1) = P(Xn = in| Xn-1 = in-1)
pj(n)  P(Xn = j) (pmf), or,
pjk(m,n)  P(Xn = k | Xm = j ),
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

3. Transition Probability
pjk(m,n): probability transition function of a DTMC.
Homogeneous DTMC: pjk(m,n) = pjk(m-n) i.e., transition probabilities exhibit stationary property. For such a DTMC,
1-step transition prob, pjk = pjk(1) = P(Xn = k| Xn-1 = j) ,
Assuming 0-step transition prob as:
Joint pmf is given by, P(X0 = i0, X1 = i1, …, Xn = in)
= P(X0 = i0, X1 = i1, …, Xn-1 = in-1). P(Xn = in| X0 = i0, X1 = i1, …, Xn-1 = in-1)
= P(X0 = i0, X1 = i1, …, Xn-1 = in-1). P(Xn = in| Xn-1 = in-1) (due to Markov prop)
= P(X0 = i0, X1 = i1, …, Xn-1 = in-1).pin-1, in :
= pi0(0)pi0, i1 (1) …pin-1, in (1) = pi0(0)pi0, i1 …pin-1, in
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

4. Transition Probability Matrix
The initial prob. is, pi0(0) = P(X0 = i0 ). In general,
p0(0) = P(X0 = 0 ), …, pk(0) = P(X0 = k ) etc, or,
p(0) = [p0(0), p1(0), … ,pk(0), ….] (initial prob. vector)
This allows us to define transition prob. matrix as,
Sum of ith row elements pi,0(0)+ pi,1(0)+ … ?
Any such sq. matrix with non-negative entries whose row sum =1 is called a stochastic matrix.
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

5. State Transition Diagram
pij : describes random state value evolution from i to j
Node with labels i, j etc. and an arc labeled pij
Concept of ri reward (cost or penalty) for each state I allows evaluation of various interesting performance measures.
Example: 2-state DTMC for a cascade binary comm. channel. Signal values: ‘0’ or ‘1’ form the state values. i j
pij
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

6. Total Probability Finding total pmf:
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

7. n-Step Transition Probability
For a DTMC, find
Events: state reaches k (from i) & reaches j (from k) are independent due to the Markov property (i.e. no history)
Invoking total probability theorem:
Let P(n) : n-step prob. transition matrix (i,j) entry is pij(n). Making m=1, n=n-1 in the above equation,
We are interested in the event i  j, which is actually made up of two independent events, Ik and kj. Therefore, pij = pik.pkj
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

8. Marginal pmf
j, in general can assume countable values, 0,1,2, …. Defining,
pj(n) for j=0,1,2,..,j,… can be written in the vector form as,
Or,
Pn can be easily computed if n is finite. However, if n is countably infinite, it may not be possible to compute Pn (and p(n) ).
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

9. Marginal pmf Example the n-step transition prob. Matrix is given by,
For a 2-state DTMC described by its 1-step transition prob. matrix,
the n-step transition prob. Matrix is given by,
Proof follows easily by using induction, that is, assuming that the above is true for Pn-1. Then, Pn = P. Pn-1
P(n) elements are:
p00(n) p01(n)
p10(n) p11(n)
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

10. Computing Marginal pmf
Previous example of a cascade digital comm channels: each stage described by a 2-state DTMC, We want to find p(n) (a=0.25 & b=0.5),
The ’11’ element for n=2 and n=3 are,
Assuming initial pmf as, p(0) = [p0(0) p1(0)] = [1/3 2/3] gives,
What happens to Pn as n becomes very large ( infinity)?
if ‘1’ enters the 0th stage and comes out as ‘1’ after the 2nd stage with the prob. P(X2=1|X0=1) = 3/8
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

11. DTMC State Classification
From the previous example, as n becomes infinity, pij(n) becomes independent of n and i ! Specifically,
Not all Markov chains may exhibit this behavior.
State classification may be based on the distinction that asymtotically:
some states may be visited infinitely many times. Whereas, some other states may be visited only a small number of times
Transient state: iff there is non-zero probability that the system will NOT return to this state.
Define Xji to be the # of visits to state i, starting from state j, then,
For a transient state (i), visit count needs to finite, which requires pji(n)  0 as n  infinity. Eventually, the system will always leave state i.
E[Xji] = sum_{n=0}^\infty pji(n) interpretation:
In 1-step = pji (1)
In 2-steps = pji (2) i.e. j  k (pjk) and then ki (pjk) or pji(2) = pjk.pki
In 1-step = pji (3) i.e. j  k1, k1k2 and then k2i etc.
Starting in state ‘j’, state ‘i’ can be visited via a number of paths (infinite many in general). Each of these paths contributes v_n=1 to the visit count. The probability of taking a particular path is pji (n), n=0,1,2,….. Therefor mean number of visits to state ‘i’ is = sum_{n=0}^{\infty} v_n pji (n).
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

12. DTMC State Classification (contd.)
State i is a said to be recurrent iff, starting from state i, the process eventually returns to the state i with probability 1.
For a recurrent state, time-to-return is a relevant measure. Define fij(n) as the cond. prob. that the first visit to j from i occurs in exactly n steps.
If j = i, then fii(n) denotes the prob. of returning to i in exactly n steps.
Known result:
Let,
Mean recurrence time for state i is
fii(n) : prob. Of returning to ‘i’ in n-steps, starting from state ‘i’.
Interpret this known result. For k=1, process takes 1-step (for ij) AND the remaining (n-1) steps are spent in the state j
If k=2, it takes 2-steps to move from i to j & process spends rest of (n-2) steps in state j etc.
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

13. Recurrent state
Let i be recurrent and pii(n) > 0, for some n > 0.
For state i, define period di as GCD of all such +ve n’s that result in pii(n) > 0
If di=1,  aperiodic and if di>1, then periodic.
Absorbing state: state i absorbing iff pii=1.
Communicating states: i and j are said to be communicating if there exits directed paths from i and j and from j and i.
Closed set of states: A commutating set of states C forms a closed set, if no state outside of C can be reached from any state in C.
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

14. Irreducible Markov Chains
Markov chain states partitioned into two distinct subsets: c1, c2, .., ck-1, ck , such that
ci, i=1,2,..k-1 are closed set of recurrent nun-null.
ck transient states.
If ci contain only one state, then ci’s form a set absorbing states
If k=2 and ck empty, then c1 forms an irreducible Markov chain
Irreducible Markov chain: is one in which every state can be reached from every other state in a finite no. of steps, i.e., for all i,j ε I, for some integer n > 0, pij(n) > 0. Examples:
Cascade of digital comm. channels is 1
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

15. Irreducible Markov Chains (contd.)
If one state is recurrent aperiodic, then so are all the other states. Same result if periodic or transient.
For a finite aperiodic irreducible Markov chain, pij(n) becomes independent of i and n as n goes to infinity.
All rows of Pn become identical
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

16. Irreducible Markov Chains (contd.)
Law of total probability gives,
Therefore, 1st eq. can be rewritten as,
In the matrix form,
v is a probability vector, therefore,
Self reading exercise (theorems on pp. 351)
For an aperiodic, irreducible, finite state DTMC,
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

17. Irreducible Markov Chain Example
Typical computer program: continuous cycle of compute & I/O
The resulting DTMC is irreducible with period =1. Therefore,
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

18. Sojourn Time
If Xn = i, then Xn+1 = j should depend only on the current state i, and not on the time spent in state i.
Let Ti be the time spent in state i, before moving to state j
DTMC will remain in state i in the next step with prob. pii and,
Next step (n+1), toss a coin, ‘H’ Xn+1 = i, ‘T’Xn+1 # i
At each step, we perform a Bernoulli trial. Then,
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

19. Markov Modulated Bernoulli Process
Generalization of a Bernoulli process: the Bernoulli process parameter is controlled by a DTMC.
Simplest case is Binary state (on-off) modulation
‘On’ Bernoulli param = c1; ‘Off’  c2’ (or =0)
Controlling process is an irreducible DTMC, and,
Reward assignment, r0 = c1 and r1=c2. This gives,
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

20. Examples of Irreducible DTMCs
Example 7.14: non-homogeneous DTMC for s/w reliability
Slotted ALOHA wireless multi-access protocol
Advantages:
2X more efficient than pure Aloha.
Automatically adapts to changes in station population
Disadvantages:
Throughput maximum of 36.8% theoretical limit.
Requires queuing (buffering) for re-transmission
Synchronization.
Advantages:
Doubles the efficiency of Aloha.
Adaptable to a changing station population.
Disadvantages:
Theoretically proven throughput maximum of 36.8%.
Requires queueing buffers for retransmission of packets.
Synchronization required.
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

21. Slotted ALOHA DTMC New and backlogged requests
Successful channel access iff :
Exactly one new req. and no backlogged req.
Exactly one backlogged req. and no new req.
DTMC state: # of backlogged requests.
backlogged
new n + + x
m-n + + x x Σ
Channel
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

22. Slloted Aloha contd.
In a particular state n, successful contention occurs with prob. rn
rn may be assigned as a reward for state n.
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

23. Discrete-time Birth-Death Processes
Special type of DTMC in which P has a tri-diagonal form,
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

24. DTMC solution steps
Solving for v = vP, gives the steady state probabilities.
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

25. Finite DTMCs with Absorbing States
Example: Program having a set of interacting modules. Absorbing state: failure state ( ps5 : unreliability)
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

26. Finite DTMCs, Absorbing States (contd.)
M contains useful information.
Xij : rv denoting random number to visits to j starting from i
E[Xij] = mij (for i, j = 1,2,…, n-1) . Need to prove this statement.
There are three distinct situations that can be enumerated
Let rv Y denote the state at step # (initial state: i)
E[Xij| y = n] = δij
E[Xij| y = k] = E[Xkj + δij]= E[Xkj]+ δij {
δij , occurs with prob. pij
Xkj + δij , occurs with prob. Pik
k=1,2,..n
(δij : term accounts for i=j case)
Xij = si sk sj sn i
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

27. Finite DTMCs, Absorbing States (contd.)
Since, P(Y=k) = pik , k=1,2,..n, total expectation rule gives,
Over all (i,j) values, we need to work with the matrix,
Therefore, fundamental matrix M elements give the expected # of visit to state j (from i) before absorption.
If the process starts in state “1”, then m1j gives the average # of visits to state j (from the start state) before absorption.
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

28. Performance Analysis, Absorbing States
By assigning rewards values to different state, a variety of performance measures may be computed.
Average time to execute a program
s1 is the start state; rjk : (fractional) execution time/visit for sj
Vj = m1j is the average # times statement block sjis executed
We need to calculate total expected execution time, I.e. until the process gets absorbed into stop state (s5 )
Software reliability: jth reward = Rj: Reliability of sj .Then,
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University