Discrete Time Markov Chains

Discrete Time Markov Chain (DTMC) Consider a system where transitions take place at discrete time instants, and where the state of the system at time k is denoted as Xk Markov property states that Future only depends on the present… The pij‘s are the state transitions probabilities System evolution is fully specified by its state transition probability matrix [P]ij = Pij

State Transition Probability Matrix P is a stochastic matrix All entries are positive and sum of entries in a row add up to 1 n-step transition probability matrix Pijn is the probability of being in state j after n steps when starting in state i Limiting probabilities: the odds of being in a given state after a very large (infinite) number of transitions

Chapman-Kolmogorov Equations Track state evolution based on transition probabilities between all states is the probability of being in state j after n transitions when starting in state i This provides a simple recursion to compute higher order transition probabilities (C-K)

More on C-K Equations In matrix form, where P is the one-step transition probability matrix The Chapman-Kolmogorov equations state Pn = Pm Pn-m, 0  m  n

Limiting Distribution From P we know that pij is the probability of being in state j after step 1 given that we start in state i In general, let be the initial sate probability vector Then the state probability vector 1 after the first step is given by 1 = 0P And in general after n steps, we have n = 0Pn Does n converges to a limit as n  ? (limiting distribution)

A Simple 3-State Example 1 2 1-p p2 p(1-p) p Note that the relation n = 0Pn implies that conditioned on starting in state i, the value of n is simply the ith row of Pn

A Simple 3-State Example Appears independent of the starting state i

Stationary Distribution A Markov chain with M states admits a stationary probability distribution  = [0, 1,…, M-1] if  P =  and {i=0 to M-1}i =1 In other words {i=0 to M-1}i pij = j ,  j and {i=0 to M-1}i =1

Stationary vs. Limiting Distributions Assume that the limiting distribution defined by  = [0, 1,…, M-1], where j = limnPijn > 0, exists and is a distribution, i.e., {i=0 to M-1}i =1 Then  is also a stationary distribution, i.e.,  = P and no other distribution exists We can find the limiting distribution of a DTMC either by solving the stationary equations or by raising the matrix P to some large power

Summary: Computing DTMC Probabilities Use the stationary equations:  = P and ii = 1 Numerically compute successive values of Pn (and stop when all rows are approximately equal) The limit converges to a matrix with identical rows, where each row is equal to  Guess a solution to the recurrence relation

Back to Our Simple 3-State Example 1 2 1-p p2 p(1-p) p

The Umbrella Example For p=0.6, this gives 1 2 1 (1-p) 1-p p For p=0.6, this gives  = [0.16667, 0.41667, 0.41667] Probability Pwet that professor gets wet = Prob[zero umbrella and rain] They are independent, so that Pwet = 0p = 0.16667  0.6 = 0.1 Average number of umbrellas at a location: E[U] = 1 + 22 = 1.25

Infinite DTMC Handling chains with an infinite state space We cannot anymore use matrix multiplication But the result that if the limiting distribution exists it is the only stationary distribution still holds So we can still use the stationary equations, provided they have “some structure” we can exploit

A Simple (Birth-Death) Example 1 2 3 4 1-r r s 1-r-s The (infinite) transition probability matrix and correspondingly the stationary equations are of the form

A Simple (Birth-Death) Example 1 2 3 4 1-r r s 1-r-s The stationary equations can be rewritten as follows 0 = 0(1-r)+1s  1 = (r/s)0 1 = 0r+1(1-r-s)+2s  2 = r/s1 = (r/s)20 2 = 1r+2(1-r-s)+3s  3 = r/s2 = (r/s)30 We can then show by induction that i = (r/s)i0, i  0 The normalization condition {i=0 to ∞}i =1 gives 0 = (1- r/s), and hence i = (r/s)i(1- r/s)

A Simple (Birth-Death) Example 1 2 3 4 1-r r s 1-r-s Defining  = r/s (a natural definition of “utilization” consistent with Little’s Law), the stationary probabilities are of the form 0 = (1- ) and i =  i(1- ) From those, we readily obtain E[N]=Σiii =  /(1- )

Putting Things on a Firmer Footing When does the limiting distribution exist? How does the limiting distribution compare to time averages (fraction of time spent in state j)? The limiting distribution is an ensemble average How does the average time between successive visits to sate j compare to j (the probability of being in state j)?

Some Definitions & Properties Period (of state j): GCD of integers n s.t. Pjjn>0 State is aperiodic if period is 1 A Markov chain is aperiodic if all states are aperiodic A Markov chain is irreducible if all states communicate For all i and j, Pijn>0 for some n State j is recurrent (transient) if the probability fj of starting at j and ever returning to j is =1 (<1) Number of visits to a recurrent (transient) state is infinite (finite) with probability 1 If state j is recurrent (transient), then ΣnPjjn =  (< ) In an irreducible Markov chain, states are either all transient or all recurrent A transient Markov chain does not have a limiting distribution Positive recurrence and null recurrence: A Markov chain is positive recurrent (null recurrent) if the mean time between returning to a state is finite (infinite) An ergodic Markov chain is aperiodic, irreducible and positive recurrent

Existence of the Limiting Distribution An irreducible, aperiodic DTMC (finite or infinite) is in either of the following two classes: States are either all transient or all null recurrent, in which case j = limnPijn= 0 for all j, and the stationary distribution does NOT exists States are all positive recurrent, the limiting distribution exists and is equal to the stationary distribution, with a positive probability for each state. In addition, j = 1/mjj, where mjj is the average time between successive visits to state j

Existence of Limiting Distribution 8 7 6 5 9 4 3 2 1 10 11 Absorbing Periodic Transient Positive recurrent, irreducible, aperiodic  ergodic Existence of limiting distribution depends on the structure of the underlying Markov chain Stationary state probabilities exist for ergodic Markov chains Many practical systems of interest give rise to ergodic Markov chains Stationary probabilities are of the form independent of  0

Time Averages For a positive recurrent, irreducible Markov chain, with probability 1 pj = limtNj(t)/t = 1/mjj > 0 Where Nj(t) is the number of transitions to state j by time t, and mjj is the average time between visits to state j pj is the time-average fraction of time is state j For an ergodic DTMC, with probability 1, pj = j = 1/mjj Time-average and stationary probabilities are equal

Probabilities and Rates For an ergodic DTMC j is the limiting probability of being in state j as well as the long-run fraction of time the chain is in state j i Pij can also be interpreted as the rate of transitions from state i to state j The stationary equation for state i gives us i = Σjj Pji but we also know i = i ΣjPij = Σji Pij So that we have Σjj Pji = Σji Pij or in other words, the total rate leaving state i equals the total rate entering state i = 1

Balance Equations For an ergodic Markov chain, the rate leaving any set of states equals the rate of entering that set of states Application to our earlier example We immediately get r1 = s2 1 2 3 4 1-r r s 1-r-s

Conservation Law – Single State k3 k3 pjk3 pk3j k4 k2 k4 k2 pjk4 pjk2 pk4j pk2j pjk5 pjk1  pk5j pk1j j j k5 k1 k5 k1 pjk6 pjk8 pk6j pk8j k6 k8 k6 k8 pjk7 pk7j k7 k7

Conservation Law – Set of States k4 k3 k2 k8 k7 k6 k1 k5 pj1k1 pj1k2 pj1k3 pj3k4 pj3k5 pj2k6 pj2k7 pj2k8 j3 j2 j1 S k4 k3 k2 k8 k7 k6 k1 k5 pk1j1 pk2j1 pk3j1 pk4j3 pk5j3 pk6j2 pk7j2 pk8j2 j3 j2 j1 S 

Back to our Simple Queue 1 2 1-p p p(1-q) q(1-p) (1-p)(1-q)+qp n n+1 S Applying the machinery we have just developed to the “right” set of states, we get where as before  = p(1-q) and β=q(1-p) Basically, it directly identifies the right recursive expression for us

Extending To Simple Chains Birth-death process: One-dimensional Markov chain with transitions only between neighbors Main difference is that we now allow arbitrary transition probabilities The balance equation for S gives us npn(n+1) = n+1p(n+1)n , for n  0 You could actually derive this directly by solving the balance equations progressively from the “left,” i.e., other terms would eventually cancel out, but after quite a bit of work… 1 2 p00 p01 p10 p11 n n+1 p22 pnn p(n+1)(n+1) p21 p12 p(n-1)n pn(n+1) p(n+1)n pn(n-1) p(n+2)(n+1) p(n+1)(n+2) S This generalizes our simple queue to allow transition rates between states to vary.

Solving Birth-Death Chains By induction on npn(n+1) = n+1p(n+1)n we get The unknown 0 can be determined from ii = 1, so that we finally obtain What is induction? Induction hypothesis is what you want to establish, e.g., above expression for \pi_n Step 1 verify it is true initially, i.e., for n=0 or n=1 Step 2 Assume it is true for a given value of n Step 3 prove it is then true for n+1 using known properties, e.g., relationship between \pi_n and \pi_{n+1}

Time Reversibility Another option for computing state probabilities exists (though not always) for aperiodic, irreducible Markov chains, namely, If x1, x2, x3,… exists s.t. for all i and j Σixi = 1 and xiPij = xj Pji Then i = xi (the xi’s are the limiting probabilities) and the Markov chain is called time-reversible To compute the i‘s we first assume time-reversibility, and check if we can find xi’s that work. If yes, we are done. If no, we fall back on the stationary and/or balance equations

Periodic Chains In an irreducible, positive recurrent periodic chain, the limiting distribution does not exist, but the stationary distribution does   P =  and ii =1 and represents the time-average fraction of time spent in each state Conversely, if the stationary distribution of an irreducible periodic DTMC exists, then the chain is positive recurrent