Courtesy of J. Akinpelu, Anis Koubâa, Y. Wexler, & D. Geiger 11. Markov Chains Courtesy of J. Akinpelu, Anis Koubâa, Y. Wexler, & D. Geiger

Random Processes A stochastic process is a collection of random variables The index t is often interpreted as time. is called the state of the process at time t. Discrete-valued or continuous-valued The set I is called the index set of the process. If I is countable, the stochastic process is said to be a discrete-time process. If I is an interval of the real line, the stochastic process is said to be a continuous-time process. The state space E is the set of all possible values that the random variables can assume.

Discrete Time Random Process If I is countable, is often denoted by n = 0,1,2,3,… time 1 2 3 4 Events occur at specific points in time

Discrete time Random Process State Space = {SUNNY, RAINY} Day Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 THU FRI SAT SUN MON TUE WED X(dayi): Status of the weather observed each DAY

Markov processes A stochastic process is called a Markov process if for all states and all If Xn’s are integer-valued, Xn is called a Markov Chain

What is “Markov Property”? PAST EVENTS NOW FUTURE EVENTS ? Probability of “R” in DAY6 given all previous states Probability of “S” in DAY6 given all previous states Day Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 THU FRI SAT SUN MON TUE WED Markov Property: The probability that it will be (FUTURE) SUNNY in DAY 6 given that it is RAINY in DAY 5 (NOW) is independent from PAST EVENTS

Markov Chains We restrict ourselves to Markov chains such that the conditional probabilities are independent of n, and for which (which is equivalent to saying that state space E is finite or countable). Such a Markov chain is called homogeneous.

Markov Chains Since probabilities are non-negative, and the process must make a transition into some state at each time in I, then We can arrange the probabilities into a square matrix called the transition matrix.

Markov Chain: A Simple Example Weather: raining today 40% rain tomorrow 60% no rain tomorrow not raining today 20% rain tomorrow 80% no rain tomorrow State transition diagram: rain no rain 0.6 0.4 0.8 0.2

Rain (state 0), No rain (state 1) Weather: raining today 40% rain tomorrow 60% no rain tomorrow not raining today 20% rain tomorrow 80% no rain tomorrow The transition (prob.) matrix P: for a given current state: - Transitions to any states - Each row sums up to 1

Examples in textbook Example 11.6 Example 11.7 Figures 11.2 and 11.3

Transition probability Note that each entry in P is a one-step transition probability, say, from the current state to the right next state Then, how about multiple steps? Let’s start with 2 steps first

2-Step Transition Prob. of 2 state system: states 0 and 1 Let pij(2) be probability of going from i to j in 2 steps Suppose i = 0, j = 0, then P(X2 = 0|X0 = 0) = P(X1 = 1|X0 = 0)  P(X2 = 0| X1 = 1) + P(X1 = 0|X0 = 0)  P(X2 = 0| X1 = 0) p00(2) = p01p10 + p00p00 Similarly p01(2) = p01p11 + p00p01 p10(2) = p10p00 + p11p10 p11(2) = p10p01 + p11p11 In matrix form, P(2) = P(1)P(1) = P2 13

In general, 2 step transition is expressed as Now note that

Two-step transition prob. State space E={0, 1, 2} Hence, 

Chapman-Kolmogorov Equations In general, for all This leads to the Chapman-Kolmogorov equations: