Courtesy of J. Akinpelu, Anis Koubâa, Y. Wexler, & D. Geiger 11. Markov Chains Courtesy of J. Akinpelu, Anis Koubâa, Y. Wexler, & D. Geiger

Random Processes A stochastic process is a collection of random variables The index t is often interpreted as time. is called the state of the process at time t. Discrete-valued or continuous-valued The set I is called the index set of the process. If I is countable, the stochastic process is said to be a discrete-time process. If I is an interval of the real line, the stochastic process is said to be a continuous-time process. The state space E is the set of all possible values that the random variables can assume.

Discrete Time Random Process If I is countable, is often denoted by n = 0,1,2,3,… time 1 2 3 4 Events occur at specific points in time

Discrete time Random Process State Space = {SUNNY, RAINNY} Day Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 THU FRI SAT SUN MON TUE WED X(dayi): Status of the weather observed each DAY

Markov processes A stochastic process is called a Markov process if for all states and all If Xn’s are integer-valued, Xn is called a Markov Chain .

What is “Markov Property”? PAST EVENTS NOW FUTURE EVENTS ? Probability of “R” in DAY6 given all previous states Probability of “S” in DAY6 given all previous states Day Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 THU FRI SAT SUN MON TUE WED Markov Property: The probability that it will be (FUTURE) SUNNY in DAY 6 given that it is RAINNY in DAY 5 (NOW) is independent from PAST EVENTS

Markov Chains We restrict ourselves to Markov chains such that the conditional probabilities are independent of n, and for which (which is equivalent to saying that E is finite or countable). Such a Markov chain is called homogeneous.

Markov Chains Since probabilities are non-negative, and the process must make a transition into some state at each time in I, then We can arrange the probabilities into a square matrix called the transition matrix.

Markov Chain: A Simple Example Weather: raining today 40% rain tomorrow 60% no rain tomorrow not raining today 20% rain tomorrow 80% no rain tomorrow State transition diagram: rain no rain 0.6 0.4 0.8 0.2

Rain (state 0), No rain (state 1) Weather: raining today 40% rain tomorrow 60% no rain tomorrow not raining today 20% rain tomorrow 80% no rain tomorrow The transition (prob.) matrix P: for a given current state: - Transitions to any states - Each row sums up to 1

Examples in textbook Example 11.6 Example 11.7 Figures 11.2 and 11.3

Transition probability Note that each entry in P is a one-step transition probability, say, from the current state to the right next state Then, how about multiple steps? Let’s start with 2 steps first

2-Step Transition Prob. of 2 state system : state 0 and state 1 Let pij(2) be probability of going from i to j in 2 steps Suppose i = 0, j = 0, then P(X2 = 0|X0 = 0) = P(X1 = 1|X0 = 0)  P(X2 = 0| X1 = 1) + P(X1 = 0|X0 = 0)  P(X2 = 0| X1 = 0) p00(2) = p01p10 + p00p00 Similarly p01(2) = p01p11 + p00p01 p10(2) = p10p00 + p11p10 p11(2) = p10p01 + p11p11 In matrix form, P(2) = P  P, 13

In general, 2 step transition is expressed as Now note that

Two-step transition prob. States 0, 1, 2 (3 state system) Hence, 

Chapman-Kolmogorov Equations In general, for all This leads to the Chapman-Kolmogorov equations: