1. . Markov Chains Tutorial #5 © Ilan Gronau. Based on original slides of Ydo Wexler & Dan Geiger

2. . Statistical Parameter Estimation Reminder The basic paradigm: MLE / bayesian approach Input data: series of observations X 1, X 2 … X t - We assumed observations were i.i.d (independent identical distributed) Data set Model Parameters: Θ Heads - P(H) Tails - 1-P(H)

3. 3 Markov Process Markov Property: The state of the system at time t+1 depends only on the state of the system at time t X1X1 X2X2 X3X3 X4X4 X5X5 Stationary Assumption: Transition probabilities are independent of time ( t ) Bounded memory transition model

4. 4 Weather: raining today40% rain tomorrow 60% no rain tomorrow not raining today20% rain tomorrow 80% no rain tomorrow Markov Process Simple Example rain no rain 0.6 0.4 0.8 0.2 Stochastic FSM:

5. 5 Weather: raining today40% rain tomorrow 60% no rain tomorrow not raining today20% rain tomorrow 80% no rain tomorrow Markov Process Simple Example Stochastic matrix: Rows sum up to 1 Double stochastic matrix: Rows and columns sum up to 1 The transition matrix:

6. 6 – Gambler starts with $10 - At each play we have one of the following: Gambler wins $1 with probability p Gambler looses $1 with probability 1-p – Game ends when gambler goes broke, or gains a fortune of $100 (Both 0 and 100 are absorbing states) 01 2 99 100 p p p p 1-p Start (10$) Markov Process Gambler’s Example

7. 7 Markov process - described by a stochastic FSM Markov chain - a random walk on this graph (distribution over paths) Edge-weights give us We can ask more complex questions, like Markov Process 01 2 99 100 p p p p 1-p Start (10$)

8. 8 Given that a person’s last cola purchase was Coke, there is a 90% chance that his next cola purchase will also be Coke. If a person’s last cola purchase was Pepsi, there is an 80% chance that his next cola purchase will also be Pepsi. coke pepsi 0.1 0.9 0.8 0.2 Markov Process Coke vs. Pepsi Example transition matrix:

9. 9 Given that a person is currently a Pepsi purchaser, what is the probability that he will purchase Coke two purchases from now? Pr [ Pepsi  ?  Coke ] = Pr [ Pepsi  Coke  Coke ] + Pr [ Pepsi  Pepsi  Coke ] = 0.2 * 0.9 + 0.8 * 0.2 = 0.34 Markov Process Coke vs. Pepsi Example (cont) Pepsi  ? ?  Coke

10. 10 Given that a person is currently a Coke purchaser, what is the probability that he will purchase Pepsi three purchases from now? Markov Process Coke vs. Pepsi Example (cont)

11. 11 Assume each person makes one cola purchase per week Suppose 60% of all people now drink Coke, and 40% drink Pepsi What fraction of people will be drinking Coke three weeks from now? Markov Process Coke vs. Pepsi Example (cont) Pr[X 3 =Coke] = 0.6 * 0.781 + 0.4 * 0.438 = 0.6438 Q i - the distribution in week i Q 0 =(0.6,0.4) - initial distribution Q 3 = Q 0 * P 3 =(0.6438,0.3562)

12. 12 Simulation: Markov Process Coke vs. Pepsi Example (cont) week - i Pr[X i = Coke] 2/3 stationary distribution coke pepsi 0.1 0.9 0.8 0.2

13. 13 Hidden Markov Models - HMM X1X1 X2X2 X L-1 XLXL XiXi Hidden states Observed data H1H1 H2H2 H L-1 HLHL HiHi

14. 14 0.9 fair loaded H H T T 0.9 0.1 1/2 1/4 3/41/2 Hidden Markov Models - HMM Coin-Tossing Example Fair/Loade d Head/Tail X1X1 X2X2 X L-1 XLXL XiXi H1H1 H2H2 H L-1 HLHL HiHi transition probabilities emission probabilities

15. 15 Hidden Markov Models - HMM C-G Islands Example Regular DNA C-G island C-G islands: Genome regions which are very rich in C and G A C G T change A C G T (1-P)/4 P/6 q/4 P P q q q qP P (1-q)/6 (1-q)/3 p/3 p/6

16. 16 Hidden Markov Models - HMM C-G Islands Example A C G T change A C G T C-G / Regular {A,C,G,T} X1X1 X2X2 X L-1 XLXL XiXi H1H1 H2H2 H L-1 HLHL HiHi To be continued…