1. MA3264 Mathematical Modelling Lecture 5 Discrete Probabilistic Modelling

2. Markov Chains vertices represent states and are labeled 1,..,K Can be illustrated using labeled directed graphs edges represent transitions between states and are labeled by numbers in the interval [0,1] Page 217, Figure 6.1 Transition Matrix T= NOTE Our TM = transpose book’s TM

3. Transition Probabilities The label on a directed edge from vertex j to vertex i is the probability that if the system is in state j at time n then it will be in state i at time n+1 FOR EVERY n (in other words these so called transition probabilities (TP) do not change over time), for convenience we place these TP into a transition matrix T Imagine a dynamic* system that can be in one of K states during each time interval [n,n+1) * marked by usually continuous and productive activity or change

4. Transition Probabilities For the situation displayed in the graph below, if a system is in state 1 at time n = 1, then the probability that the system will change to be is state 2 at time n = 2 equals 1-p and the probability that the system will stay the same to be in state 1 at time n = 2 equals p, similar considerations apply if the system is in state 2 at time n = 1 The sum of the elements in each column of T equals 1

5. State Vectors If at time n, the probability of a system being in state 1 equals a(n) and the probability of being in state 2 equals b(n) then this may be represented by a state vector Clearly the entries in this matrix are in [0,1] and their sum equals 1

6. Dynamics of State Vectors The dynamics of this vector are derived from probability

7. Law of Large Numbers If we have two (or more) large populations of individuals, each of whom can be in state 1 or state 2 at each time n, then a(n), b(n) can be interpreted as the frequency (or fractions) of individuals that are in state 1, 2 at time n Likewise, if a state vector has entries that represent frequencies, then these frequencies can be interpreted as probabilities of an individual who is chosen randomly to be in state 1, 2 at time n Likewise, the entries of a transition matrix can be either interpreted as probabilities or as frequencies This dual interpretation aspect can be initially confusing but becomes much more obvious through applications

8. Rental Car Application Example 1. Rental Car Company (pages 217-218) If we letthe fraction of cars in Orlando, Tampa at time n, then Orlando Tampa

9. Long Term Behavior Can we find the long-term behavior ? This matrix notation gives n-step transitions

10. Long Term Behavior Ifthen

11. MATLAB Experiment >> T = [.6.3;.4.7] T = 0.6000 0.3000 0.4000 0.7000 >> for k = 1:20 Tk = T^k; a1(k) = Tk(1,1); b1(k) = Tk(2,1); a2(k) = Tk(1,2); b2(k) = Tk(2,2); End >> plot([a1' a2' b1' b2'])

12. Radioactive Decay Model decay of an atom of Polonium 209 to Lead 205 after its half life (=102 years), its state vector evolves as Transition Matrix T= 1. Po209, 2. Pb205

13. Radioactive Decay State vector dynamics of # Po209 atoms remaining (after 102 years) 1. two Po209 2. one Po209 3. zero Po209

14. Traffic Light With fixed transition times of one minute 1. red 2. green 3. yellow

15. Convergence Criteria Theorem (Perron-Frobenius) If T is an n x n transition matrix then the following three statements are true (ii) (i) 1 is an eigenvalue of T since vT = v where exists (converges) if and only if all other (iii) If there exists an integer p > 0 such that every entry of the matrix is positive then the if condition in (ii) is satisfied and every entry of w is positive. eigenvalues have modulus < 1,then the right eigenvector w (Tw = w) for eigenvalue 1 (normalized so the sum of the entries of w equals 1), is the steady state of T and each column of S equals w

16. Numerical Examples MatrixEigenvaluesSteady State Vector

17. Suggested Reading&Problems in Textbook 6.1 Probabilistic Modelling with Discrete Systems, pages 217-222 http://aix1.uottawa.ca/~jkhoury/markov.htm Recommended Websites http://en.wikipedia.org/wiki/Markov_chain http://www.eng.buffalo.edu/~kofke/applets/MarkovApplet1.html

18. Tutorial 5 Due Week 29 Sept – 3 Oct Page 222. Problem 1. Page 222. Problems 2 Page 222-223. Project Problem 1