1. Markov Chains Brian Carrico

2. The Mathematical Markovs  Vladimir Andreyevich Markov (1871-1897)  Andrey Markov’s younger brother  With Andrey, developed the Markov brothers’ inequality  Andrey Andreyevich Markov Jr (1903-1979)  Andrey Markov’s son  One of the key founders of the Russian school of constructive mathematics and logic  Also made contributions to differential equations,topology, mathematical logic and the foundations of mathematics  Which brings us to:

3. Andrey Andreyevich Markov Андрей Андреевич Марков June 14, 1856 – July 20, 1922  Born in Ryazan  (roughly 170 miles Southeast of Moscow)  Began Grammar School in 1866  Started at St Petersburg University in 1874  Defended his Masters Thesis in 1880  Doctoral Thesis in 1885  Excommunicated from the Russian Orthodox Church

4. Precursors to Markov Chains  Bernoulli Series  Brownian Motion  Random Walks

5. Bernoulli Series  Jakob Bernoulli (1654-1705)  Sequence independent random variables X 1, X 2,X 3,... such that  For every i, X i is either 0 or 1  For every i, P(X i )=1 is the same  Markov’s first discussions of chains, a 1906 paper, considers only chains with two states  Closely related to Random Walks

6. Brownian Motion  Described as early as 60 BC by Roman poet Lucretius  Formalized and officially discovered by botanist Robert Brown in 1827  The seemingly random movement of particles suspended in a fluid

7. Random Walks  Formalized in 1905 by Karl Pearson  The formalization of a trajectory that consists of taking successive random steps  The results of random walk analysis have been applied to computer science, physics, ecology, economics, and a number of other fields as a fundamental model for random processes in time  Turns out to be a specific Markov chain

8. So what is a Markov Chain?  A random process where all information about the future is contained in the present state  Or less formally: a process where future states depend only on the present state, and are independent of past states  Mathematically:

9. Applications of Markov Chains  Science  Statistics  Economics and Finance  Gambling and games of chance  Baseball  Monte Carlo

10. Science  Physics  Thermodynamic systems generally have time- invariant dynamics  All relevant information is in the state description  Chemistry  An algorithm based on a Markov chain was used to focus the fragment-based growth of chemicals in silico towards a desired class of compounds such as drugs or natural products

11. Economics and Finance  Markov Chains are used model a variety of different phenomena, including asset prices and market crashes.  Regime-switching model of James D. Hamilton  Markov Switching Multifractal asset pricing model  Dynamic macroeconomics

12. Gambling and Games of Chance  In most card games each hand is independent  Board games like Snakes and Ladders

13. Baseball  Use of Markov chain models in baseball analysis began in 1960  Each at bat can be taken as a Markov chain

14. Monte Carlo  A Markov chain with a large number of steps is used to create the algorithm for the basis of the Monte Carlo simulation

15. Statistics  Many important statistics measure independent trials, which can be represented by Markov chains

16. An Example from Statistics  A thief is in a dungeon with three identical doors. Once the thief chooses a door and passes through it, the door locks behind him. The three doors lead to:  A 6 hour tunnel leading to freedom  A 3 hour tunnel that returns to the dungeon  A 9 hour tunnel that returns to the dungeon  Each door is chosen with equal probability. When he is dropped back into the dungeon by the second and third doors there is a memoryless choice of doors. He isn’t able to mark the doors in any way. What is his expected time of escape?  Note:

17. Example (cont)  We plug the values in for x i and p(x i ) to get:  E(X)=6*(1/3)+x 2 *(1/3)+x 3 *(1/3)  But what are x 2 and x 3 ?  Because the decision is memoryless, the expected time after returning from tunnels 2 or 3 doesn’t change from the initial expected time. So, x2=x3=E(X).  So,  E(X)=6*(1/3)+E(X)*(1/3)+E(X)*(1/3)  Now we’re back in Algebra 1

18. Sources  Wikipedia  The Life and Work of A.A. Markov. Basharin, Gely P. et al. http://decision.csl.illinois.edu/~meyn/pages /Markov-Work-and-life.pdf http://decision.csl.illinois.edu/~meyn/pages /Markov-Work-and-life.pdf http://decision.csl.illinois.edu/~meyn/pages /Markov-Work-and-life.pdf  Leemis (2009), Probability