Section 10.1 Basic Properties of Markov Chains Chapter 10 Markov Chains Section 10.1 Basic Properties of Markov Chains

Some Background Information Mathematical models that evolve over time in a probabilistic manner are called stochastic processes. A special kind of stochastic process is a Markov Chain, where the outcome of an experiment depends only on the outcome of the previous experiment.

Why Study Markov Chains? Markov chains are used to analyze trends and predict the future. (Weather, stock market, genetics, product success, etc.

A Sociology Example Sociologists classify people by income as lower-class, middle-class, and upper-class. They have found that the strongest determinant of the income class of an individual is the income class of that person’s parents.

Transition Diagrams

Transition Matrices

Characteristics of a Transition Matrix It is a square matrix. All entries are between 0 and 1 (because all the entries are probabilities). The sum of the entries in any row must be 1. Denoted by P .

Key Features of Markov Chains A sequence of trials of an experiment is a Markov chain if 1.) the outcome of each experiment is one of a set of discrete states; 2.) the outcome of an experiment depends only on the present state, and not on any past states; 3.) the transition probabilities remain constant from one transition to the next.

Transition Probabilities from One State to Another The transition matrix shows the probabilities of moving from state-to-state from the current generation to the next. P gives the probabilities of a transition from one state to another in k repetitions of an experiment, provided the transition probabilities remain constant from one repetition to the next. k

Example 1 Use the transition matrix from the Sociology example to find the probabilities of change for the grandchildren of the current generation.

Example 2 Write a transition matrix for the following situation. Then find the probabilities associated with a third and fourth purchase from the bookstores. In a study of the market share of the three bookstores in a university town, it was found that 75% of those who had bought from University Bookstore would buy from it again, 15% from Campus Bookstore and 10% from Bookmart. Of those who bought from Campus Bookstore, 90% would buy from it again and 5% each would buy from University Bookstore and Bookmart. 85% of those who bought from Bookmart would buy from it again, 5% from University Bookstore and 10% from Campus Bookstore.

Probability Vector (Matrix) When a study is first begun, the probabilities of the states are called the initial distribution. This distribution is written as a matrix of only one row. Denoted by X0 .

Characteristics of a Probability Vector It is a row matrix. Each entry must be between 0 and 1 inclusive. The sum of the entries of the row must be 1.

Making Predictions about the Population Proportion with Markov Chains 1.) Create a probability vector, X0 . The entries are the initial probabilities of the states. 2.) Create the transition matrix, P . The entries are the probabilities of passing from current states (rows) to the first following states (columns). 3.) Calculate X0 P This is the probability vector after n repetitions of the experiment. In other words, it is a prediction for the proportion of the population after n repetitions. Note: The columns and rows of the probability vector and the transition matrix must be labeled the same. n

Example 3 A marketing analysis shows that KickKola currently commands 14% of the cola market. Further analysis indicates that 12% of the consumers who do not currently drink KickKola will purchase KickKola the next time they buy a cola (in response to a new advertising campaign) and that 63% of the consumers who currently drink KickKola will purchase it the next time they buy a cola. Predict KickKola’s market share at: a.) the next following purchase b.) the second following purchase

Example 4 It has been noted that George the golfer tends to repeat himself. In his first shot, he will hit his ball in the fairway (F), in the rough (R), or out of bounds (B). On his first shot he hits in the fairway 60% of the time, in the rough 30% of the time, and out of bounds 10% of the time. However, on subsequent shots, if he hit in the fairway the first time, he will hit in the fairway next time 90% of the time and in the rough 10% of the time. If he hit in the rough the first time, he will hit the rough next time 50% of the time and out of bounds 5% of the time. If he hits out of bounds the first time, he will hit out of bounds next time 70% of the time and in the fairway 15% of the time. Find the probability that his next following shot is in: a.) the fairway b.) the rough c.) out of bounds