Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc 9 Markov Chains and the Theory of Games Markov Chains Regular Markov Chains Absorbing Markov Chains Game Theory and Strictly Determined Games Games with Mixed Strategies Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Any square matrix that satisfies the properties: 2. The sum of the entries in each column of T is 1. can be referred to as a stochastic matrix. Ex. Stochastic, columns add to 1 Not stochastic, column 1: sum = 1.3 Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Markov Process (Chain) Stochastic Process in which the outcomes at any stage of the experiment depend only on the outcomes of the preceding stage. The state is the outcome at any stage. The outcome at the current stage is called the current state. Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Ex. We are given a process with 2 choices: A and B. It is expected that if a person chooses A then that person has a 30% probability of choosing A the next time. If a person chooses B then that person has a 60% probability of choosing B the next time. 0.3 A 0.4 A A B Basically: 0.7 B B 0.6 This can be represented by a transition matrix Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc State 1 State 2 The probability that something in state 1 will be in state 1 in the next step is a11 = 0.3 The probability that something in state 2 will be in state 1 in the next step is a12 = 0.4 Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Transition Matrix A transition matrix associated with a Markov chain with n states is an n X n matrix T with entries aij Current State Next state Next state Current state 2. The sum of the entries in each column of T is 1. Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Ex. It has been found that of the people that eat brand X cereal, 85% will eat brand X again the next time and the rest will switch to brand Y. Also, 90% of the people that eat brand Y will eat brand Y the next time with the rest switching to brand X. At present, 70% of the people will eat brand X and the rest will eat brand Y. What percent will eat brand X after 1 cycle? X Initial state Transition: Y One cycle: So 62.5% will eat brand X Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Notice from the previous example: is called a distribution vector In general, the probability distribution of the system after n observations is given by Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Regular Markov Chain A stochastic matrix T is a regular Markov chain if the sequence T, T2, T3,… approaches a steady-state matrix in which the rows of the limiting matrix are all equal and all the entries are positive. Ex. Regular: all entries positive Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc T is Regular: all entries of T2 positive Ex. Notice Ex. Not regular: entries of T to a power will never all be positive. Notice Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Steady–State Distribution Vector Ex. Given Notice (after some work) Tends toward The steady-state distribution vector is the limiting vector from the repeated application of the transition matrix to the distribution vector. Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Finding the Steady-State Distribution Vector Let T be a regular stochastic matrix. Then the steady-state distribution vector X may be found by solving the vector equation together with the condition that the sum of the elements of the vector X be equal to 1. Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Ex. Find the steady-state vector for the transition matrix: both are Also need: Which gives: So Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Absorbing Stochastic Matrix An absorbing stochastic matrix has the properties: There is at least one absorbing state. It is possible to go from any non-absorbing state to an absorbing state in one or more stages. Ex. Absorbing Matrix State 3 is an absorbing state and an object may go from state 2 or 1 (non-absorbing states) to state 3 Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Given an absorbing stochastic matrix it is possible to rewrite it in the form: Absorbing Nonabsorbing I: identity matrix O: zero matrix Ex. 1 4 2 3 1 2 3 4 1 2 3 4 3 2 4 1 Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Finding the Steady-State Matrix for an Absorbing Stochastic Matrix Suppose an absorbing stochastic matrix A has been partitioned into submatrices Then the steady-state matrix of A is given by Where the order of the identity matrix is chosen to have the same order as R. Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Ex. Compute the steady-state matrix for the matrix from the previous example: Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc The steady-state matrix is: 1 4 2 3 Original columns 3 2 4 1 Original rows This means that an object starting in state 3 will have a probability of 0.366 of being in state 2 in the long term. Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Game Theory A combination of matrix methods with the theory of probability to determine the optimal strategies to be used when opponents are competing to maximize gains (minimize losses). Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Ex. Rafe (row player) and Carley (column player) are playing a game where each holds out a red or black chip simultaneously (neither knows the other’s choice). The betting is summarized below Carley holds black Carley holds red Rafe holds black Carley pays Rafe $5 Rafe pays Carley $10 Rafe holds red Carley pays Rafe $2 Carley pays Rafe $3 Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc We can summarize the game as a payoff matrix for Rafe: C1 C2 R1 R2 The game is a zero-sum game since one person’s payoff is the same as the other person’s loss. Rafe basically picks a row and Carley picks a column. Since the matrix is a payoff for Rafe, he wants to maximize the entry while Carley wants to minimize the entry. Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Rafe should look at the minima for each row, then pick the larger of the minima. This is called the Maximin strategy. C1 C2 minima R1 R2 –10 2 maxima 5 3 Carley should look for the maxima of the columns, then pick the smallest of the maxima. This is called the Minimax strategy. From this we see that Rafe should pick row 2 while Carley should pick column 2. Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Maxmin Strategy (R’s move) For each row (payoff matrix), find the smallest entry in that row. Choose the row for which the entry found in step 1 is as large as possible. Minmax Strategy (C’s move) For each column of the payoff matrix, find the largest entry in that column. Choose the column for which the entry found in step 1 is as small as possible. Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Ex. Determine the maximin and minimax strategies for each player in a game that has the payoff matrix: Row minima –2 –4 Column maxima 4 3 –2 The row player should pick row 1. The column player should pick column 3. *Note: the column player is favored under these strategies (win 2) Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Optimal Strategy The optimal strategy in a game is the strategy that is most profitable to a particular player. Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Strictly Determined Game A strictly determined game is characterized by the following properties: There is an entry in the payoff matrix that is simultaneously the smallest entry in its row and the largest entry in its column. This entry is called the saddle point for the game. The optimal strategy for the row (column) player is precisely the maxmin (minmax) strategy and is the row (column) containing the saddle point. Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc From a previous example: This is a strictly determined game with saddle point. The optimal strategies are for row player to pick row 1 and the column player to pick column 3. The value of the game is –2. Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Mixed Strategies Making different moves during a game. A row (column) player may choose different rows (columns) during the game. Ex. The game below has no saddle point. From a minimax/maximin strategy the row player should pick row 2 and the column player should pick column 3. Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc For a mixed strategy, let the row player pick row 2, 80% of the time and row 1, 20 % of the time. Let the column player pick column 1, 2, and 3, 10%, 20%, and 70% of the time respectively. Column player: Row player: To find the expected value of the game we compute: payoff Expected value = 1.1 Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Expected Value of a Game Let P and Q be the mixed strategies for the row player R and the column player C respectively. The expected value, E, of the game is given by: Payoff matrix Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Optimal Strategies for Nonstrictly Determined Games Payoff matrix Row player: Column player: Where Value of the game: Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc

Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc Ex. Given the payoff matrix, find the optimal strategies and then the value of the game. Row player should pick each row 50% of the time. Column player should pick column 1, 90% of the time. Value: Copyright (c) 2003 Brooks/Cole, a division of Thomson Learning, Inc