1. 9  Markov Chains  Regular Markov Chains  Absorbing Markov Chains  Game Theory and Strictly Determined Games  Games with Mixed Strategies Markov Chains and the Theory of Games

2. 9.1 Markov Chains

3. Transitional Probabilities  In this chapter we will be concerned with a special class of stochastic processes in which the probabilities associated with the outcomes at any stage of the experiment depend only on the outcomes of the preceding stage.  Such a process is called a Markov process, or Markov chain.  The outcome at any stage of the experiment in a Markov process is called the state of the experiment.  In particular, the outcome at the current stage of the experiment is called the current state of the process.

4. Applied Example: Common Stocks  An analyst at Weaver and Kline, a stock brokerage firm, observes that the closing price of the preferred stock of an airline company over a short span of time depends only on its previous closing price.  At the end of each trading day, he makes a note of the stock’s performance for that day, recording the closing price as “higher,” “unchanged,” or “lower” according to whether the stock closes higher, unchanged, or lower than the previous day’s closing price.  This sequence of observations may be viewed as a Markov chain. Applied Example 1, page 484

5. Applied Example: Common Stocks  If on a certain day the stock’s closing price is higher than that of the previous day, then the probability that it closes higher, unchanged, or lower on the next trading day is.2,.3, and.5, respectively.  Next, if the stock’s closing price is unchanged from the previous day, then the probability that it closes higher, unchanged, or lower on the next trading day is.5,.2, and.3, respectively.  Finally, if the stock’s closing price is lower than that of the previous day, then the probability that it closes higher, unchanged, or lower on the next trading day is.4,.4, and.2, respectively.  With the aid of tree diagrams, describe the transition between states and the probabilities associated with these transitions. Applied Example 1, page 484

6. Applied Example: Common Stocks Solution  The Markov chain being described has three states, each of which may be displayed by constructing a tree diagram in which the associated probabilities are shown on the appropriate limbs: ✦ If the current state is higher, the tree diagram is: Higher HigherUnchangedLower.2.3.5 Applied Example 1, page 484

7. Applied Example: Common Stocks Solution  The Markov chain being described has three states, each of which may be displayed by constructing a tree diagram in which the associated probabilities are shown on the appropriate limbs: ✦ If the current state is unchanged, the tree diagram is: Unchanged HigherUnchangedLower.5.2.3 Applied Example 1, page 484

8. Applied Example: Common Stocks Solution  The Markov chain being described has three states, each of which may be displayed by constructing a tree diagram in which the associated probabilities are shown on the appropriate limbs: ✦ If the current state is lower, the tree diagram is: Lower HigherUnchangedLower.4.4.2 Applied Example 1, page 484

9. Transition Probabilities  The probabilities encountered in the last example are called transition probabilities because they are associated with the transition from one state to the next in the Markov process.  These transition probabilities may be conveniently represented in the form of a matrix.  Suppose for simplicity that we have a Markov chain with three possible outcomes at each stage of the experiment.  Let’s refer to these outcomes as state 1, state 2, and state 3.  Then the transition probabilities associated with the transition from state 1 to each of the states 1, 2, and 3 in the next phase of the experiment are precisely the respective conditional probabilities that the outcome is state 1, state 2, and state 3 given that the outcome state 1 has occurred.

10. Transition Probabilities  In short, the desired transition probabilities are respectively P(state 1 | state 1), P(state 2 | state 1), and P(state 3 | state 1).  Thus, we can write: a 11 = P(state 1 | state 1) a 21 = P(state 2 | state 1) a 31 = P(state 3 | state 1) Next state Current state State 1 State 2 State 3 a 11 a 21 a 31  These can be represented with a tree diagram as well:

11. Transition Probabilities  Similarly, the transition probabilities associated with the transition from state 2 can be presented as conditional probabilities, as well as in a tree diagram: a 12 = P(state 1 | state 2) a 22 = P(state 2 | state 2) a 32 = P(state 3 | state 2) Next state Current state State 2 State 1 State 2 State 3 a 12 a 22 a 32

12. Transition Probabilities  Finally, the transition probabilities associated with the transition from state 3 can be presented as conditional probabilities, as well as in a tree diagram: a 13 = P(state 1 | state 3) a 23 = P(state 2 | state 3) a 33 = P(state 3 | state 3) Next state Current state State 3 State 1 State 2 State 3 a 13 a 23 a 33

13. Transition Probabilities  These observations lead to the following matrix representation of the transition probabilities: State 1 State 2 State 3 State 1 State 2 State 3 Current state Nextstate

14. Applied Example: Common Stocks  Use a matrix to represent the transition probabilities obtained earlier. Solution  There are three states at each stage of the Markov chain under consideration.  Letting state 1, state 2, and state 3 denote the states “higher,” “unchanged,” and “lower,” respectively, we find that Applied Example 2, page 484

15. Applied Example: Common Stocks  Use a matrix to represent the transition probabilities obtained earlier. Solution  Thus, the required matrix representation is given by Applied Example 2, page 484

16. Transition Matrix  A transition matrix associated with a Markov Chain with n states is an n x n matrix T with entries a ij ( 1  i  n; 1  j  n)  The transition matrix has the following properties: 1. a ij  0 for all i and j. 2. The sum of the entries in each column of T is 1.

17. Applied Example: Urban-Suburban Population Flow  Because of the continued successful implementation of an urban renewal program, it is expected that each year 3% of the population currently residing in the city will move to the suburbs and 6% of the population currently residing in the suburbs will move into the city.  At present, 65% of the total population of the metropolitan area lives in the city itself, while the remaining 35% lives in the suburbs.  Assuming that the total population of the metropolitan area remains constant, what will be the distribution of the population one year from now? Applied Example 4, page 487

18. Solution  We can use a tree diagram to see the Markov process under consideration:  Thus, the probability that a person selected at random will be a city dweller one year from now is given by (.65)(.97) + (.35)(.06) =.6515 Applied Example: Urban-Suburban Population Flow Current population City Suburb.65.35 City Suburb.06.94 City Suburb.97.03 Population one year later Applied Example 4, page 487

19. Solution  We can use a tree diagram to see the Markov process under consideration:  The probability that a person selected at random will be a suburb dweller one year from now is given by (.65)(.03) + (.35)(.94) =.3485 Applied Example: Urban-Suburban Population Flow Current population City Suburb.65.35 City Suburb.06.94 City Suburb.97.03 Population one year later Applied Example 4, page 487

20. Solution  The process under consideration may be viewed as a Markov chain with two possible states at each stage of the experiment: State 1:“living in the city” State 2:“living in the suburbs”  The transition matrix associated with this Markov chain is Applied Example: Urban-Suburban Population Flow State 1 State 2 State 1 State 2 T = Transition matrix Applied Example 4, page 487

21. Solution  Next, observe that the initial (current) probability distribution of the population may be summarized in the form of the column vector  Using the probabilities obtained with the tree diagram, we may write the population distribution one year later as Applied Example: Urban-Suburban Population Flow State 1 State 2 X 0 = Initial-state matrix State 1 State 2 X 1 = Distribution after one year Applied Example 4, page 487

22. Solution  We can now verify that so this problem may be solved using matrix multiplication. Applied Example: Urban-Suburban Population Flow Applied Example 4, page 487

23.  Now, find the population distribution of the city after two years and three years. Solution  Let X 1, X 2, and X 3 be the column vectors representing the population distribution of the metropolitan area after one year, two years, and three years, respectively.  To find X 2, we take X 1 to represent the “initial” probability distribution in this part of the calculation; thus  Similarly, for X 3 we have Applied Example: Urban-Suburban Population Flow Applied Example 4, page 487

24.  Now, find the population distribution of the city after two years and three years. Solution  Observe that we have  These results are easily generalized. Applied Example: Urban-Suburban Population Flow Applied Example 4, page 487

25. Distribution Vectors  Let there be a Markov process in which there are n possible states at each stage of the experiment.  Let the probability of the system being in state 1, state 2, …, state n, initially, be given by p 1, p 2, …, p n, respectively.  Such a distribution may be represented as an n-dimensional distribution vector and the probability distribution of the system after m observations is given by

26. 9.2 Regular Markov Chains

27. Steady-State Distribution Vectors  In the last section, we derived a formula for computing the likelihood that a physical system will be in any one of the possible states associated with each stage of a Markov process describing the system.  In this section we use this formula to help us investigate the long-term trends of certain Markov processes.

28. Applied Example: Educational Status of Women  A survey conducted by the National Commission on the Educational Status of Women reveals that 70% of the daughters of women who have completed 2 or more years of college have also completed 2 or more years of college, whereas 20% of the daughters of women who have had less than 2 years of college have completed 2 or more years of college.  If this trend continues, determine, in the long run, the percentage of women in the population who will have competed at least 2 years of college given that currently only 20% of the women have completed at least 2 years of college. Applied Example 1, page 494

29. Applied Example: Educational Status of Women Solution  This problem may be viewed as a Markov process with two possible states: State 1:“completed 2 or more years of college” State 2:“completed less than 2 years of college”  The transition matrix associated with this Markov chain is given by and the initial distribution vector is given by Applied Example 1, page 494

30. Applied Example: Educational Status of Women Solution  To study the long-term trend, let’s compute X 1, X 2, …  These vectors give the proportion of women with 2 or more years of college and that of women with less than 2 years of college after each generation. After one generation After two generations After three generations Applied Example 1, page 494

31. Applied Example: Education Status of Women Solution  Proceeding further, we obtain the following sequence of vectors:  From the result of these computations, we see that as m increases, the probability distribution vector X m approaches the probability distribution vector  Such a vector is called the limiting, or steady-state, distribution vector for the system. After nine generations Click forward to see more generations Applied Example 1, page 494

32. Applied Example: Educational Status of Women Solution  We interpret these results as follows: ✦ Initially, 20% of all women have completed 2 or more years of college, whereas 80% have completed less than 2 years of college. ✦ We see that generation after generation, the proportion of the first group increases while the proportion of the second group decreases. ✦ In the long run, the proportions stabilize, so that 40% of all women will have completed 2 or more years of college, whereas 60% will have completed less than 2 years of college. Applied Example 1, page 494

33. Regular Markov Chain  Continuing with last example, if we calculate T, T 2, T 3, … we can see that the powers T m of the transition matrix T tend toward a fixed matrix as m gets larger and larger:  We can see that the larger the value of m the closer resulting matrix approaches the matrix  Such a matrix is called the steady-state matrix for the system. Click forward to see different powers of T

34.  Note that the steady-state matrix from our last example has columns that are all equal and all the entries are positive:  A matrix T having this property is called a regular Markov chain. Regular Markov Chain

35.  A stochastic matrix T is a regular Markov chain if the sequence T, T 2, T 3, … approaches a steady-state matrix in which the columns of the limiting matrix are all equal and all the entries are positive.  A stochastic matrix T is regular if and only if some power of T has entries that are all positive. Regular Markov Chain

36. Example  Determine whether the matrix is regular: Solution  Since all the entries of the matrix are positive, the given matrix is regular. Example 2, page 497

37. Example  Determine whether the matrix is regular: Solution  One of the entries is equal to zero, so let’s compute the second power of the matrix:  Since the second power of the matrix has entries that are all positive, we conclude that the given matrix is in fact regular. Example 2, page 497

38. Example  Determine whether the matrix is regular: Solution  Denote the given matrix by A. Then  Since A 3 = A, it follows that A 4 = A 2 and so on.  Therefore, these are the only two matrices that arise for any power of A. Example 2, page 497

39. Example  Determine whether the matrix is regular: Solution  Denote the given matrix by A. Then  Some of the entries of A and A 2 are not positive, so any power of A will have entries that are not positive.  Thus, we conclude the matrix is not regular. Some entries are not positive Example 2, page 497

40. Finding the Steady-State Distribution Vector  Let T be a regular stochastic matrix.  Then the steady-state distribution vector X may be found by solving the vector equation TX = X together with the condition that the sum of the elements of the vector X be equal to 1.

41. Example  Find the steady-state distribution vector for the regular Markov chain whose transition matrix is Example 3, page 498

42. ExampleSolution  Let be the steady-state distribution vector, where the numbers x and y are to be determined.  The condition TX = X translates into the matrix equation or, equivalently, the system of linear equations Example 3, page 498

43. ExampleSolution  But each equation that makes up the system is equivalent to the single equation  Next, the condition that the sum of the elements of X add up to 1 gives  To find the values of x and y that meet both conditions we solve the system Example 3, page 498

44. ExampleSolution  The solution to the system is  So, the required steady-state distribution vector is given by which agrees with the result obtained earlier. Example 3, page 498

45. 9.3 Absorbing Markov Chains

46.  In this section we investigate the long-term trends of a certain class of Markov chains that involve transition matrices that are not regular.  In particular, we study Markov chains in which the transition matrices, know as absorbing matrices, have special properties we will describe.

47.  Consider the stochastic matrix associated with a Markov process:  We see that after one observation, the probability is 1 that an object previously in state 1 will remain in state 1. Absorbing Markov Chains

48.  Consider the stochastic matrix associated with a Markov process:  Similarly, we see that an object previously in state 2 must remain in state 2. Absorbing Markov Chains

49.  Consider the stochastic matrix associated with a Markov process:  Next, we find that an object previously in state 3 has a probability of ✦.2 of going to state 1. ✦.3 of going to state 2. ✦.5 of remaining in state 3. ✦ 0 (no chance) of going to state 4 Absorbing Markov Chains

50.  Consider the stochastic matrix associated with a Markov process:  Finally, we see that an object previously in state 4 must go to state 2.

51. Absorbing Markov Chains  This stochastic matrix exhibits certain special characteristics:  As we saw, an object in state 1 or state 2 must remain in state 1 or state 2, respectively.  Such states are called absorbing states.  In general, an absorbing state is one from which it is impossible for an object to leave.

52. Absorbing Markov Chains  This stochastic matrix exhibits certain special characteristics:  To identify the absorbing states of a stochastic matrix, we examine each column of the matrix.  If column i has a 1 in the a ii position (on the main diagonal of the matrix) and zeros elsewhere in that column, then and only then is state i an absorbing state.

53. Absorbing Markov Chains  This stochastic matrix exhibits certain special characteristics:  Also note that states 3 and 4, although not absorbing states, have the property that an object in each of these states has a possibility of going to an absorbing state.

54. Absorbing Markov Chains  This stochastic matrix exhibits certain special characteristics: For example  An object in state 3 has probabilities of.2 and.3 of ending up in states 1 and 2, respectively, which are absorbing states.

55. Absorbing Markov Chains  This stochastic matrix exhibits certain special characteristics: For example  An object in state 4 must end up in state 2, an absorbing state.

56. Absorbing Stochastic Matrix  An absorbing stochastic matrix has the following properties: 1.There is at least one absorbing state. 2.It is possible to go from each nonabsorbing state to an absorbing state in one or more steps.

57. Applied Example: Gambler’s Ruin  John has decided to risk $2 in the following game of chance.  He places a $1 bet on each repeated play of the game in which the probability of his winning $1 is.4, and he continues to play until he has accumulated a total of $3 or he has lost all of his money.  Write the transition matrix for the related absorbing Markov chain. Applied Example 2, page 506

58. Applied Example: Gambler’s Ruin Solution  There are four states in this Markov chain, which correspond to John accumulating a total of $0, $1, $2, and $3.  The state of accumulating $0 (losing everything) and the state of accumulating $3 are both absorbing states.  We will list the absorbing states first (it will later become evident why this is convenient). $0 $3$1$2 $0$3$1$2 AbsorbingNonabsorbing Applied Example 2, page 506

59. Applied Example: Gambler’s Ruin Solution  For the case of the nonabsorbing state “$1”: ✦ The probability of going from an accumulated amount of $1 to $0, is a 31 =.6. $0 $3$1$2 $0$3$1$2 AbsorbingNonabsorbing Applied Example 2, page 506

60. Applied Example: Gambler’s Ruin Solution  For the case of the nonabsorbing state “$1”: ✦ It is not possible to go from an accumulated amount of $1 to either $3 or $1, so a 23 = a 33 = 0. $0 $3$1$2 $0$3$1$2 AbsorbingNonabsorbing Applied Example 2, page 506

61. Applied Example: Gambler’s Ruin Solution  For the case of the nonabsorbing state “$1”: ✦ The probability of going from an accumulated amount of $1 to $2, is a 43 =.4. $0 $3$1$2 $0$3$1$2 AbsorbingNonabsorbing Applied Example 2, page 506

62. Applied Example: Gambler’s Ruin Solution  For the case of the nonabsorbing state “$2”: ✦ It is not possible to go from an accumulated amount of $2 to either $0 or $2, so a 41 = a 44 = 0. $0 $3$1$2 $0$3$1$2 AbsorbingNonabsorbing Applied Example 2, page 506

63. Applied Example: Gambler’s Ruin Solution  For the case of the nonabsorbing state “$2”: ✦ The probability of going from an accumulated amount of $2 to $3, is a 42 =.4. $0 $3$1$2 $0$3$1$2 AbsorbingNonabsorbing Applied Example 2, page 506

64. Applied Example: Gambler’s Ruin Solution  For the case of the nonabsorbing state “$2”: ✦ The probability of going from an accumulated amount of $2 to $1, is a 43 =.6. $0 $3$1$2 $0$3$1$2 AbsorbingNonabsorbing Applied Example 2, page 506

65. Finding the Steady-State Matrix for an Absorbing Stochastic Matrix  Suppose an absorbing stochastic matrix A has been partitioned into submatrices  Then the steady-state matrix of A is given by where the order of the identity matrix appearing in the expression (I – R) –1 is chosen to be the same order as R.

66. Applied Example: Gambler’s Ruin (Continued)  If John continues to play the game until he has accumulated a sum of $3 or he has lost all of his money, what is the probability that he will accumulate $3? Applied Example 3, page 507

67. Applied Example: Gambler’s Ruin (Continued) Solution  The transition matrix associated with the Markov process is  We need to find the steady-state matrix of A. In this case,  Thus, Applied Example 3, page 507

68. Applied Example: Gambler’s Ruin (Continued) Solution  Next, we find the inverse of matrix I – R: and so Applied Example 3, page 507

69. Applied Example: Gambler’s Ruin (Continued) Solution  Therefore, the required steady-state matrix of A is given by  Thus, we see that starting with $2, the probability is.53 that John will leave the game with an accumulated amount of $3 (that is, he wins $1). $0 $3 $1$2 $0$3$1$2 Applied Example 3, page 507

70. 9.4 Game Theory and Strictly Determined Games

71. Game Theory  The theory of games combines matrix methods with the theory of probability to determine the optimal strategies to be employed by two or more opponents involved in a competitive situation, with each opponent seeking to maximize his or her “gains,” or, equivalently, to minimize his or her “losses.”  For simplicity, we limit our discussion to games with two players.

72. Applied Example: Coin-Matching Game  Richie and Chuck play a coin-matching game in which each player selects a side of a penny without prior knowledge of the other’s choice.  Then upon a predetermined signal, both players disclose their choices simultaneously: ✦ Chuck agrees to pay Richie $3 if both choose heads. ✦ If Richie chooses heads and Chuck chooses tails, then Richie pays Chuck $6. ✦ If Richie chooses tails and Chuck chooses heads, then Chuck pays Richie $2. ✦ Finally, if both choose tails, then Chuck pays Richie $1.  In this game, the objective of each player is to discover a strategy that will ensure that his winnings are maximized (or losses minimized). Applied Example 1, page 512

73. Applied Example: Coin-Matching Game  This coin-matching game is an example of a zero-sum game: ✦ a game in which the payoff to one party results in an equal loss to the other  For such games, the sum of the payment made by both players at the end of each play adds up to zero. Applied Example 1, page 512

74. Applied Example: Coin-Matching Game  We can represent the game’s data in the form of a matrix:  Each row corresponds to one of the two possible moves by Richie (referred to as the row player, R).  Each column corresponds to one of the two possible moves by Chuck (the column player, C). Heads Tails HeadsTails C’s moves R’s moves Chuck Richie Applied Example 1, page 512

75. Applied Example: Coin-Matching Game  We can represent the game’s data in the form of a matrix:  Each entry in the matrix represents the payoff from C to R: ✦ a 11 = 3 represents a $3 payoff from Chuck to Richie (C to R) when both choose to play heads. Heads Tails HeadsTails C’s moves R’s moves Chuck Richie Applied Example 1, page 512

76. Applied Example: Coin-Matching Game  We can represent the game’s data in the form of a matrix:  Each entry in the matrix represents the payoff from C to R: ✦ a 12 = – 6 represents a $6 payoff from Richie to Chuck (the payoff from C to R is negative) when Richie chooses heads and Chuck chooses tails. Heads Tails HeadsTails C’s moves R’s moves Chuck Richie Applied Example 1, page 512

77. Applied Example: Coin-Matching Game  We can represent the game’s data in the form of a matrix:  Each entry in the matrix represents the payoff from C to R: ✦ a 21 = 2 represents a $2 payoff from Chuck to Richie when Richie chooses tails and Chuck chooses heads. Heads Tails HeadsTails C’s moves R’s moves Chuck Richie Applied Example 1, page 512

78. Applied Example: Coin-Matching Game  We can represent the game’s data in the form of a matrix:  Each entry in the matrix represents the payoff from C to R: ✦ a 22 = 1 represents a $1 payoff from Chuck to Richie when both choose to play tails. Heads Tails HeadsTails C’s moves R’s moves Chuck Richie Applied Example 1, page 512

79. More Generally: The Payoff Matrix  Consider a two-person game with players R and C.  Suppose R has m possible moves, R 1, R 2, …, R m and C has n possible moves C 1, C 2, …, C n.

80. More Generally: The Payoff Matrix  We can represent the game in terms of an m × n matrix in which each row represents one of the m possible moves of R and each column represents one of the n possible moves of C:

81. More Generally: The Payoff Matrix  The entry a ij in the ith row and jth column of the payoff matrix represents the payoff from C to R when R chooses move R i and C chooses move C j.  Note that a negative value for entry a ij means that the payoff will be from R to C.

82. Optimal Strategies  Let’s return to the payoff matrix of the coin-matching game:  Let’s consider first R’s point of view: ✦ The entries in the matrix represent payoffs to him, so he might at first consider choosing the row containing the largest entry (R 1 ) as a possible move.  By choosing R 1, R would certainly gain the largest possible payoff of $3 if C chose C 1.  However, if C chose C 2 instead, then R would lose $6! Heads Tails HeadsTails C’s moves R’s moves Chuck Richie

83. Optimal Strategies  Let’s return to the payoff matrix of the coin-matching game:  Let’s consider first R’s point of view: ✦ A more prudent approach would be to assume that no matter what row he chose, C will counter with a move (column) that will result in the smallest payoff to him. ✦ To maximize his payoff under these circumstances, R would then select from among the moves (rows) the one in which the smallest payoff is as large as possible. ✦ This strategy is called the maximin strategy. Heads Tails HeadsTails C’s moves R’s moves Chuck Richie

84. Maximin Strategy 1.For each row of the payoff matrix, find the smallest entry in that row. 2.Choose the row for which the entry found in step 1 is as large as possible. This row constitutes R’s “best” move.

85. Optimal Strategies  Let’s return to the payoff matrix of the coin-matching game:  Let’s apply the maximin strategy from R’s point of view: ✦ We can write to the right of the matrix the smallest payoff R can get in each move. ✦ Of these, we can now see that the largest possible guaranteed payoff for R is with move R 2. ✦ Thus, according to the maximin strategy, R’s “best” move is R 2, which means he should choose tails. Heads Tails HeadsTails C’s moves R’s moves Row Minima

86. Optimal Strategies  Let’s return to the payoff matrix of the coin-matching game:  Next, let’s consider the game from C’s point of view: ✦ His objective is to minimize the payoff to R. ✦ This is accomplished by choosing the column whose largest payoff is as small as possible. ✦ This strategy for C is called the minimax strategy. Heads Tails HeadsTails C’s moves R’s moves

87. Minimax Strategy 1.For each column of the payoff matrix, find the largest entry in that column. 2.Choose the column for which the entry found in step 1 is as small as possible. This column constitutes C’s “best” move.

88. Optimal Strategies  Let’s return to the payoff matrix of the coin-matching game:  Let’s apply the minimax strategy from C’s point of view: ✦ We can write at the bottom of the matrix the largest payoff C can pay in each move. ✦ We can now see that the smallest possible guaranteed payoff from C is with move C 2. ✦ Thus, according to the minimax strategy, C’s “best” move is C 2, which means he should choose tails. Heads Tails HeadsTails C’s moves R’s moves Smaller of the Column Maxima 3 1 3 1

89. Example  For the game with the following payoff matrix, determine the maximin and minimax strategies for each player. Solution  The maximin strategy for the row player is to play row 3.  The minimax strategy for the column player is to play column 2. Column Maxima 6 04 6 04 Row Minima Largest of row minima Smallest of column maxima

90. Playing the Game Repeatedly  If both players are always rational and assume the other is also always rational, it would seem that they both end up following the same strategy again and again, yielding the same exact outcome every time.  But what if one player realizes that his opponent is employing the maximin (or minimax) strategy?  Perhaps this player can use this knowledge to his advantage and choose a different strategy.

91. Example  Let's consider our last example to see how this may work.  Let's consider first the row player.  Let's suppose the row player realized that the column player consistently follows the minimax strategy, always choosing column 2. Column Maxima Row Minima Largest of row minima 6 04 6 04 Smallest of column maxima Example 3, page 515

92. Example  Let's consider our last example to see how this may work.  The row player can now change his strategy, by playing the row that yields the highest payoff on column 2 (given that the column player always plays column 2).  Thus, the row player will choose to play row 2, instead of row 3, which will reduce his loses from 1 to 0. Column Maxima New Strategy Smallest of column maxima 6 04 6 04 Example 3, page 515

93. Example  Let's consider our last example to see how this may work.  Note that this change in strategy works only if at least one of the other payoffs in column 2 is preferable to the payoff of row 3 played with the maximin strategy. Column Maxima 6 04 6 04 New Strategy Smallest of column maxima Example 3, page 515

94. Example  Let's consider our last example to see how this may work.  For example, what if a 22 were equal to –3 instead of 0?  Then the row player’s optimal strategy would be to play row 3, as with the maximin strategy, even though he knows that the column player is always going to play column 2. Column Maxima New Strategy 6 04 6 04 –3 –3 Smallest of column maxima Example 3, page 515

95. Optimal Strategy  The optimal strategy in a game is the strategy that is most profitable to a particular player.

96. Strictly Determined Game A strictly determined game is characterized by the following properties: 1.There is an entry in the payoff matrix that is simultaneously the smallest entry in its row and the largest entry in its column. This entry is called the saddle point for the game. 2.The optimal strategy for the row player is precisely the maximin strategy and is the row containing the saddle point. The optimal strategy for the column player is the minimax strategy and is the column containing the saddle point.

97. Example  For the game with the following payoff matrix, determine the optimal strategy for each player. Solution  The maximin strategy for the row player is to play row 2.  The minimax strategy for the column player is to play column 3. Column Maxima 3 4–3 3 4–3 Row Minima Largest of row minima Smallest of column maxima Example 4, page 515

98. Example  For the game with the following payoff matrix, determine the optimal strategy for each player. Solution  In repeated plays, R discovers that C consistently chooses to play column 3. Column Maxima 3 4–3 3 4–3 Row Minima Largest of row minima Smallest of column maxima Example 4, page 515

99.  For the game with the following payoff matrix, determine the optimal strategy for each player. Solution  Knowing that C always chooses column 3, R’s best choice is to play the row with the highest payoff in column 3.  Thus, R’s optimal strategy is to play row 2.  Note that row 2 was also the outcome of the maximin strategy. Example Column Maxima 3 4–3 3 4–3 Optimal Strategy Smallest of column maxima Example 4, page 515

100. Example  For the game with the following payoff matrix, determine the optimal strategy for each player. Solution  Let's consider now player C.  In repeated plays, C discovers that R consistently chooses to play row 2. Column Maxima 3 4–3 3 4–3 Row Minima Largest of row minima Smallest of column maxima Example 4, page 515

101.  For the game with the following payoff matrix, determine the optimal strategy for each player. Solution  Knowing that R always chooses Row 2, C’s best choice is to play the column with the lowest payoff in row 2.  Thus, C’s optimal strategy is to play column 3.  Note that column 3 was also the outcome of the minimax strategy. Example Optimal Strategy Row Minima Largest of row minima Example 4, page 515

102.  For the game with the following payoff matrix, determine the optimal strategy for each player. Solution  Thus, we conclude that whether the game is only played only once or repeatedly, the outcome is always the same: row 2 and column 3.  This is because this a strictly determined game, with entry a 23 as the saddle point for the game. Example Optimal Strategy Example 4, page 515

103. Saddle Point  We just saw that when a game has a saddle point, the optimal strategies for the players are to choose, respectively, the row and column that contain the saddle point.  Furthermore, in repeated plays of the game, each player’s optimal strategy consists of making the same move over and over again, since the discovery of the opponent’s optimal strategy cannot be used to advantage.  Such strategies are called pure strategies.

104. Saddle Point  The saddle point of a strictly determined game is also referred to as the value of the game.  If the value of a strictly determined game is positive, then the game favors the row player.  If the value is negative, it favors the column player.  If the value of the game is zero, the game is called a fair game.

105. 9.5 Games with Mixed Strategies Heads Tails

106. Mixed Strategies  Let’s consider a slightly modified version of the coin- matching game played by Richie and Chuck:  Note that it contains no saddle point and is therefore not strictly determined.  In this section, we shall look at games like this one, that are not strictly determined and the strategies associated with such games. Heads Tails HeadsTails C’s moves R’s moves Chuck Richie

107. Mixed Strategies  Let’s consider a slightly modified version of the coin- matching game played by Richie and Chuck:  What strategy should Richie follow?  It would seem that Richie should select row 1 since he stands to win $3 instead of the $1 he would get by playing row 2, at a risk, in either case, of losing $2. Heads Tails HeadsTails C’s moves R’s moves Chuck Richie

108. Mixed Strategies  Let’s consider a slightly modified version of the coin- matching game played by Richie and Chuck:  However, when Chuck realizes that Richie is consistently playing row 1, he would counter by playing column 2.  This would cause Richie to lose $2 on each play.  Thus, Richie considers a strategy of choosing row 1 sometimes and row 2 at other times. Heads Tails HeadsTails C’s moves R’s moves Chuck Richie

109. Mixed Strategies  Let’s consider a slightly modified version of the coin- matching game played by Richie and Chuck:  A similar analysis will lead Chuck to follow a strategy of choosing column 1 sometimes and column 2 at other times.  Such strategies are called mixed strategies. Heads Tails HeadsTails C’s moves R’s moves Chuck Richie

110. Mixed Strategies  There are many ways in which a player may choose moves in a game with mixed strategies.  Richie, for example, might choose to play heads half the time and tails half the time, making the choice randomly by, say, flipping a coin.  He could also determine beforehand the proportion of the time row 1 should be chosen, by means, say, of a spinning wheel: Heads Tails

111. Mixed Strategies  Mathematically, we could describe the mixed strategy as a row vector whose dimension coincides with the number of possible moves the player has.  For example, if Richie had decided on a strategy in which he chose to play row 1 half the time and row 2 half the time, then this strategy is represented by the row vector  Similarly, the mixed strategy for the column player can be represented by a column vector.  Let’s say that Chuck has decided that 20% of the time he will chose column 1 and 80% of the time he will chose column 2.  This strategy can be represented by the column vector

112. Expected Value of a Game  In order to compare the merits of a player’s different mixed strategies in a game, we can calculate the expected value of a game.  The expected value of a game measures the average payoff to the row player when both players adopt a particular set of mixed strategies.  We can now explain this notion using a 2 × 2 matrix game whose payoff matrix has the general form

113. Expected Value of a Game  Suppose in repeated plays of the game, the row player R adopts the mixed strategy of selecting row 1 with probability p 1 and row 2 with probability p 2.  The column player, in turn, adopts the mixed strategy of selecting column 1 with probability q 1 and column 2 with probability q 2.

114. Expected Value of a Game  Now, in each play of the game, there are four possible outcomes that may be represented by ordered pairs (row 1, column 1) (row 1, column 2) (row 2, column 1) (row 2, column 2) where the first number of each ordered pair represents R’s selection and the second represents C’s selection.

115. Expected Value of a Game  The choice of moves made by one player is made without knowing the other’s choice, making these independent events.  Therefore, the probability of R choosing row 1 and C choosing column 1, is given by P(row 1, column 1)= P(row 1) · P(column 1) P(row 1, column 1)= P(row 1) · P(column 1) = p 1 q 1 = p 1 q 1

116. Expected Value of a Game  Similarly, the probabilities of all four possible outcomes, together with the payoffs associated with each of the four possible outcomes, may be summarized as follows:  Thus, the expected payoff of the game is which can be expressed in terms of the matrices P, A, and Q: OutcomeProbabilityPayoff (row 1, column 1) p1q1p1q1p1q1p1q1 a 11 (row 1, column 2) p1q2p1q2p1q2p1q2 a 12 (row 2, column 1) p2q1p2q1p2q1p2q1 a 21 (row 2, column 2) p2q2p2q2p2q2p2q2 a 22

117. Expected Value of a Game  Let be the vectors representing the mixed strategies for the row player R and the column player C, respectively, in a game with an m × n payoff matrix

118. Expected Value of a Game  Then the expected value of the game is given by

119. Applied Example: Coin-Matching Game  Consider Richie and Chuck playing a coin-matching game with a payoff matrix given by  Compute the expected payoff of the game if Richie adopts the mixed strategy P and Chuck adopts the mixed strategy Q, where a. a. b. b. Applied Example 1, page 524

120. Applied Example: Coin-Matching Game Solution a.We compute Thus, in repeated plays of the game, it may be expected in the long-term that the payoff to each player is 0. Applied Example 1, page 524

121. Applied Example: Coin-Matching Game Solution b.We compute Thus, in the long run Richie may be expected to lose $1.06 on the average in each play. Applied Example 1, page 524

122. End of Chapter