Arguments for Recovering Cooperation Conclusions that some have drawn from analysis of prisoner’s dilemma: the game theory notion of rational action is wrong! somehow the dilemma is being formulated wrongly This isn’t rational. We may not defect for a few cents. If sucker’s payoff really hurts, more likely to be rational. 1Wool6Iterated Games

2 Arguments to recover cooperation: We are not all self-centered! But sometimes we are nice because there is a punishment. If we don’t give up seat on bus, we receive rude stares. If this were true, places like Honor Copy would be exploited. The other prisoner is my twin! When I decide what to do, the other agent will do the same. (but can’t force it, as wouldn’t be autonomous). Your mother would say, “What if everyone were to behave like that?” You say, “I would be a fool to act any other way.” The shadow of the future…we will meet again. Wool6Iterated Games

3 The Iterated Prisoner’s Dilemma One answer: play the game more than once If you know you will be meeting your opponent again, then the incentive to defect appears to evaporate Cooperation is the rational choice in the infinitely repeated prisoner’s dilemma (Hurrah!) Wool6Iterated Games

4 Backwards Induction But…suppose you both know that you will play the game exactly n times On round n - 1, you have an incentive to defect, to gain that extra bit of payoff… But this makes round n – 2 the last “real”, and so you have an incentive to defect there, too. This is the backwards induction problem. Playing the prisoner’s dilemma with a fixed, finite, pre-determined, commonly known number of rounds, defection is the best strategy Wool6Iterated Games

5 Axelrod’s Tournament Suppose you play iterated prisoner’s dilemma against a range of opponents… What strategy should you choose, so as to maximize your overall payoff? Axelrod (1984) investigated this problem, with a computer tournament for programs playing the prisoner’s dilemma Wool6Iterated Games

6 Axelrod’s tournament: invited political scientists, psychologists, economists, game theoreticians to play iterated prisoners dilemma All-D – always defect Random: randomly pick a strategy Tit-for-Tat – On first round cooperate. Then do whatever your opponent did last. Tester – first defect, If the opponent ever retaliated, then use tit-for-tat. If the opponent did not defect, cooperate for two rounds, then defect. Joss: Tit-for-tat, but 10% of the time, defect instead of cooperating. Wool6Iterated Games

7 Tit-for-Tat Why? Because you were averaging over all types of strategy If you played only All-D, tit-for-tat would lose. Wool6Iterated Games

8 Two Trigger Strategies Grim trigger strategy Cooperate until a rival deviates Once a deviation occurs, play non-cooperatively for the rest of the game Tit-for-tat Cooperate if your rival cooperated in the most recent period Cheat if your rival cheated in the most recent period Wool6Iterated Games

9 Axelrod's rules for success Do not be envious – not necessary to beat your opponent in order to do well. This is not zero sum. Do not be the first to defect. Be nice. Start by cooperating. Retaliate appropriately: Always punish defection immediately, but use “measured” force — don’t overdo it Don’t hold grudges: Always reciprocate cooperation immediately do not be too clever when you try to learn from the other agent, don’t forget he is trying to learn from you. Be forgiving – one defect doesn’t mean you can never cooperate The opponent may be acting randomly Wool6Iterated Games

10 The centipede game Jack stop (2, 0) Go on Jill stop (1, 4) Jill Go on stop (5, 3) Jack Go on stop (4, 7) Jill (98, 96) stop (99, 99) Go on (97, 100) stop Go on Jack Go on Jill (94, 97) Wool6Iterated Games

11 The centipede game Jack stop (2, 0) Go on Jill stop (1, 4) Jill Go on stop (5, 3) Jack Go on stop (4, 7) Jill (98, 96) stop (99, 99) Go on (97, 100) stop Go on Jack Go on Jill (94, 97) The solution to this game through roll back is for Jack to stop in the first round! Wool6Iterated Games

12 The centipede game What actually happens? In experiments the game usually continues for at least a few rounds and occasionally goes all the way to the end. But going all the way to the (99, 99) payoff almost never happens – at some stage of the game ‘cooperation’ breaks down. So still do not get sustained cooperation even if move away from ‘roll back’ as a solution Wool6Iterated Games

13 Lessons from finite repeated games Finite repetition often does not help players to reach better solutions Often the outcome of the finitely repeated game is simply the one-shot Nash equilibrium repeated again and again. There are SOME repeated games where finite repetition can create new equilibrium outcomes. But these games tend to have special properties For a large number of repetitions, there are some games where the Nash equilibrium logic breaks down in practice. Wool6Iterated Games

14 Threats Threatening retaliatory actions may help gain cooperation Threat needs to be believable Wool6Iterated Games

15 What is Credibility? “The difference between genius and stupidity is that genius has its limits.” – Albert Einstein You are not credible if you propose to take suboptimal actions.: If a rational actor proposes to play a strategy which earns suboptimal profit. How can one be credible? Wool6Iterated Games

16 non-credible threat A non-credible threat is a threat made by a player in a Sequential Game which would not be in the best interest for the player to carry out. The hope is that the threat is believed in which case there is no need to carry it out. While Nash equilibria may depend on non-credible threats, Backward Induction eliminates them.Sequential GameNash equilibriaBackward Induction Wool6Iterated Games

17 Trigger Strategy Extremes Tit-for-Tat is most forgiving shortest memory proportional credible but lacks deterrence Tit-for-tat answers: “Is cooperation easy?” Grim trigger is least forgiving longest memory MAD adequate deterrence but lacks credibility Grim trigger answers: “Is cooperation possible?” Wool6Iterated Games

18 concepts of rationality [doing the rational thing] undominated strategy (problem: too weak) can’t always find a single one (weakly) dominating strategy (alias “duh?”) (problem: too strong, rarely exists) Nash equilibrium (or double best response) (problem: equilibrium may not exist) randomized (mixed) Nash equilibrium – players choose various options based on some random number (assigned via a probability) Theorem [Nash 1952]: randomized Nash Equilibrium always exists Wool6Iterated Games

19 Mixed strategy equilibria  i (s j )) is the probability player i selects strategy sj (0,0,…1,0,…0) is a pure strategy Strategy profile:  =(  1,…,  n) Expected utility: u i (  )=  s  S (  j  (s j ))u i (s) (chance the combination occurs times utility) Nash Equilibrium:  * is a (mixed) Nash equilibrium if  i  i defines a probability distribution over Si ui(  * i,  * -i )  ui(  i,  * -i ) for all  i  i, for all i Wool6Iterated Games

20 Example: Matching Pennies no pure strategy Nash Equilibrium -1, 11,-1 -1, 1 H HT T So far we have talked only about pure strategy equilibria [I make one choice.]. Not all games have pure strategy equilibria. Some equilibria are mixed strategy equilibria. Wool6Iterated Games

21 Example: Matching Pennies -1, 11,-1 -1, 1 p H q H1-q T 1-p T Want to play each strategy with a certain probability. If player 2 is optimally mixing strategies, player 1 is indifferent between his own choices! Compute expected utility given each pure possibility of other player. Wool6Iterated Games

22 I am player 2. What should I do? I pick a defensive strategy If player1 picks head: -q+(1-q) If Player 1 picks tails q + -(1-q) Want my opponent NOT to care what I pick. The idea is, if my opponent gets excited about what my strategy is, it means I have left open an opportunity for him. When he doesn’t have to analyze what he should do, it says there is no way he wins big. So: -q +(1-q) =q + -1+q 1-2q=2q-1 so q=1/2 Wool6Iterated Games

23 Example: Bach/Stravinsky 2, 10,0 1, 2 p B q B1-q S 1-p S Want to play each strategy with a certain probability. If player 2 is optimally mixing strategies, player 1 is indifferent to what player1 does. Compute expected utility given each pure possibility of yours. p = 2(1-p) p=2/3 2q = (1-q)q=1/3 player 1 is optimally mixing player 2 is optimally mixing Wool6Iterated Games

24 This is consistent with Dan’s advice –look after yourself. ‘ “I Used to Think I Was Indecisive - But Now I’m Not So Sure” -Anonymous Wool6Iterated Games

25 Mixed Strategies Unreasonable predictors of one-time human interaction Reasonable predictors of long-term proportions Wool6Iterated Games

26 Employee Monitoring Employees can work hard or shirk Salary: $100K unless caught shirking Cost of effort: $50K (We are assuming that when he works he loses something. Think of him running a business of his own while getting paid as his day job – so if he works, he can’t do that and loses the money the business makes.) Managers can monitor or not Value of employee output: $200K (We assume he must be worth more than we pay him to cover profit, infrastructure, manager time, mistakes, etc.) Profit if employee doesn’t work: $0 Cost of monitoring: $10K Wool6Iterated Games

27 From the problem statement, VERIFY the numbers in the table are correct. No equilibrium in pure strategies - SHOW IT What do the players do in mixed strategies? DO AT SEATS Please do not consider this instruction for how to cheat your boss. Rather, think of it as advice in how to deal with employees. Employee Monitoring Manager MonitorNo Monitor Employee Work 50, 9050, 100 Shirk 0, , -100 Wool6Iterated Games

28 Mixed Strategies Randomize – surprise the rival Mixed Strategy: Specifies that an actual move be chosen randomly from the set of pure strategies with some specific probabilities. Nash Equilibrium in Mixed Strategies: A probability distribution for each player The distributions are mutual best responses to one another in the sense of expectations Wool6Iterated Games

29 Finding Mixed Strategies Suppose: Employee chooses (shirk, work) with probabilities (p,1-p) Manager chooses (monitor, no monitor) with probabilities (q,1- q) Find expected payoffs for each player Use these to calculate best responses Wool6Iterated Games

30 Employee’s Payoff First, find employee’s expected payoff from each pure strategy If employee works: receives 50 Profit(work)= 50  q+ 50  (1-q) = 50 If employee shirks: receives 0 or 100 Profit(shirk)= 0  q+ 100  (1-q) = 100 – 100q Wool6Iterated Games

31 Employee’s Best Response Next, calculate the best strategy for possible strategies of the opponent For q<1/2: SHIRK Profit (shirk) = q > 50 = Profit(work) SHIRK For q>1/2: WORK Profit (shirk) = q < 50 = Profit(work) WORK For q=1/2: INDIFFERENT Profit(shirk) = q = 50 = Profit(work) ???? Wool6Iterated Games

32 Manager’s Best Response u2(mntr)= 90  (1-p)- 10  p u2(no m)= 100  (1-p)-100  p For p<1/10: NO MONITOR u2 (mntr) = p < p = u2 (no mntr) For p>1/10: MONITOR u2 (mntr) = p > p = u2 (no mntr) For p=1/10: INDIFFERENT u2 (mntr) = p = p = u2 (no mntr) Wool6Iterated Games

33 Cycles q 01 1/2 p 0 1/10 1 shirk work monitorno monitor

34 Mixed Strategy Equilibrium Employees shirk with probability 1/10 Managers monitor with probability ½ Expected payoff to employee: chance of each of four outcomes x payoff from each Expected payoff to manager: Wool6Iterated Games

35 Properties of Equilibrium Both players are indifferent between any mixture over their strategies E.g. employee: If shirk: If work: Regardless of what employee does, expected payoff is the same Wool6Iterated Games

36 Use Indifference to Solve I q 1-q MonitorNo Monitor Work 50, 90 50, 100= 50q+50(1-q) Shirk 0, , -100= 0q+100(1-q) 50q+50(1-q) = 0q+100(1-q) 50 = q 50 = 100q q = 1/2 Wool6Iterated Games

37 Use Indifference to Solve II MonitorNo Monitor 1-pWork 50, 90 50, 100 pShirk 0, , -100 = 90(1-p)-10p= 100(1-p)-100p 90(1-p)-10p = 100(1-p)-100p p = 100 – 200p 100p = 10 p = 1/10 Wool6Iterated Games

38 Indifference 1/2 MonitorNo Monitor 9/10Work 50, 90 50, 100= 50 1/10Shirk 0, , -100= 50 = 80 Wool6Iterated Games

39 Upsetting? This example is upsetting as it appears to tell you, as workers, to shirk. Think of it from the manager’s point of view, assuming you have unmotivated (or unhappy) workers. A better option would be to hire dedicated workers, but if you have people who are trying to cheat you, this gives a reasonable response. Sometimes you are dealing with individuals who just want to beat the system. In that case, you need to play their game. For example, people who try to beat the IRS. On the positive side, even if you have dishonest workers, if you get too paranoid about monitoring their work, you lose! This theory tells you to lighten up! This theory might be applied to criticising your friend or setting up rules/punishment for your (future?) children. Wool6Iterated Games

40 Why Do We Mix? I don’t want to give my opponent an advantage. When my opponent can’t decide what to do based on my strategy, I win – as there is not way he is going to take advantage of me. COMMANDMENT Use the mixed strategy that keeps your opponent guessing. Wool6Iterated Games

41 Mixed Strategy Equilibriums Anyone for tennis? Should you serve to the forehand or the backhand? Wool6Iterated Games

42 Tennis Payoffs Wool6Iterated Games

43 Tennis: Fixed Sum If you win (the points), I lose (the points) AKA: Strictly competitive q1-q p 1-p Wool6Iterated Games

44 Solving for Server’s Optimal Mix What would happen if the the server always served to the forehand? A rational receiver would always anticipate forehand and 90% of the serves would be successfully returned. Wool6Iterated Games

45 Solving for Server’s Optimal Mix What would happen if the the server aimed to the forehand 50% of the time and the backhand 50% of the time and the receiver always guessed forehand? (0.5*0.9) + (0.5*0.2) = 0.55 successful returns Wool6Iterated Games

46 Solving for Server’s Optimal Mix What is the best mix for each player? Receiver thinks: if server serves forehand.10*p +.70*(1-p) if server serves backhand.80*p +.40*(1-p) I want them to be the same.10*p +.70*(1-p) =.80*p +.40*(1-p).10*p p =.80*p p -.6p+.7 =.4p =p Use similar argument to solve for q - Wool6Iterated Games

Draw a graph which shows two lines (1) the utility of server of “picking forehand” as a function of p. (2) the utility of server of “picking backhand” as a function of p. Wool6Iterated Games47

What can you learn from the graph? Wool6Iterated Games48

Receiver’s view of opponent Above 1/3, backhand wins. Wool6Iterated Games49 p

Receiver’s view of opponent Above.3, serving backhand wins. Wool6Iterated Games50 p

Server’s view of opponent Above.4 plan forehand wins Wool6Iterated Games 51 q

52 % of Successful Returns Given Server and Receiver Actions Where would you shoot knowing the other player will respond to your choices? In other words, you pick the row but will likely get the smaller value in a row. Wool6Iterated Games

53 Consider: Bach or Stravinsky If the other player is maximally mixing, my payoffs are the same, so 2(Y) = 1(1-Y); Y = 1/3 1(X) = 2 (1-X); X =2/3 2,10,0 1,2 B(X) B (Y)S(1-Y) S(1-X) No dom. str. equil. Xavier Yolanda Wool6Iterated Games

54 Best Response Function If 0 < Y < 1/3, then player 1’s best response is X=0. If y = 1/3, then ALL of player 1’s responses are best responses If y > 1/3, then player 1’s best response is X=1. Using excel, prove this to yourself! Wool6Iterated Games

55 Best Response Function (The dotted line is a function only if you mentally switch the axes.) Y X 1/3 1 2/3 1 Fixed Point – where best response functions intersect is the nash Equilibrium The best response of player 1 is shown as a dotted line. Wool6Iterated Games

56 pq player 1player pqplayer 1 player pqplayer 1 play er

57 Hints to understanding graph: The solid line represents Yolanda’s thinking. If Xavier is going to select B less than 2/3’s of the time, Yolanda is best selecting S (which happens when Y=0). HOWEVER, if Xavier is going to select B more than 2/3’s of the time, Xavier should immediately start selecting S (which happens when y=1). Wool6Iterated Games

58 Computing mixed stategies for two players (the book’s way) Write the matrix game in bi matrix form A=[a ij ] B=[b ij ] Compute payoffs Replace p m =1- and q n =1- Consider the partial derviatives of  1 and  2 with respect to all pi and all qi respectively. Solve system of equations with all partials set to zero

59 Example  1 = 3 p1q1 + p2q2 = 3p1q1 +(1-p1)(1-q1)= 1 +-p1 –q1 +4p1q1  2 = p1q1 + 4p2q2 = p1q1 +4(1-p1)(1-q1)= 4 -4p1-4q1 +5p1q1 d  1 /dp1 = -1 +4q1 so q1 = ¼ d  2 /dq1 = -4 +5p1 so p1 = 4/5 So strategies are ((4/5,1/5)(¼, ¾))

60 Example 2  1 = 3 p1q1 + -p1q2 -2p2q1 +p2q2 = 3 p1q1 + -p1(1-q1) -2(1-p1)q1 +(1-p1)(1-q1) =3p1q1 –p1 +p1q1 -2q1 +2p1q p1 –q1 +p1q1 =1+7p1q1-2p1-3q1  2 = p1q1 + 4p2q2 = p1q1 +4(1-p1)(1-q1)= 4 -4p1-4q1 +5p1q1 d  1 /dp1 = -2 +7q1 so q1 = 2/7 d  2 /dq1 = -4 +5p1 so p1 = 4/5 So strategies are ((4/5,1/5)(2/7,5/7))

61 Tennis Example  1 = 90 p1q1 + 20p1q2 +30p2q1 +60p2q2 = 90pq +20p(1-q) + 30(1-p)q +60(1-p)(1-q)= 90pq + 20p-20pq +30q-30pq p-60q+60pq= pq -40p -30q  2 = 10pq + 80p(1-q)+70(1-p)q+40(1-p)(1-q) = 10pq +80p -80pq +70q-70pq+40-40p-40q+40pq= -100pq +40p+30q +40 d  1 /dp1 =100q-40 so q =.4 d  2 /dq1 = -100p +30 so p =.3 So strategies are ((.3,.7)(.4,.6))