1. Multiagent Systems
Repeated Games
© Manfred Huber 2018

2. Repeated Games
Repeated games are an extension to normal form games where the same single-stage game is played multiple times sequentially
What information is available to the agents ?
Utility of a repeated game ?
Strategies in repeated games ?
There are two main types of repeated games
Finitely-repeated games
Infinitely-repeated games
© Manfred Huber 2018

3. Finitely-Repeated Games
A fixed single-stage game is repeated a finite number of time
After each game agents are aware what the other agents did in the previous stages
Payoff for the repeated game is the sum of the payoffs in the stage games
Finitely-repeated games can be translated into imperfect information games in extensive form
© Manfred Huber 2018

4. Finitely-Repeated Games
Twice repeated Prisoners’ Dilemma C S
-5, -5
-1, -10
-10, -1
-3, -3 C S
-5, -5
-1, -10
-10, -1
-3, -3  1 2 C S
(-5,-5)
(-1,-10)
(-10,-1)
(-3,-3) 1 2 C S 1 2 C S 1 2 C S 1 2 C S 1 2 C S
(-10,
-10)
(-6,
-15)
(-15,
-6)
(-8,
-8)
(-6,
-15)
(-2,
-20)
(-11,
-11)
(-4,
-13)
(-15,
-6)
(-11,
-11)
(-20,
-2)
(-13,
-4)
(-8,
-8)
(-4,
-13)
(-13,
-4)
(-6,
-6)
© Manfred Huber 2018

5. Strategies in Finitely-Repeated Games
Strategy space is richer than the strategy space for the single stage normal form game
Strategies can depend on actions played in previous iterations of the game
Stationary strategies are a special case where the strategy reduces to the same strategy for each single-stage game
© Manfred Huber 2018

6. Nash Equilibria in Finitely-Repeated Games
Nash equilibria can be computed using the corresponding extensive form game
Exponential increase in complexity with the number of repetitions
A stationary strategy repeating a Nash equilibrium for the stage game is a Nash equilibrium for the repeated game
In cases where the stage game has a dominant strategy, backward induction can be used to determine a Nash equilibrium
© Manfred Huber 2018

7. Infinitely-Repeated Games
A fixed single-stage game is repeated an infinite number of times
After each game agents are aware what the other agents did in the previous stages
Can not be converted into an extensive form game
Conversion would require an infinite tree
Payoff for the repeated game can not be calculated as the sum of payoffs of the stage games
Sum of payoffs is in general infinite (not a usable utility)
© Manfred Huber 2018

8. Utility/Payoff in Infinitely-Repeated Games
Average reward: game payoff is measured as the average payoff of each stage game
Future discounted reward: game payoff is measured as the discounted sum of payoff of the stage games
Two interpretations of discounted reward:
The agent cares less about future returns than current ones
The game ends with probability 1-beta in each round
© Manfred Huber 2018

9. Strategies and Equilibria in Infinitely-Repeated Games
A pure strategy defines an action choice for every stage game (decision point)
Strategies are infinite
There is an infinite number of pure strategies
Famous strategies:
Tit-for-tat
Trigger strategy
© Manfred Huber 2018

10. Nash Equilibria in Infinitely-Repeated Games
A stationary strategy repeating a Nash equilibrium for the stage game is again a Nash equilibrium for the repeated game
Other equilibria can depend on the choice of payoff function and the discount parameter (in the case of future discounted reward)
There can be an infinite number of pure strategy equilibria
There is no way to construct the induced normal form or the sequence form for infinitely repeated games
© Manfred Huber 2018

11. Computing Nash Equilibria in Infinitely-Repeated Games
Nash equilibria for infinitely repeating games can not be calculated by reducing the problem to a normal (or extensive) form game
The payoff received in a Nash equilibrium of an infinitely repeating game can be characterized independently of the strategy by looking only at the stage game
The average payoffs in an equilibrium have to be achievable under a mixed strategy in the stage game
The average payoff achieved for player i in an equilibrium has to be at least as big as the one received if the other players adopt a minmax strategy against it
© Manfred Huber 2018

12. Computing Nash Equilibria The Folk Theorem
Consider an n-player game G=(N,A,u) and a payoff vector r=(r1, …, rn)
The minmax value of an agent is the value it can achieve if all other agents are antagonistic (i.e. attempt to minimize the agent’s payoff)
minmax value:
minmax represents a conservative value below which other agents can not push agent i ’s payoff
© Manfred Huber 2018

13. Enforceability
A payoff profile is enforceable if for every agent the payoff is at least as high as its minmax value
© Manfred Huber 2018

14. Feasibility
A payoff profile is feasible if it is a convex, rational (non-negative) combination of the outcomes of the stage game
© Manfred Huber 2018

15. The Folk Theorem
In any n-player infinitely repeated game with stage game G, the following holds:
If r is the payoff profile for a Nash equilibrium in the infinitely repeated game with average rewards, then ri is enforceable
If ri were not enforceable then agent i would receive a better payoff when pursuing the minmax strategy
If r is feasible and enforceable then r is the payoff profile for some Nash equilibrium of the infinitely repeated game with average rewards
If r is feasible and enforceable then we can construct a Nash equilibrium strategy in the form of a trigger strategy that follows the mixed strategy using the rational weights for enforcability and the minmax strategy after an agent diverts from the policy.
© Manfred Huber 2018

16. The Folk Theorem Constructing Equilibrium Strategies
Find a payoff profile that is enforceable and feasible
Find the rational coefficients for feasibility
Construct a trigger strategy that performs the following:
As long as no agent deviates from it, follow a pure strategy that repeatedly cycles through all pure strategies a of the stage game G, executing them βa times
If in any cycle one of the agents deviates from this, the other agents will follow that agent’s minmax strategy for the rest of the game
© Manfred Huber 2018

17. Infinitely-Repeated Games and Bounded Rationality
Solutions to infinitely repeated games assume:
Arbitrarily deep reasoning
Arbitrarily deep mutual modeling
In real problems this is rarely feasible
Discounted reward can limit the reliance on perfect future information
Bounded rationality can address this problem
Approximate ε-Nash equilibria
Machine games: Games played by automata (limiting memory and strategy complexity)
© Manfred Huber 2018