February 2, 2016 Stochastic Games Mr Sujit P Gujar. e-Enterprise Lab Computer Science and Automation IISc, Bangalore.

February 2, 2016e-Enterprise Lab Agenda Stochastic Game Special Class of Stochastic Games Analysis : Shapley’s Result. Applications

February 2, 2016e-Enterprise Lab Repeated Game When players interact by playing a similar stage game (such as the prisoner's dilemma) numerous times, the game is called a repeated game.prisoner's dilemma

February 2, 2016e-Enterprise Lab Stochastic Game Stochastic game is repeated game with probabilistic/stochastic transitions. There are different states of a game. Transition probabilities depend upon actions of players. Two player stochastic game : 2 and 1/2 player game.

February 2, 2016e-Enterprise Lab Repeated Prisoner’s Dilemma Consider Game tree for PD repeated twice. What is Player 1’s strategy set? (Cross product of all choice sets at all information sets…) {C,D} x {C,D} x {C,D} x {C,D} x {C,D} 2 5 = 32 possible strategies First Iteratio n Second Iteratio n subga me 1 2 Assume each player has the same two options at each info set: {C,D}

February 2, 2016e-Enterprise Lab Issues in Analyzing Repeated Games How to we solve infinitely repeated games? Strategies are infinite in number. Need to compare sums of infinite streams of payoffs

February 2, 2016e-Enterprise Lab Stochastic Game : The Big Match Every day player 2 chooses a number, 0 or 1 Player 1 tries to predict it. Wins a point if he is correct. This continues as long as player 1 predicts 0. But if he ever predicts 1, all future choices for both players are required to be the same as that day's choices.

February 2, 2016e-Enterprise Lab The Big Match S = {0,1 *,2 * } : State space P 01 = s 0 ={0,1} s 1 ={0} s 2 ={1} P 02 = N = {1,2} P 00 = A = Payoff Matrix = 1*1* 0*0* 01

February 2, 2016e-Enterprise Lab The "Big-Match" game is introduced by Gillette (1957) as a difficult example. The Big Match David Blackwell; T. S. Ferguson The Annals of Mathematical Statistics, Vol. 39, No. 1. (Feb., 1968), pp

February 2, 2016e-Enterprise Lab Scenario NTotal number of States/Positions mkmk Choices for row player at position k nknk Choices for column player at position k s k ij > 0The probability with which the game in position k stops when player 1 plays i and player 2, j. p kl ij The probability with which the game in position k moves to l when player 1 plays i and player 2, j. sMin s k ij a k ij Payoff to row player in stage k. MMax |a k ij |

February 2, 2016e-Enterprise Lab Stationary Strategies Enumerating all pure and mixed strategies is cumbersome and redundant. Behavior strategies those which specify a player the same probabilities for his choices every time the same position is reached by whatever route. x = (x 1,x 2,…,x N ) each x k = (x k 1, x k 2,…, x k m k )

February 2, 2016e-Enterprise Lab Notation Given a matrix game B, val[B] = minimax value to the first player. X[B] = The set of optimal strategies for first player. Y[B] = The set of optimal strategies for second player. It can be shown, (B and C having same dimensions) |val[B] - val[C]| ≤ max |b ij - c ij |

February 2, 2016e-Enterprise Lab When we start in position k, we obtain a particular game, We will refer stochastic game as, Define,

February 2, 2016e-Enterprise Lab Shapley’s 1 Results 1 L.S. Shapley, Stochastic Games. PNAS 39(1953)

February 2, 2016e-Enterprise Lab Let, denote the collection of games whose pure strategies are the stationary strategies of. The payoff function of these new games must satisfy,

February 2, 2016e-Enterprise Lab Shapley’s Result,

February 2, 2016e-Enterprise Lab Applications 1 When N = 1, By setting all s k ij = s > 0, we get model of infinitely repeated game with future payments are discounted by a factor = (1-s). If we set n k = 1 for all k, the result is “dynamic programming model”. 1 von Neumann J., Ergennise eines Math, Kolloquims, (1937)

February 2, 2016e-Enterprise Lab Example Consider the game with N = 1, A = 1-s P1 = x=(0.6,0.4) y=(0.4, 0.6) 1-2s 1-s1-2s P2 = x=(0.61,0.39) y=(0.39, 0.61)

February 2, 2016e-Enterprise Lab Thank You!!