Krishnendu Chatterjee1 Graph Games with Reachabillity Objectives: Mixing Chess, Soccer and Poker Krishnendu Chatterjee 5 th Workshop on Reachability Problems, Genova, Sept 30, 2011 TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A A A

Krishnendu Chatterjee2 Games on Graphs  Games on graphs.  History  Zermelo’s theorem about Chess in 1913  From every configuration  Either player 1 can enforce a win.  Or player 2 can enforce a win.  Or both players can enforce a draw.

Krishnendu Chatterjee3 Chess: Games on Graph  Chess is a game on graph.  Configuration graph.

Krishnendu Chatterjee4 Graphs vs. Games Two interacting players in games: Player 1 (Box) vs Player 2 (Diamond).

Krishnendu Chatterjee5 Game Graph

Krishnendu Chatterjee6 Game Graphs  A game graph G= ((S,E), (S 1, S 2 ))  Player 1 states (or vertices) S 1 and similarly player 2 states S 2, and (S 1, S 2 ) partitions S.  E is the set of edges.  E(s) out-going edges from s, and assume E(s) non- empty for all s.  Game played by moving tokens: when player 1 state, then player 1 chooses the out-going edge, and if player 2 state, player 2 chooses the outgoing edge.

Krishnendu Chatterjee7 Game Example

Krishnendu Chatterjee8 Game Example

Krishnendu Chatterjee9 Game Example

Krishnendu Chatterjee10 Strategies  Strategies are recipe how to move tokens or how to extend plays. Formally, given a history of play (or finite sequence of states), it chooses a probability distribution over out-going edges.  ¾ : S * S 1  D(S).  ¼ : S * S 2 ! D(S).

Krishnendu Chatterjee11 Strategies  Strategies are recipe how to move tokens or how to extend plays. Formally, given a history of play (or finite sequence of states), it chooses a probability distribution over out-going edges.  ¾ : S * S 1 ! D(S).  History dependent and randomized.  History independent: depends only current state (memoryless or positional).  ¾ : S 1 ! D(S)  Deterministic: no randomization (pure strategies).  ¾ : S * S 1 ! S  Deterministic and memoryless: no memory and no randomization (pure and memoryless and is the simplest class).  ¾ : S 1 ! S  Same notations for player 2 strategies ¼.

Krishnendu Chatterjee12 Objectives  Objectives are subsets of infinite paths, i.e., Ã µ S !.  Reachability: there is a set of good vertices (example check-mate) and goal is to reach them. Formally, for a set T if vertices or states, the objective is the set of paths that visit the target T at least once.

Krishnendu Chatterjee13 Applications: Verification and Control of Systems  Verification and control of systems  Environment  Controller M satisfies property ( Ã ) E C

Krishnendu Chatterjee14 Applications: Verification and Control of Systems  Verification and control of systems  Question: does there exists a controller that against all environment ensures the property. M satisfies property ( Ã ) EC ||

Krishnendu Chatterjee15 Applications: Systems for Specification  Synthesis of systems from specification  Input/Output signals.  Automata over I/O that specifies the desired set of behaviors.  Can the input player present input such that no matter how the output player plays the generated sequence of I/O signals is accepted by automata ?  Deterministic automata: Games.

Krishnendu Chatterjee16 -synthesis [Church, Ramadge/Wonham, Pnueli/Rosner] -model checking of open systems -receptiveness [Dill, Abadi/Lamport] -semantics of interaction [Abramsky] -non-emptiness of tree automata [Rabin, Gurevich/ Harrington] -behavioral type systems and interface automata [deAlfaro/ Henzinger] -model-based testing [Gurevich/Veanes et al.] -etc. Game Models Applications

Krishnendu Chatterjee17 Reachability Games  Pre(X): given a set X of states, Pre(X) is the set of states such that player 1 can ensure next state in X. X T

Krishnendu Chatterjee18 Reachability Games  Pre(X): given a set X of states, Pre(X) is the set of states such that player 1 can ensure next state in X. X T

Krishnendu Chatterjee19 Reachability Games  Pre(X): given a set X of states, Pre(X) is the set of states such that player 1 can ensure next state in X. X T

Krishnendu Chatterjee20 Reachability Games  Pre(X): given a set X of states, Pre(X) is the set of states such that player 1 can ensure next state in X.  Fix-point X T

Krishnendu Chatterjee21 Reachability Games  Winning set for a partition: Determinacy  Player 1 wins: then no matter what player 2 does, certainly reach the target.  Player 2 wins: then no matter what player 1 does, the target is never reached.  Memoryless winning strategies.  Can be computed in linear time [Beeri 81, Immerman 81].

Krishnendu Chatterjee22 Chess Theorem  Zermelo’s Theorem Win 1 Win 2 Both draw

Krishnendu Chatterjee23 Game Graphs Till Now  Game graphs we have seen till now  Many rounds (possibly infinite).  Turn-based.

Krishnendu Chatterjee24 Simultaneous Games  Theory of rational behavior as game theory  von Neumann- Morgenstern games  Matrix zero-sum games R P S R (0,0) (-1,1) (1,-1) P (1,-1) (0,0) (-1,1) S (-1,1) (1,-1) (0,0)

Krishnendu Chatterjee25 Simultaneous Games  Theory of rational behavior as game theory  von Neumann- Morgenstern games  Matrix zero-sum games R P S R (0,0) (-1,1) (1,-1) P (1,-1) (0,0) (-1,1) S (-1,1) (1,-1) (0,0)

Krishnendu Chatterjee26 Simultaneous Games  Example: Prisoners dilemma.  Another example. R L C R (1,-1) (-1,1) (-1,1) L (-1,1) (1,-1) (-1,1) C (-1,1) (-1,1) (1,-1)

Krishnendu Chatterjee27 Simultaneous Games  Example: Prisoners dilemma.  Another example. R L C R (1,-1) (-1,1) (-1,1) L (-1,1) (1,-1) (-1,1) C (-1,1) (-1,1) (1,-1)

Krishnendu Chatterjee28 Simultaneous Games  Another example: Penalty shoot-out (Soccer) R L C R (1,-1) (-1,1) (-1,1) L (-1,1) (1,-1) (-1,1) C (-1,1) (-1,1) (1,-1)

Krishnendu Chatterjee29 Chess Vs. Soccer (Penalty)  Chess:  Turn-based  Possibly infinite rounds  Theory of simultaneous games (Soccer)  Concurrent  One-shot (one-round)  Mix chess and soccer  Concurrent games on graphs

Krishnendu Chatterjee30 Mixing Chess and Soccer: Concurrent Graph Games

Krishnendu Chatterjee31 Concurrent Game Graphs A concurrent game graph is a tuple G =(S,M, ¡ 1, ¡ 2, ± ) S is a finite set of states. M is a finite set of moves or actions. ¡ i : S ! 2 M n ; is an action assignment function that assigns the non-empty set ¡ i (s) of actions to player i at s, where i 2 {1,2}. ± : S £ M £ M ! S, is a transition function that given a state and actions of both players gives the next state.

Krishnendu Chatterjee32 An Example: Snow-ball Game s R run, wait hide, throw hide, wait run, throw [Everett 57] Run Hide Throw Wait

Krishnendu Chatterjee33 New Solution Concepts  Sure winning for turn-based.  New solution concepts  Almost-sure winning.  Limit-sure winning.

Krishnendu Chatterjee34 Almost-sure Winning Example s R head, head tail, tail head, tail tail, head Almost-sure winning strategy: say head and tail with probability ½. Randomization is necessary.

Krishnendu Chatterjee35 Concurrent reachability games: limit-sure s R run, wait hide, throw hide, wait run, throw [Everett 57] Move Probability run q hide 1-q (q>0) Win at s with probability 1-q, for all q > 0. Run Hide Throw Wait

Krishnendu Chatterjee36 Concurrent reachability games: limit-sure s R run, wait hide, throw hide, wait run, throw Run Hide Throw Wait [Everett 57] Move Probability run q hide 1-q (q>0) Win at s with probability 1-q, for all q > 0. w = 011 Player 1 cannot achieve w(s) = 1, only w(s) = 1-q for all q > 0.

Krishnendu Chatterjee37 Concurrent reachability games  Almost-sure winning: requires alternation of mu-calculus formula.  Positive to lower rank and with probability 1 in the almost-sure winning set. R(1) R R(0)

Krishnendu Chatterjee38 Concurrent reachability games  Limit-sure winning: requires alternation of mu-calculus formula.  Positive to lower rank and can allow to escape but with vanishing probability. Let ® be green prob, and ¯ be red prob, then it can be ensured that ¯ · ® ¢ ², for all ² >0. R(1) R R(0)

Krishnendu Chatterjee39 Results for Concurrent Reachability Games  Sure winning:  Deterministic memoryless sufficient.  Linear time.  Almost-sure winning:  Randomization is necessary.  Randomized memoryless is sufficient.  Quadratic time algorithm.  Limit-sure winning:  Randomization is necessary.  Randomized memoryless is sufficient.  Quadratic time algorithm.  Results from [dAHK98, CdAH06, CdAH09]

Krishnendu Chatterjee40 Games Till Now  Turn-based graph games  Concurrent graph games  Applications: again verification and synthesis with synchronous interaction.  Both these games are perfect-information games. Players know the precise state of the game.  The game of Poker: players play but do not know the perfect state of the game.

Krishnendu Chatterjee41 Summary: Theory of Graph Games Winning Mode/ Game Graphs SureAlmost-sureLimit-sure Turn-based Games (CHESS) Linear time (PTIME-complete) Linear-time (PTIME-complete) Linear-time (PTIME-complete) Concurrent Games (CHESS+ SOCCER) Linear time (PTIME-complete) Quadratic time (PTIME-complete) Quadratic time (PTIME-complete) Partial-information Games (CHESS + SOCCER+ POKER)

Krishnendu Chatterjee42 Mixing Chess and Poker: Partial-information Graph Games

Krishnendu Chatterjee43 Why Partial-information  Perfect-information: controller knows everything about the system.  This is often unrealistic in the design of reactive systems because systems have internal state not visible to controller (private variables) noisy sensors entail uncertainties on the state of the game  Partial-observation Hidden variables = imperfect information. Sensor uncertainty = imperfect information.

Krishnendu Chatterjee44 Partial-information Games  A PIG G =(L, A, , O) is as follows  L is a finite set of locations (or states).  A is a finite set of input letters (or actions).   µ L £ A £ L non-deterministic transition relation that for a state and an action gives the possible next states.  O is the set of observations and is a partition of the state space. The observation represents what is observable.  Perfect-information: O={{l} | l 2 L}.

Krishnendu Chatterjee45 PIG: Example a,b a ba b

Krishnendu Chatterjee46 New Solution Concepts  Sure winning: winning with certainty (in perfect information setting determinacy).  Almost-sure winning: win with probability 1.  Limit-sure winning: win with probability arbitrary close to 1.  We will illustrate the solution concepts with card games.

Krishnendu Chatterjee47 Card Game 1  Step 1: Player 2 selects a card from the deck of 52 cards and moves it from the deck (player 1 does not know the card).  Step 2:  Step 2 a: Player 2 shuffles the deck.  Step 2 b: Player 1 selects a card and view it.  Step 2 c: Player 1 makes a guess of the secret card or goes back to Step 2 a.  Player 1 wins if the guess is correct.

Krishnendu Chatterjee48 Card Game 1  Player 1 can win with probability 1: goes back to Step 2 a until all 51 cards are seen.  Player 1 cannot win with certainty: there are cases (though with probability 0) such that all cards are not seen. Then player 1 either never makes a guess or makes a wrong guess with positive probability.

Krishnendu Chatterjee49 Card Game 2  Step 1: Player 2 selects a new card from an exactly same deck and puts is in the deck of 52 cards (player 1 does not know the new card). So the deck has 53 cards with one duplicate.  Step 2:  Step 2 a: Player 2 shuffles the deck.  Step 2 b: Player 1 selects a card and view it.  Step 2 c: Player 1 makes a guess of the secret duplicate card or goes back to Step 2 a.  Player 1 wins if the guess is correct.

Krishnendu Chatterjee50 Card Game 2  Player 1 can win with probability arbitrary close to 1: goes back to Step 2 a for a long time and then choose the card with highest frequency.  Player 1 cannot win probability 1, there is a tiny chance that not the duplicate card has the highest frequency, but can win with probability arbitrary close to 1, (i.e., for all ² >0, player 1 can win with probability 1- ², in other words the limit is 1).

Krishnendu Chatterjee51 Sure winning for Reachability  Result from [Reif 79]  Memory is required.  Exponential memory required.  Subset construction: what subsets of states player 1 can be. Reduction to exponential size turn-based games.  EXPTIME-complete.

Krishnendu Chatterjee52 Partial-information Games a a a a b b a b a a b b In starting play a. In yellow play a and b at random. In purple: if last was yellow then a if last was starting, then b. Requires both randomization and memory

Krishnendu Chatterjee53 Almost-sure winning for Reachability  Result from [CDHR 06, CHDR 07]  Standard subset construction fails: as it captures only sure winning, and not same as almost-sure winning.  More involved subset construction is required.  EXPTIME-complete.

Krishnendu Chatterjee54 Summary: Theory of Graph Games Winning Mode/ Game Graphs SureAlmost-sureLimit-sure Turn-based Games (CHESS) Linear time (PTIME-complete) Linear-time (PTIME-complete) Linear-time (PTIME-complete) Concurrent Games (CHESS+ SOCCER) Linear time (PTIME-complete) Quadratic time (PTIME-complete) Quadratic time (PTIME-complete) Partial-information Games (CHESS + SOCCER+ POKER) EXPTIME-complete

Krishnendu Chatterjee55 Limit-sure winning for Reachability  Limit-sure winning for reachability is undecidable [GO 10, CH 10].  Reduction from the Post-correspondence problem (PCP).

Krishnendu Chatterjee56 Mixing Chess, Soccer and Poker  Partial-information concurrent games  Concurrency can be obtained for free (polynomial reduction) for partial-information games.  So all the results for partial-information turn-based games also hold for partial-information concurrent games.

Krishnendu Chatterjee57 Summary: Theory of Graph Games Winning Mode/ Game Graphs SureAlmost-sureLimit-sure Turn-based Games (CHESS) Linear time (PTIME-complete) Linear-time (PTIME-complete) Linear-time (PTIME-complete) Concurrent Games (CHESS+ SOCCER) Linear time (PTIME-complete) Quadratic time (PTIME-complete) Quadratic time (PTIME-complete) Partial-information Games (CHESS + SOCCER+ POKER) EXPTIME-complete Undecidable.

Krishnendu Chatterjee58 Strategy Complexity

Krishnendu Chatterjee59 Classes of strategies rand. action-invisible pure rand. action-visible Classification according to the power of strategies

Krishnendu Chatterjee60 Classes of strategies rand. action-invisible pure rand. action-visible Classification according to the power of strategies Poly-time reduction from decision problem of rand. act.-vis. to rand. act.-inv.

Krishnendu Chatterjee61 Known results Almost-sure player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) [CDHR’06] memoryless [BGG’09] exponential (belief) [BGG’09] rand. act.-inv. exponential (belief) [CDHR’06(remark), GS’09] exponential (belief) [GS’09] pure ??? Reachability - Memory requirement (for player 1) Positive player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis.memoryless rand. act.-inv.memoryless pure ???

Krishnendu Chatterjee62 Beliefs Belief is sufficient. Randomized action invisible or visible almost same. The general case memory is similar (or in some cases exponential blow up) as compared to the one-sided case. Three prevalent beliefs:

Krishnendu Chatterjee63 Pure Strategies Belief is sufficient. Proofs Doubts. Belief

Krishnendu Chatterjee64 Pure Strategies Belief is sufficient. Proofs Doubts Lesson: Doubt your belief and believe in your doubts!!! See the unexpected. Belief

Krishnendu Chatterjee65 New results Almost-sure player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) [CDHR’06] memoryless [BGG’09] exponential (belief) [BGG’09] rand. act.-inv. exponential (belief) [CDHR’06(remark), GS’09] exponential (belief) [GS’09] pure??? Reachability - Memory requirement (for player 1) Positive player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis.memoryless rand. act.-inv.memoryless pure???

Krishnendu Chatterjee66 New results Almost-sure player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) [CDHR’06] memoryless [BGG’09] exponential (belief) [BGG’09] rand. act.-inv. exponential (more than belief) exponential (belief) [GS’09] pure exponential (more than belief) ?? Reachability - Memory requirement (for player 1) Positive player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis.memoryless rand. act.-inv.memoryless pure exponential (more than belief) ??

Krishnendu Chatterjee67 New results Almost-sure player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) [CDHR’06] memoryless [BGG’09] exponential (belief) [BGG’09] rand. act.-inv. exponential (more than belief) exponential (belief) [GS’09] pure exponential (more than belief) ?? Reachability - Memory requirement (for player 1) Positive player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis.memoryless rand. act.-inv.memoryless pure exponential (more than belief) ??

Krishnendu Chatterjee68 Pure Strategies: Player 1 Perfect, Player 2 Partial (positive) Pl1 Perfect, Pl 2 Partial : Non-stochastic, Pure. Memoryless Pl1 Perfect, Pl 2 Partial : Stochastic, Randomized. Memoryless Pl1 Perfect, Pl 2 Perfect: Stochastic, Pure. Memoryless Pl1 Partial, Pl 2 Perfect: Stochastic, Pure. Exponential

Krishnendu Chatterjee69 Pure Strategies: Player 1 Perfect, Player 2 Partial (positive) Pl1 Perfect, Pl 2 Partial : Non-stochastic, Pure. Memoryless Pl1 Perfect, Pl 2 Partial : Stochastic, Randomized. Memoryless Pl1 Perfect, Pl 2 Perfect: Stochastic, Pure. Memoryless Pl1 Partial, Pl 2 Perfect: Stochastic, Pure. Exponential Pl1 Perfect, Pl 2 Partial: Stochastic, Pure. Add probability Restrict to pure Pl 2 less informed Pl 1 more informed, Pl 2 less informed

Krishnendu Chatterjee70 Pure Strategies: Player 1 Perfect, Player 2 Partial (positive) Pl1 Perfect, Pl 2 Partial : Non-stochastic, Pure. Memoryless Pl1 Perfect, Pl 2 Partial : Stochastic, Randomized. Memoryless Pl1 Perfect, Pl 2 Perfect: Stochastic, Pure. Memoryless Pl1 Partial, Pl 2 Perfect: Stochastic, Pure. Exponential Pl1 Perfect, Pl 2 Partial: Stochastic, Pure. Non-elementary complete Add probability Restrict to pure Pl 2 less informed Pl 1 more informed, Pl 2 less informed

Krishnendu Chatterjee71 New results Almost-sure player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) [CDHR’06] memoryless [BGG’09] exponential (belief) [BGG’09] rand. act.-inv. exponential (more than belief) exponential (belief) [GS’09] pure exponential (more than belief) non-elementary complete ? Reachability - Memory requirement (for player 1) Positive player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis.memoryless rand. act.-inv.memoryless pure exponential (more than belief) non-elementary complete ?

Krishnendu Chatterjee72 New results Almost-sure player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) [CDHR’06] memoryless [BGG’09] exponential (belief) [BGG’09] rand. act.-inv. exponential (more than belief) exponential (belief) [GS’09] pure exponential (more than belief) non-elementary complete ? Reachability - Memory requirement (for player 1) Positive player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis.memoryless rand. act.-inv.memoryless pure exponential (more than belief) non-elementary complete ? Player 1 wins from more states, but needs more memory !

Krishnendu Chatterjee73 New results Almost-sure player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) [CDHR’06] memoryless [BGG’09] exponential (belief) [BGG’09] rand. act.-inv. exponential (more than belief) exponential (belief) [GS’09] pure exponential (more than belief) non-elementary complete finite (at least non- elementary) Reachability - Memory requirement (for player 1) Positive player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis.memoryless rand. act.-inv.memoryless pure exponential (more than belief) non-elementary complete finite (at least non- elementary)

Krishnendu Chatterjee74 Player 1 perfect, player 2 partial Win from more places. Winning strategy is very hard to implement. Information is useful, but ignorance is bliss !!! More information:

Krishnendu Chatterjee75 Reductions for equivalence Equivalence of the decision problems for almost-sure reach with pure strategies and rand. act.-inv. strategies Reduction of rand. act.-inv. to pure choice of a subset of actions (support of prob. dist.) Reduction of pure to rand. act.-inv. (holds for almost-sure only) It follows that the memory requirements for pure hold for rand. act.-inv. as well !

Krishnendu Chatterjee76 New results Almost-sure player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) [CDHR’06] memoryless [BGG’09] exponential (belief) [BGG’09] rand. act.-inv. exponential (more than belief) finite (at least non- elementary) pure exponential (more than belief) non-elementary complete finite (at least non- elementary) Reachability - Memory requirement (for player 1) Positive player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis.memoryless rand. act.-inv.memoryless pure exponential (more than belief) non-elementary complete finite (at least non- elementary)

Krishnendu Chatterjee77 Beliefs Belief is sufficient. Randomized action invisible or visible almost same. The general case memory is similar (or in some cases exponential blow up) as compared to the one-sided case. Three prevalent beliefs: Belief Fails! [CD11] Chatterjee, Doyen. Partial-Observation Stochastic Games: How to Win when Belief Fails. CoRR abs/ , July 2011.

Krishnendu Chatterjee78 The Message Play Chess; Play Soccer; But stay away from Poker !!!

Krishnendu Chatterjee79 Conclusion  Theory of graph games  Turn-based, concurrent, and partial-information games.  Different solution concepts and different complexity.  Several algorithmic questions open.  Partial information games  Problem with clear practical motivation.  Challenging to establish the right frontier of complexity.  Important generalization of perfect-information games.  Unfortunately, undecidable and also high complexity.  Current research: identifying decidable and more efficient sub-classes.

Krishnendu Chatterjee80 Collaborators  Luca de Alfaro  Laurent Doyen  Thomas A. Henzinger  Jean-Francois Raskin

Krishnendu Chatterjee81 Thank you ! Questions ? The end