32: Games 3 Functional Data Estimated Value and search subtrees Finite horizon minimax Functional data structures

How to make a move: setup You're at some state, s, in the game For each terminal state, t, there's a value (to P1), value(t) Assume that for any nonterminal state n, you can estimate the value of that state to P1, via "estimate_value(n)" In fact, extend "estimate_value" to apply to terminal states as well: estimate_value(t) = value(t)

Plan of action We'll look at a "subtree", starting at s and going forward, say, 4 moves. We'll assign a value to each leaf of this subtree, using estimate_value(n) We'll propagate these values up to get a really good value-guess for s.

How to write "estimate_value" for a game with points If you're playing a game where you accumulate points, you might say estimate_value(n) = P1's points minus P2's points, at state n. In short "pretend the game stopped right here…how much is P1 winning by?" negative for "losing"

How to write "estimate_value" for connect-4 If you're playing connect-4, you might say "Is it P1's turn, and … does P1 have a row of three with both ends playable? If so, +100; if not, does P1 have two intersecting rows of 2 each, so that playing at the intersection will give two rows of three? If so, +50. If not, does P1 have any rows of 2 that can be extended to 3? If so, +5 for each of these." …and if it's P2's turn, do a whole different set of computations… Writing "estimate_value" is the hardest part of GAME for some games (connect 4), and easy for others (Mancala: count my beads vs your beads!)

Where we stand For each game-state, we have a value (given by "estimate_value") We're at state s; we want to make a move Naïve: look at the value of the state resulting from each possible move; pick the one with the best value … for you! P1: look for large numbers; P2: look for small numbers Better: look ahead about 4 moves, and use minimax to find the best move!

Finite-horizon minimax Input: a state s in a game tree, and a depth d to search. Output: If s is non-terminal, and d > 0, a (value, move option) pair telling you the best move for the current player to make, and the value of making that move (in the sense of the "value to P1"); if s is terminal or d = 0, a (value, None) pair telling you the value of state s.

Finite Horizon Minimax algorithm Input: a state s in a game tree, and a depth d to search. Output: If s is non-terminal, and d > 0, a (value, move option) pair telling you the best move for the current player to make, and the value of making that move (in the sense of the "value to P1"); if s is terminal or d = 0, a (value, None) pair telling you the value of state s. Finite Horizon Minimax algorithm Algorithm (recursive!) if d is 0, return (estimated_value(s), None) if s is terminal, return (value(s), None) If whose_turn(s) is P1: among all moves, find the move m with the largest next-state value, v; return (v, Some m) If whose_turn(s) is P2: among all moves, find the move m with the smallest next-state value, v; return (v, Some m)  redundant if you defined estimated_value right!

AI Players

What's needed: scaffolding for an AI player estimate_value (s:state):float For a given state, say how good this state appears to be for player 1. easy to write for some games: in a game like "War", you might compute "player 1's card-count minus player 2's card count" Tough to write for others. In connect-4, you might look for three-blue- marbles-in-a-row, with both ends playable, and say that's great for player 1…but three reds in a row with both ends playable is horrible for player 1 Hardest part of Game for many folks Tradeoff: a good estimate_value procedure is helpful…but only if it's fast. If not, a weaker one that's fast will allow "deeper search" and get better results. This function is part of the GAME signature, so that it's available to the AI player.

What we have to write to make a computer-player "smart" next_move: select, at a given state, the move the computer-player "wants to make" Method: use depth-4 (perhaps) limited-horizon minimax to choose a move! Amazing fact: you don't really need to know what game you're playing! Different way to say that: you only need the GAME signature, not the Game itself !

General approach to write next_move(s) Given a state s, where it's your move Let (m, v) = minimax(s, d) Return m.

Questions If you have an "estimate_value" procedure, why not just look at every possible move from s and take the argmax or argmin of the values of the next states? Errors in estimate_value lead to bad moves Doing minimax on a deep tree tends to mask these errors Why not make "estimate_value" include minimax search of subsequent states? Because the "minimax" procedure's going to do that anyhow The point of estimate_value is to add something above and beyond that – actual game knowledge

type tree = Leaf | Node of float * tree * tree Game trees and trees You'll notice that nowhere in the Game code do we construct a "tree" type tree = Leaf | Node of float * tree * tree The "tree" structure is implicit in the signature module type GAME =sig ... val initial_state : state val legal_moves : state -> move list val next_state : state -> move -> state val game_status : state -> status end root of the tree edges from a node nodes at the ends of edges is this a leaf?

Traversing a "tree" without a tree use "game_status" to check whether you're at a leaf apply "next_move" to a list of all available moves to get "children" for a non-leaf

Functional lists

Do lists have to be data? A list is… The rules for lists are empty cons item lst The rules for lists are (first (cons a b)) => a (rest (cons a b)) => b

A new kind of list: functional lists Start with a function f : N -> … Let's make the codomain be… int options. I'm going to show you how to treat f as a list let first f = match (f 0) with | None -> failwith "Can't compute first of an empty list" | Some x -> x let emptyP f = ((f 0) = None) let consP f = not emptyP f let rest f = ... Need something that's a function!

Not quite right… What about empty lists? let rest f = fun x -> f (x+1) Not quite right… What about empty lists? let rest f = match f 0 with | None -> failwith "Empty lists have no rest" | _ -> fun x -> f (x + 1);;

What about "cons"? Need something that's a function, g Need g(0) = hd let cons hd tl = ... Need something that's a function, g Need g(0) = hd Need g(1, 2, 3, … ) = tl (0, 1, 2, … let cons hd tl = fun x -> match x with | 0 -> hd | n -> (tl (n-1))

Compute the length of a list let rec flist_length f = if (emptyP f) then else flist_length (rest f) Not quite as pretty as the "cond case" or "match case" version Those rely on data types for splitting things up: we've given that up

Problems let f(n) = Some 0;; flist_length f;; What we've really got here is something that includes possibly infinite lists Requires a rethinking of just about everything! Same deal for game-trees: a game like checkers has an infinite game tree! That's why we restricted to finite games! Generalization: "streams" – a topic for another course.