2. 32: Games 3 Functional Data
Estimated Value and search subtrees
Finite horizon minimax
Functional data structures

3. How to make a move: setup
You're at some state, s, in the game
For each terminal state, t, there's a value (to P1), value(t)
Assume that for any nonterminal state n, you can estimate the value of that state to P1, via "estimate_value(n)"
In fact, extend "estimate_value" to apply to terminal states as well: estimate_value(t) = value(t)

4. Plan of action
We'll look at a "subtree", starting at s and going forward, say, 4 moves.
We'll assign a value to each leaf of this subtree, using estimate_value(n)
We'll propagate these values up to get a really good value-guess for s.

5. How to write "estimate_value" for a game with points
If you're playing a game where you accumulate points, you might say estimate_value(n) = P1's points minus P2's points, at state n.
In short "pretend the game stopped right here…how much is P1 winning by?"
negative for "losing"

6. How to write "estimate_value" for connect-4
If you're playing connect-4, you might say "Is it P1's turn, and …
does P1 have a row of three with both ends playable? If so, +100;
if not, does P1 have two intersecting rows of 2 each, so that playing at the intersection will give two rows of three? If so, +50.
If not, does P1 have any rows of 2 that can be extended to 3? If so, +5 for each of these."
…and if it's P2's turn, do a whole different set of computations…
Writing "estimate_value" is the hardest part of GAME for some games (connect 4), and easy for others (Mancala: count my beads vs your beads!)

7. Where we stand
For each game-state, we have a value (given by "estimate_value")
We're at state s; we want to make a move
Naïve: look at the value of the state resulting from each possible move; pick the one with the best value … for you!
P1: look for large numbers;
P2: look for small numbers
Better: look ahead about 4 moves, and use minimax to find the best move!

8. Finite-horizon minimax
Input: a state s in a game tree, and a depth d to search.
Output: If s is non-terminal, and d > 0, a (value, move option) pair telling you the best move for the current player to make, and the value of making that move (in the sense of the "value to P1"); if s is terminal or d = 0, a (value, None) pair telling you the value of state s.

9. Finite Horizon Minimax algorithm
Input: a state s in a game tree, and a depth d to search.
Output: If s is non-terminal, and d > 0, a (value, move option) pair telling you the best move for the current player to make, and the value of making that move (in the sense of the "value to P1"); if s is terminal or d = 0, a (value, None) pair telling you the value of state s.
Finite Horizon Minimax algorithm
Algorithm (recursive!)
if d is 0, return (estimated_value(s), None)
if s is terminal, return (value(s), None)
If whose_turn(s) is P1: among all moves, find the move m with the largest next-state value, v; return (v, Some m)
If whose_turn(s) is P2: among all moves, find the move m with the smallest next-state value, v; return (v, Some m)
 redundant if you defined estimated_value right!

10. AI Players

11. What's needed: scaffolding for an AI player
estimate_value (s:state):float
For a given state, say how good this state appears to be for player 1.
easy to write for some games: in a game like "War", you might compute "player 1's card-count minus player 2's card count"
Tough to write for others. In connect-4, you might look for three-blue- marbles-in-a-row, with both ends playable, and say that's great for player 1…but three reds in a row with both ends playable is horrible for player 1
Hardest part of Game for many folks
Tradeoff: a good estimate_value procedure is helpful…but only if it's fast.
If not, a weaker one that's fast will allow "deeper search" and get better results.
This function is part of the GAME signature, so that it's available to the AI player.

12. What we have to write to make a computer-player "smart"
next_move: select, at a given state, the move the computer-player "wants to make"
Method: use depth-4 (perhaps) limited-horizon minimax to choose a move!
Amazing fact: you don't really need to know what game you're playing!
Different way to say that: you only need the GAME signature, not the Game itself !

13. General approach to write next_move(s)
Given a state s, where it's your move
Let (m, v) = minimax(s, d)
Return m.

14. Questions
If you have an "estimate_value" procedure, why not just look at every possible move from s and take the argmax or argmin of the values of the next states?
Errors in estimate_value lead to bad moves
Doing minimax on a deep tree tends to mask these errors
Why not make "estimate_value" include minimax search of subsequent states?
Because the "minimax" procedure's going to do that anyhow
The point of estimate_value is to add something above and beyond that – actual game knowledge

15. type tree = Leaf | Node of float * tree * tree
Game trees and trees
You'll notice that nowhere in the Game code do we construct a "tree"
type tree = Leaf | Node of float * tree * tree
The "tree" structure is implicit in the signature
module type GAME =sig
...
val initial_state : state
val legal_moves : state -> move list
val next_state : state -> move -> state
val game_status : state -> status
end
root of the tree
edges from a node
nodes at the ends of edges
is this a leaf?

16. Traversing a "tree" without a tree
use "game_status" to check whether you're at a leaf
apply "next_move" to a list of all available moves to get "children" for a non-leaf

17. Functional lists

18. Do lists have to be data? A list is… The rules for lists are empty
cons item lst
The rules for lists are
(first (cons a b)) => a
(rest (cons a b)) => b

19. A new kind of list: functional lists
Start with a function f : N -> …
Let's make the codomain be… int options.
I'm going to show you how to treat f as a list
let first f = match (f 0) with
| None -> failwith "Can't compute first of an empty list"
| Some x -> x
let emptyP f = ((f 0) = None)
let consP f = not emptyP f
let rest f = ...
Need something that's a function!

20. Not quite right… What about empty lists?
let rest f = fun x -> f (x+1)
Not quite right…
What about empty lists?
let rest f = match f 0 with
| None -> failwith "Empty lists have no rest"
| _ -> fun x -> f (x + 1);;

21. What about "cons"? Need something that's a function, g Need g(0) = hd
let cons hd tl = ...
Need something that's a function, g
Need g(0) = hd
Need g(1, 2, 3, … ) = tl (0, 1, 2, …
let cons hd tl = fun x -> match x with
| 0 -> hd
| n -> (tl (n-1))

22. Compute the length of a list
let rec flist_length f =
if (emptyP f) then
else
flist_length (rest f)
Not quite as pretty as the "cond case" or "match case" version
Those rely on data types for splitting things up: we've given that up

23. Problems
let f(n) = Some 0;;
flist_length f;;
What we've really got here is something that includes possibly infinite lists
Requires a rethinking of just about everything!
Same deal for game-trees: a game like checkers has an infinite game tree!
That's why we restricted to finite games!
Generalization: "streams" – a topic for another course.