1. ICS-171:Notes 6: 1 Notes 6: Two-Player Games and Search ICS 171 Winter 2001

2. ICS-171:Notes 6: 2 Outline Computer programs which play 2-player games –game-playing as search –with the complication of an opponent General principles of game-playing and search –evaluation functions, minimax principle –alpha-beta-pruning, heuristic techniques Status of Game-Playing Systems –in chess, checkers, backgammon, Othello, etc, computers routinely defeat leading world players Applications? –think of “nature” as an opponent: economics, medicine, etc Reading: Nilsson Ch.II-12 ; R.&N. Ch.5 (5.1-5.3)

3. ICS-171:Notes 6: 3 Chess Rating Scale Garry Kasparov (current World Champion ) Deep Blue Deep Thought

4. ICS-171:Notes 6: 4 Game-Playing and AI Game-playing is a good problem for AI research: –all the information is available i.e., human and computer have equal information –game-playing is non-trivial need to display “human-like” intelligence some games (such as chess) are very complex requires decision-making within a time-limit –more realistic than other search problems –games are played in a controlled environment can do experiments, repeat games, etc: good for evaluating research systems –can compare humans and computers directly can evaluate percentage of wins/losses to quantify performance

5. ICS-171:Notes 6: 5 Search and Game Playing Consider a board game –e.g., chess, checkers, tic-tac-toe –configuration of the board = unique arrangement of “pieces” –each possible configuration = state in search space Statement of Game as a Search Problem –States = board configurations –Operators = legal moves –Initial State = current configuration –Terminal State (Goal) = winning configuration

6. ICS-171:Notes 6: 6 Game Tree Representation New aspect to search problem –there’s an opponent we cannot control –how can we handle this? S Computer Moves Opponent Moves Computer Moves G Possible Goal State lower in Tree (winning situation for computer)

7. ICS-171:Notes 6: 7 Complexity of Game Playing Imagine we could predict the opponent’s moves given each computer move How complex would search be in this case? –worst case, it will be O(b d ) –Chess: b ~ 35 (average branching factor) d ~ 100 (depth of game tree for typical game) b d ~ 35 100 ~10 154 nodes!! (“only” about 10 40 legal states) –Tic-Tac-Toe ~5 legal moves, total of 9 moves 5 9 = 1,953,125 9! = 362,880 (Computer goes first) 8! = 40,320 (Computer goes second) well-known games can produce enormous search trees

8. ICS-171:Notes 6: 8 Utility Functions Utility Function: –defined for each terminal state in a game –assigns a numeric value for each terminal state –these numbers represent how “valuable” the state is for the computer positive for winning negative for losing zero for a draw –Typical values from -infinity (lost) to +infinity (won) or [-1, +1].

9. ICS-171:Notes 6: 9 Greedy Search with Utilities A greedy search strategy using utility functions –Expand the search tree as far as the terminal states on each branch –Generate utility functions for each board configuration –Make the initial move that results in the board configuration with the maximum value –but this ignores what the opponent might do! i.e. opponent’s moves are interleaved with the computer’s

10. ICS-171:Notes 6: 10 Minimax Principle “Assume the worst” –say we are two plays (in the tree) away from the terminal states –high numbers favor the computer so we want to choose moves which maximize utility –low numbers favor the opponent so they will choose moves which minimize utility Minimax Principle –you (the computer) assume that the opponent will choose the minimizing move next (after your move) –so you now choose the best move under this assumption i.e., the maximum (highest-value) option considering both your move and the opponent’s optimal move. –we can generalize this argument more than 2 moves ahead: we can search ahead as far as we can afford.

11. ICS-171:Notes 6: 11 Minimax Search Example Explore the tree as far as the terminal states Calculate utilities for the resulting board configurations The computer will make the move such that when the opponent makes his best move, the board configuration will be in the best position for the computer

12. ICS-171:Notes 6: 12 Propagating Minimax Values up the Game Tree Starting from the leaves –Assign a value to the parent node as follows Children are Opponent’s moves: Minimum of all immediate children Children are Computer’s moves: Maximum of all immediate children

13. ICS-171:Notes 6: 13 Deeper Game Trees Minimax can be generalized to deeper game trees (than just 2 levels): “propagate” utility values upwards in the tree

14. ICS-171:Notes 6: 14 General Minimax Algorithm on a Game Tree For each move by the computer 1. perform depth-first search as far as the terminal state 2. assign utilities at each terminal state 3. propagate upwards the minimax choices if the parent is a minimizer (opponent) propagate up the minimum value of the children if the parent is a maximizer (computer) propagate up the maximum value of the children 4. choose the move (the child of the current node) corresponding to the maximum of the minimax values if the children Note: - minimax values are gradually propagated upwards as depth-first search proceeds, i.e., minimax values propagate up the tree in a “left-to-right” fashion - minimax values for sub-tree are propagated upwards “as we go”, so only O(bd) nodes need to be kept in memory at any time

15. ICS-171:Notes 6: 15 Complexity of Minimax Algorithm Assume that all terminal states are at depth d Space complexity –depth-first search, so O(bd) Time complexity –branching factor b, thus, O(b d ) The O(b d ) time complexity is a major problem since the computer typically only has a finite amount of time to make a move, –e.g., in chess, 2 minute time-limit, but b=35, d=50 ! So the direct minimax algorithm is impractical in practice –instead what about depth-limited search to depth m? –but utilities are defined only at terminal states –=> need to know the “utility” of non-terminal states these are called heuristic/static evaluation functions

16. ICS-171:Notes 6: 16 Idea of Static Evaluation Functions An Evaluation Function: –provide an estimate of how good the current board configuration is for the computer. –think of it as the expected utility averaged over all possible game outcomes from that position –it reflects the computer’s chances of winning from that node –but it must be easy to calculate, i.e., cannot involve searching the tree below the depth-limit node –thus, we use various easily calculated heuristics, e.g., Chess: Value of all white pieces - Value of all black pieces –Typically, one figures how good it is for the computer, and how good it is for the opponent, and subtracts the opponent’s score from the computer’s Must agree with the utility function when calculated at terminal nodes

17. ICS-171:Notes 6: 17 Minimax with Evaluation Functions The same as general minimax, except –only goes to depth m –uses evaluation functions (estimates of utility) How would this algorithm perform at chess? –could look ahead about 4 “ply” (pairs of moves) –would be consistently beaten even by average chess players! For each move by the computer 1. perform depth-first search with depth-limit m 2. assign evaluation functions at each state at depth m 3. propagate upwards the minimax choices if the parent is a minimizer (opponent) propagate up the minimum value of the children if the parent is a maximizer (computer) propagate up the maximum value of the children 4. choose the move (the child of the current node) corresponding to the maximum of the minimax values if the children

18. ICS-171:Notes 6: 18 The Alpha Beta Principle: Marry Search Tree Generation with Position Evaluations Computer's 1 0 -8 1 4120 XX 3 X Opponent’s Moves (min of evaluations) ( X indicates ‘not evaluated’)

19. ICS-171:Notes 6: 19 Alpha Beta Procedure Depth first search of game tree, keeping track of: –Alpha: Highest value seen so far on maximizing level –Beta: Lowest value seen so far on minimizing level Pruning –When Maximizing, do not expand any more sibling nodes once a node has been seen whose evaluation is smaller than Alpha –When Minimizing, do not expand any sibling nodes once a node has been seen whose evaluation is greater than Beta

20. ICS-171:Notes 6: 20 2 2 -2 1 0 -3 8 The Alpha Beta Principle: 2nd Example

21. ICS-171:Notes 6: 21 Effectiveness of Alpha-Beta Search Worst-Case –branches are ordered so that no pruning takes place –alpha-beta gives no improvement over exhaustive search Best-Case –each player’s best move is the left-most alternative (i.e., evaluated first) –in practice, performance is closer to best rather than worst-case In practice often get O(b (d/2) ) rather than O(b d ) –this is the same as having a branching factor of sqrt(b), since (sqrt(b)) d = b (d/2) i.e., we have effectively gone from b to square root of b –e.g., in chess go from b ~ 35 to b ~ 6 this permits much deeper search in the same amount of time makes computer chess competitive with humans!

22. ICS-171:Notes 6: 22 Iterative (Progressive) Deepening In real games, there is usually a time limit T on making a move How do we take this into account? –using alpha-beta we cannot use “partial” results with any confidence unless the full breadth of the tree has been searched – So, we could be conservative and set a conservative depth-limit which guarantees that we will find a move in time < T disadvantage is that we may finish early, could do more search In practice, iterative deepening search (IDS) is used –IDS runs alpha-beta search with an increasing depth-limit note: the “inner” loop is a full alpha-beta search with a specified depth limit m –when the clock runs out we use the solution found at the previous depth limit

23. ICS-171:Notes 6: 23 Heuristics and Game Tree Search The Horizon Effect –sometimes there’s a major “effect” (such as a piece being captured) which is just “below” the depth to which the tree has been expanded –the computer cannot see that this major event could happen –it has a “limited horizon” –there are heuristics to try to follow certain branches more deeply to detect to such important events, i.e., “focusing attention” –this helps to avoid catastrophic losses due to “short-sightedness” Heuristics for Tree Exploration –it may be better to explore some branches more deeply in the allotted time –various heuristics exist to identify “promising” branches

24. ICS-171:Notes 6: 24 More on Evaluation Functions Features are numeric characteristics of a board position, e.g., –feature 1 = f1 = number of white pieces –feature 2 = f2 = number of black pieces –feature 3 = f3 = f1/f2 –feature 4 = f4 = number of white bishops –feature 5 = f5 = estimate of “threat” to white king Evaluation function is an estimate of a useful a board position is –it is a function of the pieces on the board –it is like the heuristic function in general search –generally evaluation function = function(f1, f2, f3,......) i.e., it is a function of individual features

25. ICS-171:Notes 6: 25 Linear Evaluation Functions A linear function of f1, f2, f3 is a weighted sum of f1, f2, f3.... A linear evaluation function is = w1.f1 + w2.f2 + w3.f3 +..... wn.fn –where f1, f2, f3, are the features –and w1, w2, w3, are the weights –Idea: more important features get more weight The quality of play will depend directly on the quality of the evaluation function –to build an evaluation function we have to 1. construct good features (using expert knowledge, heuristics) 2. pick good weights (can be learned)

26. ICS-171:Notes 6: 26 Learning the Weights in a Linear Evaluation Function A linear evaluation function is = w1.f1 + w2.f2 + w3.f3 +..... wn.fn How could we “learn” these weights? –basic idea: play lots of games against an opponent –for every move (or game) look at the error = true outcome - evaluation function –if error is positive (underestimating) adjust weights to increase the evaluation function –if error is zero do nothing –if error is negative (overestimatng) adjust weights to decrease the evaluation function –details on learning of weights will be taught later in quarter when we discuss learning algorithms

27. ICS-171:Notes 6: 27 Examples of Algorithms which Learn to Play Well Checkers –A. L. Samuel, “Some studies in machine learning using the game of checkers,” IBM Journal of Research and Development, 11(6):601- 617, 1959. –Learned by playing a copy of itself thousands of times –using only an IBM 704 with 10,000 words of RAM, magnetic tape, and a clock speed of 1 kHz –successful enough to compete well at human tournaments –a computer performing full flawless search is expected soon Backgammon –G. Tesauro and T. J. Sejnowski, “A parallel network that learns to play backgammon,” Artificial Intelligence, 39(3), 357-390, 1989. –Also learns by playing copies of itself –uses a non-linear evaluation function (a neural network) –rates in the top 3 players in the world.

28. ICS-171:Notes 6: 28 Computers can play GrandMaster Chess “Deep Blue” (IBM) –parallel processor, 32 nodes –each node has 8 dedicated VLSI “chess chips” –each chip can search 200 million configurations/second –uses minimax, alpha-beta, heuristics: can search to depth 14 –memorizes starts, end-games –power based on speed and memory: no common sense Kasparov v. Deep Blue, May 1997 –6 game full-regulation chess match (sponsored by ACM) –Kasparov lost the match (2.5 to 3.5) –a historic achievement for computer chess: the first time a computer is the best chess-player on the planet Note that Deep Blue plays by “brute-force”: there is relatively little which is similar to human intuition and cleverness

29. ICS-171:Notes 6: 29 Status of Computers in Other Games Checkers/Draughts –current world champion is Chinook, can beat any human –uses alpha-beta search Othello –computers can easily beat the world experts Backgammon –system which learns is ranked in the top 3 in the world –uses neural networks to learn from playing many many games against itself Go –branching factor b ~ 360: very large! –$2 million prize for any system which can beat a world expert

30. ICS-171:Notes 6: 30 Web site of the Day http://www.chess.ibm.com Describes Kasparov v. Deep Blue match in May 1997 –descriptions of the games –quotes, background info on people involved –expert commentary and analysis –detailed information about Deep Blue –essays about computer chess in general

31. ICS-171:Notes 6: 31 Summary Game playing is best modeled as a search problem Game trees represent alternate computer/opponent moves Evaluation functions estimate the quality of a given board configuration for each player Minimax is a procedure which chooses moves by assuming that the opponent will always choose the move which is best for them Alpha-Beta is a procedure which can prune large parts of the search tree and allow search to go deeper For many well-known games, computer algorithms based on heuristic search match or out-perform human world experts. Reading: Nilsson Ch.II-12 ; R.&N. Ch.5 (5.1-5.3) Do alpha-beta pruning exercise on page 206 Nilsson