1. 1 GP End-Chess Evolution of Chess Endgame Players Ami Hauptman & Moshe Sipper

2. 2 Outline  Introduction  The Game of Chess – a solved problem?  Important differences between human and artificial chess players  Chess Endgames - features & building blocks  GP problem definition  Experiment and Results  Future work

3. 3 The game of Chess  First developed in India and Persia  Considered an extremely complex game of strategy and inventiveness  Enormous search space  Roughly 50 possible moves at mid-game  A typical game consists of a few dozen moves  Estimated at 10 43 in 40-move game (Shannon, 1950)

4. 4 The game of Chess – AI history  First chess AI at 1958 – novice level  Machine strength increasing linearly  1997 – defeat of former world champion, Garry Kasparov, by IBM’s deep blue  Last years – performance still increasing  Mainly Hardware  Also Software  … The End ?

5. 5 The game of Chess  …NO!  Deep(er) blue uses extreme brute-force, traversing several millions of boards ps  Very little generalization  Virtually no human resemblance  Deemed theoretically uninteresting  Chomsky: As interesting as a weight lifting competition between machine and man  Low A to I ratio; Low return

6. 6 The game of Chess – Basic concepts  8x8 board  Each player starts with 16 pieces, of 6 different types, and may only move 1 piece per turn  A piece can only move into an empty square or into one containing an opponent’s piece (a capture)  Win by capturing the opponent’s king

7. 7 The game of Chess – pieces  Pawn: may only move forward (or capture diagonally)  Bishop: diagonals  Knight: L shaped moves. The only “unblockable” piece  Rook: Ranks & files  Queen: Bishop & Rook combined  King: 1 square in any direction. May not move into attacked square Values : 1 3 3/3.5 5 9 ∞

8. 8 The game of Chess – example  White has over 30 possible moves  If black ’ s turn – can capture pawn at c3 and check (also fork)

9. 9 The game of Chess – Check and Checkmate  “Checking” is attacking opponent’s king. Opponent must respond  “Mating” (Checkmate) is when the opponent can’t avoid losing the king – and thus forfeiting the game

10. 10 Human & Artificial Players - AI  AI uses search to assign a score to a board  Traverse the move tree from leaves - up  Select the best child using scores found  Only partial tree Computer is O (Max) opponent is X

11. 11 Human & Artificial Players – AI Search  Millions of boards (nodes) per second  Little time for each board – less knowledge  Smart search algorithms –  pruning  Alpha-beta variants (negascout etc.)  Still use heuristics at end – can’t see all tree  Most research revolves around search  Human resemblance minimal – humans use little search

12. 12 Human & Artificial Players – Humans(1)  Humans use problem solving cognition  Deeply knowledge based – extensive “Chess theory”  Massive use of pattern recognition  Also use search but  Less deep  Only develop “good” positions  More efficient – less nodes for “same” result  Reminiscent of greedy search  Not only in chess

13. 13 Human & Artificial Players – Grand Masters - Findings  Play against several opponents at the same level they play against a single opponent  GMs and novices: same level of performance when memorizing a random board; differ when memorizing real game positions  GM eye movements show they only scan “correct” parts of board  Strong Amateurs use the same meta-search as GMs - equally deep, same nodes, same speed; Differ in knowledge of domain (De Groot)

14. 14 Endgames - example  White’s turn: mate in 5, with Qe6+  Features include:  #moves for black king minimal  Attacking, un-attacked  Checking  Officers same line\row  Black’s turn: draw with: Rc1+, then Qg5 – fork & exchange

15. 15 Endgames (2) - features  Few pieces remain (typically: king, 0-3 officers and sometimes pawns)  Fewer options, but more moves for each piece  Trees still extremely large

16. 16 Endgames - Building Blocks  Main goals  Reduce search by “smart” features of the board  Use more game knowledge as humans do  Allow more complex features to be built by supplying basic ones (terminals) and building methods (functions)  Schemata evolution

17. 17 Features Example - Fork  My piece is:  Attacking 2 or more pieces  Protected or not attacked  Opponent pieces:  Unprotected  OR protected but of greater value  Example: black must exchange Q for R because of fork

18. 18 Fork: Traditional AI search  Only 3 legal moves for black  Find that one of white’s next moves (out of 23 possible) captures black queen  Check all following moves for more piece exchanges  Sometimes, still check other moves (non capturing)  At end of search – compare remaining pieces  No fork “concept”

19. 19 Features Example – Fork Feature Search (GP)  One of the features is isMyFork function – Checks all previously defined conditions  Also, use some smaller building blocks:  Is Opponent piece Attacked?  Is attacking piece protected?  Is opponent in check?  Value of attacked piece

20. 20 GP Problem Definition  Terminals & Functions  Fitness  Run parameters  Termination  We will see each element in the following experiments

21. 21 Endgame experiments conducted  KRKR – each player has 1 king and 1 rook  KRRKR – King with 2 Rook against King and Rook  KRRKRR  KQKQ – Kings and Queens  KQRKQR – Combined

22. 22 Basic program architecture  Generate all possible moves (depth=1)  Evaluate each board with GP individual  Select board with best score (or stochastically decide between equal)  Perform best move  Repeat process with GP opponent until game ends (or until only kings left)

23. 23 KRKR Endgame  Each player has 1 King, 1 Rook  “Toy” problem for chess endgames  Theoretical draw (experts never lose this)  Some interesting positions exist

24. 24 KRKR Endgame - what needs to be learned (1)  Avoid losing rook  Don’t move to attacked, unprotected squares  Vice versa - capture opponent’s rook if able Black to move – white loses Rook

25. 25 KRKR Endgame - what need to be learned (2)  Avoid getting king stuck in edges  Again, take advantage if opponent does this Black to move – mate in 1

26. 26 KRKR Endgames - Terminals  Used in first runs:  Is My Rook Attacked, Is Opp Rook attacked  Is {My, Opp} Rook Protected (two as above)  Is {My, Opp} Rook In Play  Num Moves {My, Opp} king  {My, Opp}-King’s proximity to edges  Is Mate  ERCs: ± {0.25, 0.5, 1} * MAX  MAX = 1000 (empirically)

27. 27 KRKR Endgames - Functions  Boolean  OR2, OR3, OR4  AND2, AND3, AND4  NOT  Arithmetic - +, -, *  Combined -, IF  STGP  For now, no “chess” functions, only terminals

28. 28 KRKR Endgames - Fitness  Competitive, Random-2-ways  Each individual plays against k randomly selected opponents  Each game counts for both players  For each encounter  Several games (typically 4) are played  Short games - ~5-8 moves per player  Each game starts at a random legal position  Safe start - no piece is attacked at the beginning

29. 29 KRKR Endgames – Fitness (2)  Scoring method:  Victory: 1-2 points  Piece count advantage (theoretical win) – ¾ point  Draw: ½ point  After advantage – 0 points  Loss: 0 points

30. 30 KRKR Endgames – Parameters  Population size - 80  #Generations - 150..250  Operators:  Reproduction 0.35  Crossover 0.5  Mutation 0.15 (including ERC mutation)  Termination – ~10-20 hours

31. 31 KRKR Endgames – Results  Every 10 generations, best individual played against:  Best of generation 0  An opponent performing random moves  Longer games: ~10-12 moves per player  50-150 games  Games were doubled – each player staring from both positions

32. 32 KRKR Endgames – Results  Bad results – no distinct improvement  Several reasons:  Arithmetic operations problematic – we get large numbers  Mate not distinct enough (traditionally terminates the search)  Boolean functions not clear enough  Slow Runs due to large trees with repeating functions

33. 33 KRKR Endgames –Improvements  Boolean functions  Divided to good and bad  Example: Is-My-King-In-Check changed to Is-My- King-Not-In-Check  Mate changed to 1000*Mate  Added Not-My-Rook-Attacked-Unprotected and Opp-Rook-Attacked-Unprotected

34. 34 KRKR Endgames – Results - Improvements  Also consulted Chess Experts – added more:  Is-Opp-King-Behind- Rook  Split to  Opp-King-Prox-Rook  Opp-King-Behind-Rook  Is-Stalemate (only kings left) Black moves and White loses Rook

35. 35 KRKR Endgames – Results - Improvements  Arithmetic functions canceled  Although Still using floats for terminals  Also divide to good and bad: NumNotMovesOppKing  Theoretically justified – more “logical” search in literature  Empirically - need more logical rules, and not : ( > (+ (#moves-k #moves-opp-k) 5.5))  Memoization – saves more than ½ the time

36. 36 KRKR Endgames – Final Results  Improvement  Above 75% of games against random end in advantage or mate  Still, too few mates, even when score for win is increased – difficult to learn move sequence  Same against best of generation 0 (almost random)  The main thing that was learned was avoiding getting the rook captured

37. 37 KRRKR Endgames  Example (right)  Very good for white  Black king exposed  2 rooks close  Next move – captures rook  (mate in 5)

38. 38 KRRKR Endgames - goals  One player has 2 rooks, the other – 1  Not theoretically drawn  We want one generalized individual for all endgames and positions (Not one for each endgame):  Each player needs to play both advantage, draw (KRKR) and disadvantage  Terminals need to be more general

39. 39 KRRKR Endgames - changes  Terminals - changed and added to cope with changing state  Material-Count (recall each rook = 5)  Num-My-Pieces-Not-Attacked, since now there are more than 1  Is-My-King-Protecting-Piece and My-Officers- Same-Line to allow more complex considerations  Functions  If-Adv-Then-(left child)-Else-(right child)  Eventually divided to 3 trees

40. 40 KRRKR Endgames - changes  Also added – comparing differences to parent node  Boolean Is-Material-Increase, which compares to the parent node (board)  Material decrease is not needed since considering only my move  Not-My-King-Moves-Decrease to further use number of moves for king

41. 41 KRRKR Endgames – Opponents  Random forsaken; Best-of-0 still used but less  Added new opponent – MASTER  a program we wrote based on consultation with experts, highest being International Master Boris Gutkin, ELO 2400 (only about 3000 of those…)  Used ~50 general positions and rules derived from them, together with scores for each  Defined a strategy (“Expert”) accordingly  Tested evolved programs against it  Human competitive?

42. 42 KRRKR Endgames – Fitness  Test were conducted by assigning each player both roles for each position  Fitness was refined – score effected by:  Starting position (advantage or disadvantage)  End result – win, loss or draw  Adv position ending in draw receives a score of near zero  Dis-adv ending in a draw will receive better than 0.5

43. 43 KRRKR Endgames – Results  Expert-defined performed extremely well against Random and Best0  Evolved programs performed generally as well as expert defined, sometimes better Percent of favorable results in game outcomes

44. 44 Main Experiment – KQRKQR  Most complex endgame we worked with  Still theoretical draw  Highly position dependant – “noisy”  Larger trees  2 officers  Queens  Easier to mate

45. 45 KQRKQR Endgames - changes  Added – more “heavy” terminals (and components)  Boolean Is-Not-Mate-in-one, most time consuming but necessary  Boolean Is-My-King-Not-Trapped  Not all king’s moves lead closer to edges  Important but vague – usually happens with complex terminals  My-Officers-Same-Line

46. 46 Function SetComplex TerminalsSimple Terminals If3(B,F,F)EvaluateMaterialNotMyKingInCheck Or2(B,B)IsMaterialIncreaseIsOppKingInCheck Or3(B,B,B)IsMateMyKingDistEdges And2(B,B)IsMateInOneOppKingProximityToEdges And3(B,B,B)OppPieceCanBeCapturedNumMyPiecesNotAttacked Smaller(B,B)MyPieceCannotBeCapturedNumOppPiecesAttacked Not(B)IsOppKingStuckValueMyPiecesAttacking IsMyKingNotStuckValueOppPiecesAttacking IsOppKingBehindPieceIsMyQueenNotAttacked IsMyKingNotBehindPieceIsOppQueenAttacked IsOppPiecePinnedIsMyFork IsMyPieceNotPinnedIsOppNotFork NumMovesMyKing NumNotMovesOppKing MyKingProxRook OppKingDistRook MyPiecesSameLine OppPiecesNotSameLine IsOppKingProtectingPiece IsMyKingProtectingPiece Genome Summary

47. 47 KQRKQR Endgames – New Opponent  CRAFTY, second in the 2004 Computer Chess Championship (held at Bar-Ilan)  Uses brute force methods; State-of-the-art search algorithms  Specializes in Blitz games (typically 3 minutes per game)  We limited to 5 secs per move, enough to scan ~1.5 Million boards with pruning

48. 48 KQRKQR Endgames – Our parameters  Used lookahead of 2  Typically ~5 secs per move  Simple Minimax search, but not Alpha-Beta  Played 5-6 moves per game  Never cancelled a game, even if it started with mate-in-4 (which CRAFTY easily saw)  Played each position 2 times, switching places  ~100 games - reduce noises in starting positions

49. 49 Results: Master

50. 50 Results: CRAFTY

51. 51 Multiple Endgames  Aim for general-purpose strategies  All endgames used during evolution  Results: %Draws%Advs%Wins 6826 Master 7242CRAFTY

52. 52 Sample GP-Endchess Tree 0: (If3 (Or2 (Not (Or2 (And2 OppPieceAttUnprotected NotMyKingInCheck) (Or2 NotMyPieceAttUnprotected 100*Increase))) (And2 (Or3 (And2 OppKingStuck NotMyPieceAttUnprotected) (And2 OppPieceAttUnprotected OppKingStuck) (And3 - 1000*MateInOne OppKingInCheckPieceBehind NotMyKingStuck)) (Or2 (Not NotMyKingStuck) OppKingInCheck))) NumMyPiecesUNATT (If3 (< (If3 (Or2 NotMyPieceAttUnprotected NotMyKingInCheck) (If3 NotMyPieceAttUnprotected #NotMovesOppKing OppKingInCheckPieceBehind) (If3 OppKingStuck OppKingInCheckPieceBehind -1000*MateInOne)) (If3 (And2 100*Increase 1000*Mate?) (If3 (< NumMyPiecesUNATT (If3 NotMyPieceAttUnprotected -1000*MateInOne OppKingProxEdges)) (If3 (< MyKingDistEdges #NotMovesOppKing) (If3 -1000*MateInOne -1000*MateInOne NotMyPieceATT) (If3 100*Increase #MovesMyKing OppKingInCheckPieceBehind)) NumOppPiecesATT) (If3 NotMyKingStuck -100.0 OppKingProxEdges))) (If3 OppKingInCheck (If3 (Or2 NotMyPieceAttUnprotected NotMyKingInCheck) (If3 (< MyKingDistEdges #NotMovesOppKing) (If3 -1000*MateInOne -1000*MateInOne NotMyPieceATT) (If3 100*Increase #MovesMyKing OppKingInCheckPieceBehind)) NumOppPiecesATT) (If3 (And3 - 1000*MateInOne NotMyPieceAttUnprotected 100*Increase) (If3 (< NumMyPiecesUNATT (If3 NotMyPieceAttUnprotected -1000*MateInOne OppKingProxEdges)) (If3 (< MyKingDistEdges #NotMovesOppKing) (If3 -1000*MateInOne - 1000*MateInOne NotMyPieceATT) (If3 100*Increase #MovesMyKing OppKingInCheckPieceBehind)) NumOppPiecesATT) -1000*MateInOne)) (If3 (< (If3 100*Increase MyKingDistEdges 100*Increase) (If3 OppKingStuck OppKingInCheckPieceBehind -1000*MateInOne)) -100.0 (If3 (And2 NotMyPieceAttUnprotected -1000*MateInOne) (If3 (< NumMyPiecesUNATT (If3 NotMyPieceAttUnprotected -1000*MateInOne OppKingProxEdges)) (If3 (< MyKingDistEdges #NotMovesOppKing) (If3 -1000*MateInOne -1000*MateInOne NotMyPieceATT) (If3 100*Increase #MovesMyKing OppKingInCheckPieceBehind)) NumOppPiecesATT) (If3 OppPieceAttUnprotected NumMyPiecesUNATT MyFork))))) Tree 1: (If3 NotMyPieceAttUnprotected #NotMovesOppKing 1000*Mate?) Tree 2: (If3 1000*Mate? NumMyPiecesUNATT -1000*MateInOne)

53. 53 Summary  Draw and better against Master-defined  Draw against a world class opponent  On limited conditions (not a full game, time,etc.)  Shows deep search may have an alternative  Fast, pattern-oriented search suggests more human resemblance  Search and lookahead are still important

54. 54 Future Work  Add more pieces  Improve evolution speed  Parallel nets  Stronger board representations  Develop more cognitive models using evolution  Search scheme space as well as game space  Tackle beyond endgames  Openings and mid-game  General game concept schemas (?)