1. Research Related to Real-Time Strategy Games Robert Holte November 8, 2002

2. No Games Knowledge, but… Single-agent search = sequential decisions with additive costs –path-finding occurs in RTS games –are there other occurrences ? –transfer from 1-agent to 2-agent or N-agent games ? –is a 2-agent game with a deterministic opponent a single-agent search problem ? Path-finding research group (P. Yap, Yngvi, …)

3. and … Combinatorial optimization = resource allocation with complex constraints –combinatorial auctions –“boolean programming” –hill-climbing, ESG, a bit of CPLEX

4. and … Machine Learning –opponent-modelling (poker) –user-modelling (adaptive games, personalization) –learning to make good moves (M. Mueller) Interactive Systems –mostly-automatic evaluation methodology Abstraction in Poker

5. Using Abstraction to Speed Up Search

6. Search = path-finding in a graph GIVEN: –A graph –Two nodes in the graph (“start”, “goal”) FIND: – a shortest path from start to goal

7. Labyrinth at the gardens of Versailles (could be a computer game!)

8. “Implicit” Graphs Instead of an adjacency list/matrix, Define a graph by a successor function, succ, and one or more “seed” nodes. Nodes of the graph = transitive closure of succ applied to the seed nodes An edge exists from n 1 to n 2 iff n 2  succ(n 1 ) A graph can be exponentially larger than its description.

9. Towers of Hanoi puzzle for D disks: –graph has 3 D nodes –description is O(D 2 )

10. 3-disk State Space

11. Speed up Search by Using Extra Information Solution “skeleton” = chain of subgraphs leading from start to goal –Refinement: search in the first subgraph until you reach the second, then search in the second until you reach the third, etc. –very fast but path found not guaranteed to be optimal Heuristic = h(n) = estimate of the distance from node n to the goal node. –if it never overestimates, it can eliminate (prune) parts of the graph and still guarantee finding an optimal path

12. My Research Aim: to generate heuristics or skeletal solutions automatically Method: 1.Create an abstraction of the given graph (simplified version of your problem that “approximately” preserves “important” features of the problem). 2.Find a solution path in the abstract graph. 3.Use the abstract solution as a skeletal solution, or use its length as a heuristic.

13. Example: Towers of Hanoi start goal

14. Abstraction: “Ignore the smallest disk” start goal

15. Abstract Solution start goal

16. Key Techniques For Automatically Generating Abstractions –explicit graphs: STAR abstraction –implicit graphs: domain abstraction –select among abstractions using Korf & Reid’s method for predicting search time Search Methods –refinement: AltO –heuristic: Hierarchical A*, pattern databases –anytime combination of refinement and heuristic search

17. STAR abstraction Abstract state = connected subgraph (all nodes within distance R of a given node) Fully automatic (but need to amortize cost) Fine control over granularity (R) Can be applied recursively (abstraction hierarchy) Guarantees refinement will succeed Produces a consistent heuristic

18. AltO Refinement Technique Conduct abstract search in the opposite direction to the base-level search; Use the whole abstract search graph for refinement. 12% faster than standard refinement and produces better solutions (20% shorter) Robust – not sensitive to the abstract solution found, and not very sensitive to the radius of abstraction 10-40 times faster than A* (same abstraction) and solutions usually within 30% of optimal

19. Pattern Database Successes Joe Culberson & Jonathan Schaeffer (1994). –15-puzzle –PDB + Manhattan Distance (MD) reduced MD- alone search tree 1000-fold Rich Korf (1997) –PDBs enabled optimal solutions of Rubik’s Cube Stefan Edelkamp (2001) –PDBs competitive with best planners Istvan Hernadvolgyi –PDBs reduced macro-table solution lengths from 90 to 50

20. Example: 8-puzzle 12 345 678 Domain = blank 1 2 3 4 5 6 7 8 181,440 states

21. “Patterns” created by domain mapping 12 345 678 This mapping produces 9 patterns Domain = blank 1 2 3 4 5 6 7 8 Abstract = blank corresponding patternoriginal state

22. Pattern Database Pattern Distance to goal 0 1 1 2 2 2 Pattern Distance to goal 3 3 4

23. Calculating h(s) Given a state in the original problem Compute the corresponding pattern and look up the abstract distance-to-goal 814 35 672 2 Heuristics defined by PDBs are consistent, not just admissible.

24. Domain Abstraction 12 345 678 678 30,240 patterns Domain = blank 1 2 3 4 5 6 7 8 Abstract = blank 6 7 8

25. Automatic Creation of Domain Abstractions Easy to enumerate all possible domain abstractions They form a lattice, e.g. is “more abstract” than the domain abstraction above Domain = blank 1 2 3 4 5 6 7 8 Abstract = blank Domain = blank 1 2 3 4 5 6 7 8 Abstract = blank

26. 8-puzzle: A* vs. PDB size # nodes expanded (A*) pattern database size (# of abstract states)

27. Hierarchical A* Unlike pattern databases, each h(n) is computed on demand. If h(n) is not already known compute it by searching at the abstract level. The search at an abstract level can be informed by a heuristic defined by a higher abstract level. * many abstract searches for one base-level search * naïve implementation is 10 times slower than blind base-level search.

28. Speed versus Optimality Refinement –very fast (when refinability guaranteed) –suboptimal paths A* –much slower –optimal paths See also: bidirectional search,  -admissible A*

29. Hybrid of HA* and Refinement (1) Although the algorithms seem very different there are actually only two differences. (1)The relative weight given to g and h. HA*: g + h Refinement: h (use g to break ties) Generalization:  *g + (1-  )*h ( 0    1 )

30. (2) What to do if a node n is encountered for which h(n) is not known from the first abstract search ? HA*: compute h(n) by doing another abstract search Refinement: ignore n Generalization: choose randomly between the two strategies with probability P ( 0  P  1 ) Hybrid of HA* and Refinement (2)

31. 2-d Space of Algorithms 0 0.5 1.0  = weight given to g 1.0 P 0.5 0.0 A* Refinement Graph Traverser Optimal Refinement