Research Related to Real-Time Strategy Games Robert Holte November 8, 2002

No Games Knowledge, but… Single-agent search = sequential decisions with additive costs –path-finding occurs in RTS games –are there other occurrences ? –transfer from 1-agent to 2-agent or N-agent games ? –is a 2-agent game with a deterministic opponent a single-agent search problem ? Path-finding research group (P. Yap, Yngvi, …)

and … Combinatorial optimization = resource allocation with complex constraints –combinatorial auctions –“boolean programming” –hill-climbing, ESG, a bit of CPLEX

and … Machine Learning –opponent-modelling (poker) –user-modelling (adaptive games, personalization) –learning to make good moves (M. Mueller) Interactive Systems –mostly-automatic evaluation methodology Abstraction in Poker

Using Abstraction to Speed Up Search

Search = path-finding in a graph GIVEN: –A graph –Two nodes in the graph (“start”, “goal”) FIND: – a shortest path from start to goal

Labyrinth at the gardens of Versailles (could be a computer game!)

“Implicit” Graphs Instead of an adjacency list/matrix, Define a graph by a successor function, succ, and one or more “seed” nodes. Nodes of the graph = transitive closure of succ applied to the seed nodes An edge exists from n 1 to n 2 iff n 2  succ(n 1 ) A graph can be exponentially larger than its description.

Towers of Hanoi puzzle for D disks: –graph has 3 D nodes –description is O(D 2 )

3-disk State Space

Speed up Search by Using Extra Information Solution “skeleton” = chain of subgraphs leading from start to goal –Refinement: search in the first subgraph until you reach the second, then search in the second until you reach the third, etc. –very fast but path found not guaranteed to be optimal Heuristic = h(n) = estimate of the distance from node n to the goal node. –if it never overestimates, it can eliminate (prune) parts of the graph and still guarantee finding an optimal path

My Research Aim: to generate heuristics or skeletal solutions automatically Method: 1.Create an abstraction of the given graph (simplified version of your problem that “approximately” preserves “important” features of the problem). 2.Find a solution path in the abstract graph. 3.Use the abstract solution as a skeletal solution, or use its length as a heuristic.

Example: Towers of Hanoi start goal

Abstraction: “Ignore the smallest disk” start goal

Abstract Solution start goal

Key Techniques For Automatically Generating Abstractions –explicit graphs: STAR abstraction –implicit graphs: domain abstraction –select among abstractions using Korf & Reid’s method for predicting search time Search Methods –refinement: AltO –heuristic: Hierarchical A*, pattern databases –anytime combination of refinement and heuristic search

STAR abstraction Abstract state = connected subgraph (all nodes within distance R of a given node) Fully automatic (but need to amortize cost) Fine control over granularity (R) Can be applied recursively (abstraction hierarchy) Guarantees refinement will succeed Produces a consistent heuristic

AltO Refinement Technique Conduct abstract search in the opposite direction to the base-level search; Use the whole abstract search graph for refinement. 12% faster than standard refinement and produces better solutions (20% shorter) Robust – not sensitive to the abstract solution found, and not very sensitive to the radius of abstraction times faster than A* (same abstraction) and solutions usually within 30% of optimal

Pattern Database Successes Joe Culberson & Jonathan Schaeffer (1994). –15-puzzle –PDB + Manhattan Distance (MD) reduced MD- alone search tree 1000-fold Rich Korf (1997) –PDBs enabled optimal solutions of Rubik’s Cube Stefan Edelkamp (2001) –PDBs competitive with best planners Istvan Hernadvolgyi –PDBs reduced macro-table solution lengths from 90 to 50

Example: 8-puzzle Domain = blank ,440 states

“Patterns” created by domain mapping This mapping produces 9 patterns Domain = blank Abstract = blank corresponding patternoriginal state

Pattern Database Pattern Distance to goal Pattern Distance to goal 3 3 4

Calculating h(s) Given a state in the original problem Compute the corresponding pattern and look up the abstract distance-to-goal Heuristics defined by PDBs are consistent, not just admissible.

Domain Abstraction ,240 patterns Domain = blank Abstract = blank 6 7 8

Automatic Creation of Domain Abstractions Easy to enumerate all possible domain abstractions They form a lattice, e.g. is “more abstract” than the domain abstraction above Domain = blank Abstract = blank Domain = blank Abstract = blank

8-puzzle: A* vs. PDB size # nodes expanded (A*) pattern database size (# of abstract states)

Hierarchical A* Unlike pattern databases, each h(n) is computed on demand. If h(n) is not already known compute it by searching at the abstract level. The search at an abstract level can be informed by a heuristic defined by a higher abstract level. * many abstract searches for one base-level search * naïve implementation is 10 times slower than blind base-level search.

Speed versus Optimality Refinement –very fast (when refinability guaranteed) –suboptimal paths A* –much slower –optimal paths See also: bidirectional search,  -admissible A*

Hybrid of HA* and Refinement (1) Although the algorithms seem very different there are actually only two differences. (1)The relative weight given to g and h. HA*: g + h Refinement: h (use g to break ties) Generalization:  *g + (1-  )*h ( 0    1 )

(2) What to do if a node n is encountered for which h(n) is not known from the first abstract search ? HA*: compute h(n) by doing another abstract search Refinement: ignore n Generalization: choose randomly between the two strategies with probability P ( 0  P  1 ) Hybrid of HA* and Refinement (2)

2-d Space of Algorithms  = weight given to g 1.0 P A* Refinement Graph Traverser Optimal Refinement