RADHA-KRISHNA BALLA 19 FEBRUARY, 2009 UCT for Tactical Assault Battles in Real-Time Strategy Games

Overview I. Introduction II. Related Work III. Method IV. Experiments & Results V. Conclusion

I. Introduction II. Related Work III. Method IV. Experiments & Results V. Conclusion

Domain RTS games  Resource Production  Tactical Planning Tactical Assault battles

RTS game - Wargus Screenshot of a typical battle scenario in Wargus Battle 1 Battle 2 Enemy group Friendly group

Planning problem Large state space Temporal actions Spatial reasoning Concurrency Stochastic actions Changing goals

I. Introduction II. Related Work III. Method IV. Experiments & Results V. Conclusion

Related Work Board games – bridge, poker, Go etc.,  Monte Carlo simulations RTS games  Resource Production  Means-ends analysis  Tactical Planning  Monte Carlo simulations  Nash strategies  Reinforcement learning Bandit-based problems, Go  UCT

Our Approach Monte Carlo simulations UCT algorithm Advantage  Complex plans from simple abstract actions  Exploration/Exploitation tradeoff  Changing goals

I. Introduction II. Related Work III. Method IV. Experiments & Results V. Conclusion

Method Planning architecture UCT Algorithm Search space formulation Monte Carlo simulations Challenges

Online Planning Framework UCT planner Action dispatcher Stratagus engine Current game state: Unit locations and hit points Ground actions: Move (unit1, pos1, pos2) Attack (unit1, unit2) Abstract game state: Group locations and hit points Abstract actions: Join (f1, f2) Attack (f1, e1)

Abstraction Abstract state space  Grouping of units Abstract actions  Join(G)  Attack(f,e) f1 f2 f3 e1 e2

UCT Algorithm Monte Carlo simulation – get subsequent states Search tree  Root node – current state  Edges – available actions  Intermediate nodes – subsequent states  Leaf nodes – terminal states Rollout-based construction Value estimates Exploration/Exploitation tradeoff

UCT Source: Achieving Master Level Play in Computer Go – Sylvain Gelly and David Silver

UCT Algorithm - Formulae Action Selection: Value Updation:

Exploitation Source: Achieving Master Level Play in Computer Go – Sylvain Gelly and David Silver

Exploration Source: Achieving Master Level Play in Computer Go – Sylvain Gelly and David Silver

Search Space Join Actions f1 f2 f3 e1 e2 f1 f2’ e1 e2 Join (f1, f2)

Search Space Attack Actions f1 f2’ e1 e2 f1 f2’’ e1 Attack (f2’, e2)

Monte Carlo Simulations Domain-specific Actual game play  Join actions  Attack actions Reward calculation – objective function  Time  Hit points

Domain-specific Challenges State space abstraction  Grouping of units (proximity-based) Concurrency  Aggregation of actions  Join actions – simple  Attack actions – complex (partial simulations)

Planning problem - revisited Large state space Temporal actions Spatial reasoning Stochastic actions Changing goals Concurrency - Abstraction - Monte Carlo simulations - UCT (online planning) - UCT (objective functions) - Aggregation of actions

I. Introduction II. Related Work III. Method IV. Experiments & Results V. Conclusion

Experiments # Scenario Name # of friendly groups Friendly groups composition # of enemy groups Enemy groups composition # of possible ‘Join’ actions # of possible ‘Attack’ actions Total # of possible actions 12vs22{6,6}2{5,5}145 23vs23{6,2,4}2{5,5}369 34vs2_14{2,4,2,4}2{5,5} vs2_24{2,4,2,4}2{5,5} vs2_34{2,4,2,4}2{5,5} vs2_44{2,4,2,4}2{5,5} vs2_54{2,4,2,4}2{5,5} vs2_64{2,4,2,4}2{5,5} vs2_74{3,3,6,4}2{5,9} vs2_84{3,3,3,6}2{5,8} vs4_12{9,9}4{4,5,5,4} vs4_22{9,9}4{5,5,5,5} vs4_32{9,9}4{5,5,5,5} vs5_12{9,9}5{5,5,5,5,5} vs5_22{10,10}5{5,5,5,5,5} vs43{12,4,4}4{5,5,5,5}31215 Table 1: Details of the different game scenarios

Planners UCT Planners  UCT(t)  UCT(hp) Number of rollouts – 5000 Averaged over – 5 runs - minimize time - maximize hit points

Planners Baseline Planners  Random  Attack-Closest  Attack-Weakest  Stratagus-AI  Human

Video – Planning in action Simple scenario – 2 vs 2 UCT(t) – optimize timeUCT(hp) – optimize hit points

Video – Planning in action Complex scenario – 3 vs 4 UCT(t) – optimize timeUCT(hp) – optimize hit points

Results Figure 1: Time results for UCT(t) and baselines.

Results Figure 2: Hit point results for UCT(t) and baselines.

Results Figure 3: Time results for UCT(hp) and baselines.

Results Figure 4: Hit point results for UCT(hp) and baselines.

Results - Comparison Figures 1, 2, 3 & 4: Comparison between UCT(t) and UCT(hp) metrics Time resultsHit point results U C T (t) U C T (hp)

Results Figure 5: Time results for UCT(t) with varying rollouts.

I. Introduction II. Related Work III. Method IV. Experiments & Results V. Conclusion

Conclusion  Hard planning problem  Less expert knowledge  Different objective functions Future Work  Computational time – engineering aspects  Machine Learning techniques  Beyond Tactical Assault

Thank you