1. Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 1 Authors : Siming Liu, Christopher Ballinger, Sushil Louis simingl@cse.unr.edu http://www.cse.unr.edu/~simingl

2. Outline  Motivation  Prior work  Starcraft2 (the RTS game)  Data extraction and cleanup  Methodology  Weka, machine learning toolkit  Results  Conclusions and Future Work  Acknowledgements Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 2

3. Motivation  Group tactics (naval)  Allow autonomous movement for individual boats.  But allow boats to work together toward a goal.  Investigate Influence Maps (IMs) as a representation  Evolve (co-evolve) tactics for a given scenario and opponent  Discover new tactics  Generate variations on a theme Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 3

4. Previous Work  Asymmetrical game  Evolved (green) attackers to break through (red) defender’s line.  Influence Map used to guide navigation  Results:  Unbalanced game – rush always wins  The IM showed promise as a representation Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 4

5. Representation:Influence Maps  Influence Maps give a numeric value for every cell on a discretized map  Cell: -50 (red) to +50 (green)  Each entity uses an influence map for navigation  Entities navigate using A* towards red and away from green  A* distance heuristic: sum of cell values along path  IM updated every time step  A* NOT updated every time step – too expensive Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 5

6. Representation…  Each entity uses own IM:  Defines the spatial objectives for the entity for the current game state.  Provides a destination point for each entity (reddest).  Provides a cost function for A* – avoid green, prefer red.  By evolving the IMs for each entity in a group together – in one chromosome, we can evolve co-adapted IMs embodying cooperative tactics.  Decentralized control (but co-operation possible) Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 6

7. Genetic Representation  A chromosome defines a set of IMs  The gene for each entity is a set of parameters that specify an IM function  The IM function takes the game map as input.  The output is an IM for each entity in the group. Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 7

8. Genetic Representation Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 8

9. The New Game – Capture the Flag  Two symmetrical forces.  Each has their own flag to defend, and an enemy flag to reach.  Increased difficulty due to moving enemies.  Objectives:  Reach the Flag 1.Destroy Opposition 2.Survive  Time limit to capture the flag.  Game ends when either  flag is captured, or  time runs out. Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 9

10. The New Game…  Symmetric and (hopefully) balanced  Previously evolved attackers to beat static defenders, now evolving two symmetrical teams.  Two populations of individuals, each species representing a different team.  Each individual used the same chromosome representation.  Fitness determined through repeat game play with opposition individual.  Fitness score calculated with games won, enemy and player boats survived, time taken.  Beating a harder enemy (one that has repeatedly scored well) yields a bonus.  Added land – more complex tactics Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 10

11. Methodology  Coevolved two populations of 20 individuals  100 generations, 5% elite  10 games per individual against randomly selected opponent  10 runs on a cluster Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 11

12. Results  Continued improvement – previously only initial improvement.  Evolved attacking and defensive strategies.  Also saw more adaptability – boats changing tactics dependent on opposition movement.  Some strategies involved a stalemate, where the team would prioritize flag defense.  Most strategies involved some form of cat and mouse game, where boats would go towards the enemy then run away again when close. Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 12

13. Fitness over time Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 13

14. Movie - Evasive Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 14

15. Baseline Measurement  To provide baseline measurement of the coevolutionary growth, we programmed a Rush tactic to plot progress against.  Simple strategy, but suggested by prior work feedback from reviewer. Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 15

16. Baseline Measurement …  Graph shows the average results (fitness) of the best individual from each generation for one population against Rush over all runs.  Even with no knowledge of Rush, the coevolutionary process was able to repeatedly come up with solutions that beat the tactic.  Movie of evolved individual beating the tactic. Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 16

17. Movie – vs Rush Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 17

18. Discussion & Conclusions  IM representation + A* scales up to more complex scenarios.  More complex tactics  Even “simple” co-evolution generates these tactics  But we still have issues involving ‘stagnation’ of the evolved strategies; not showing enough adaptability.  This could also be due to the way the fitness function is implemented. The use of a multi-objective function might be more applicable. Hall of fame, …, Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 18

19. Future Work  Include further information:  Speed  Health  Investigate (other) co-evolutionary approaches  Changes to the EA:  More appropriate crossover for the larger chromosome created from land.  The fitness function needs to be redesigned to encompass the different goals of the game.  Long term goals:  Implement hierarchy of tactics.  Create a full RTS style game to test with humans. Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 19

20. Acknowledgements  This research is supported by ONR grants N00014-08-1-0781 and N00014-09-1-1121.  More information (papers, movies)  pippa@cse.unr.edu (http://www.cse.unr.edu/~pippa) pippa@cse.unr.edu  sushil@cse.unr.edu (http://www.cse.unr.edu/~sushil) sushil@cse.unr.eduhttp://www.cse.unr.edu/~sushil  New game engine: http://lagoon.cse.unr.edu Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 20

21. Fitness Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 21

22. Example Chromosome Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 22

23. The new chromosome  A chromosome is comprised of a set of genes for each entity, representing the IM function parameters for that entity’s IM.  There are parameters for each enemy (e) and friendly (f) unit, plus the two team flags (flg)  Additional parameters are given for the land points  The IM function takes the game map as input  The game map is a matching grid to the IM, with each cell containing 0 for empty, or the corresponding game entity value.  The output is an IM for each entity in the group.  Each cell is given the corresponding evolved parameter for the entity.  Enemy cells have the surrounding cells assigned a decreasing influence representing the attack range. Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 23

24. Including Land Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 24 Used image representing map in grid format: Black cells are marked as off-limits for the A* path. Land represented by four points The extreme top, bottom, left and right points chosen. This technique can be applied to any map. Sub-goals on the way to goal point Closest most red on the way to goal

25. Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 25