1. Evolving Multimodal Networks for Multitask Games
Jacob Schrum –
Risto Miikkulainen –
University of Texas at Austin
Department of Computer Science

2. Evolution in videogames
Automatically learn interesting behavior
Complex but controlled environments
Stepping stone to real world
Robots
Training simulators
Complexity issues
Multiple contradictory objectives
Multiple challenging tasks

3. Multitask Games NPCs perform two or more separate tasks
Each task has own performance measures
Task linkage
Independent
Dependent
Not blended
Inherently multiobjective

4. Test Domains Designed to study multimodal behavior
Two tasks in similar environments
Different behavior needed to succeed
Main challenge: perform well in both

5. Front/Back Ramming Same goal, opposite embodiments Front Ramming
Attack w/front ram
Avoid counterattacks
Back Ramming
Attack w/back ram
Avoid counterattacks

6. Predator/Prey Same embodiment, opposite goals Predator Prey
Attack prey
Prevent escape
Prey
Avoid attack
Stay alive

7. Multiobjective Optimization
High health but did not deal much damage
Game with two objectives:
Damage Dealt
Remaining Health
A dominates B iff A is strictly better in one objective and at least as good in others
Population of points not dominated are best: Pareto Front
Weighted-sum provably incapable of capturing non-convex front
Tradeoff between objectives
Dealt lot of damage,
but lost lots of health

8. NSGA-II Evolution: natural approach for finding optimal population
Non-Dominated Sorting Genetic Algorithm II*
Population P with size N; Evaluate P
Use mutation to get P´ size N; Evaluate P´
Calculate non-dominated fronts of {P È P´} size 2N
New population size N from highest fronts of {P È P´}
*K. Deb et al. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II. Evol. Comp. 2002

9. Constructive Neuroevolution
Genetic Algorithms + Neural Networks
Build structure incrementally (complexification)
Good at generating control policies
Three basic mutations (no crossover used)
Perturb Weight
Add Connection
Add Node

10. Multimodal Networks (1)
Multitask Learning*
One mode per task
Shared hidden layer
Knows current task
Previous work
Supervised learning context
Multiple tasks learned quicker than individual
Not tried with evolution yet
* R. A. Caruana, "Multitask learning: A knowledge-based source of inductive bias" ICML 1993

11. Multimodal Networks (2)
Starting network with one mode
Mode Mutation
Extra modes evolved
Networks choose mode
Chosen via preference neurons
MM Previous
Links from previous mode
Weights = 1.0
MM Random
Links from random sources
Random weights
Supports mode deletion
MM(P)
MM(R)

12. Experiment Compare 4 conditions: 500 generations Population size 52
Control: Unimodal networks
Multitask: One mode per task
MM(P): Mode Mutation Previous
MM(R): Mode Mutation Random + Delete Mutation
500 generations
Population size 52
“Player” behavior scripted
Network controls homogeneous team of 4

13. MO Performance Assessment
Reduce Pareto front to single number
Hypervolume of dominated region
Pareto compliant
Front A dominates front B implies HV(A) > HV(B)
Standard statistical comparisons of average HV

15. Front/Back Ramming Behaviors
Multitask
MM(R)

17. Predator/Prey Behaviors
Multitask
MM(R)

18. Discussion (1) Front/Back Ramming
Control < MM(P), MM(R) < Multitask
Multiple modes help
Explicit knowledge of task helps

19. Discussion (2) Predator/Prey MM(P), Control, Multitask < MM(R)
Multiple modes not necessarily helpful
Disparity in relative difficulty of tasks
Multitask ends up wasting effort
Mode deletion aids search for one good mode

20. How To Apply Multitask good if: Mode mutation good if:
Task division known, and
Tasks are comparably difficult
Mode mutation good if:
Task division is unknown, or
“Obvious” task division is misleading

21. Future Work Games with more tasks Games with independent tasks
Does method scale?
Control mode bloat
Games with independent tasks
Ms. Pac-Man
Collect pills while avoiding ghosts
Eat ghosts after eating power pill
Games with blended tasks
Unreal Tournament 2004
Fight while avoiding damage
Fight or run away?
Collect items or seek opponents?

22. Conclusion Domains with multiple tasks are common
Both in real world and games
Multimodal networks improve learning in multitask games
Will allow interesting/complex behavior to be developed in future

23. Questions? Jacob Schrum – schrum2@cs.utexas.edu
Risto Miikkulainen –
University of Texas at Austin
Department of Computer Science