Evolution, Brains and Multiple Objectives By Jacob Schrum schrum2@cs.utexas.edu

About Me B.S. from S.U. in 2006 Currently Ph.D. student at U.T. Austin Majors: Math, Computer Science and German Honors Thesis w/ Walt Potter: Genetic Algorithms and Neural Networks Currently Ph.D. student at U.T. Austin Received M.S.C.S. in 2009 Neural Networks Research Group: Genetic Algorithms and Neural Networks

Evolution Change in allele frequencies in population Alleles = variant gene forms Genes ⇨ traits Traits affect: Survival Reproduction Natural selection favors good traits

Genetic Algorithms Abstraction of evolution Genes = bits, integers, reals Natural selection = fitness function Mutation = bit flip, integer swap, random perturbation, … Crossover = parents swap substrings Other representations, mutation ops, crossover ops, …

Boolean Satisfiability Applications Boolean Satisfiability K. A. De Jong and W. M. Spears, “Using Genetic Algorithms to Solve NP-Complete Problems” ICGA 1989

Applications Magic Squares T. Xie and L. Kang, "An evolutionary algorithm for magic squares" CEC 2003

Applications Circuit Design J. D. Lohn and S. P. Colombano, "A circuit representation technique for automated circuit design" EC 3:3 Sep. 1999

Wing Design/Cost Optimization Applications Wing Design/Cost Optimization J. L. Rogers and J. A. Samareh, "Cost Optimization with a Genetic Algorithm" NASA Langley Research Center, RTA 705-03-11-03, October 2000

Traveling Salesman Problem Applications Traveling Salesman Problem P. Jog, J. Y. Suh, and D. van Gucht. "The effects of population size, heuristic crossover and local improvement on a genetic algorithm for the traveling salesman problem" ICGA 1989.

Resource-Constrained Scheduling Applications Resource-Constrained Scheduling S. Hartmann, "A competitive genetic algorithm for resource-constrained project scheduling" NRL 45 1998

Applications Lens Design X. Chen and K. Yamamoto, "Genetic algorithm and its application in lens design", SPIE 1996

Weight Selection for Fixed Neural Networks Applications Weight Selection for Fixed Neural Networks F.H.F. Leung, H.K. Lam, S.H. Ling and P.K.S. Tam, "Tuning of the structure and parameters of a neural network using an improved genetic algorithm" NN 14:1 Jan. 2003

What Are Neural Networks? Applications What Are Neural Networks?

Artificial Neural Networks Brain = network of neurons ANN = simple model of brain Neurons organized into layers

What Can Neural Networks Do? In theory, anything! Universal Approximation Theorem NNs are function approximators In practice, learning is hard Supervised: Backpropagation Unsupervised: Self-organizing maps Reinforcement Learning: Temporal-difference learning and Evolutionary computation

Neuro-Evolution Genetic Algorithms + Neural Networks Many different network representations Fixed length string Subpopulations for each Evolve topology and weights hidden layer neuron [1] [2] [1] F. Gomez and R. Miikkulainen, "Incremental Evolution Of Complex General Behavior" Adaptive Behavior 5, 1997. [2] K. O. Stanley and R. Miikkulainen, "Evolving Neural Networks Through Augmenting Topologies" EC 10:2, 2002.

Constructive Neuroevolution Population of networks w/ no hidden nodes Random weights and connections

Constructive Neuroevolution Evaluate, assign fitness Select the fittest to survive

Constructive Neuroevolution Fill out population Crossover and/or cloning Crossover Clone

Constructive Neuroevolution Random mutations Perturb weight, add link, splice neuron No mutation Perturb weight Add link Splice neuron

Constructive Neuroevolution Can add recurrent links as well Provide a form of memory

Neuroevolution Applications Double Pole Balancing F. Gomex and R. Miikkulainen, “2-D Pole Balancing With Recurrent Evolutionary Networks” ICANN 1998

Neuroevolution Applications Robot Duel K. O. Stanley and R. Miikkulainen, "Competitive Coevolution through Evolutionary Complexification" JAIR 21, 2004

Neuroevolution Applications Vehicle Crash Warning System N. Kohl, K. Stanley, R. Miikkulainen, M. Samples, and R. Sherony, "Evolving a Real-World Vehicle Warning System" GECCO 2006

Neuroevolution Applications http://nerogame.org/ Training Video Game Agents K. O. Stanley, B. D. Bryant, I. Karpov, R. Miikkulainen, "Real-Time Evolution of Neural Networks in the NERO Video Game" AAAI 2006

What I Do With Neuroevolution Discover complex behavior Multiagent domains Simulations, robotics, video games Support for multiple modes of behavior Multiobjective optimization

Mutiobjective Optimization Pareto dominance: iff Assumes maximization Want nondominated points NSGA-II [3] used Popular EMO method Nondominated [3] K. Deb, S. Agrawal, A. Pratap, T. Meyarivan, "A Fast Elitist Non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II" PPSN VI, 2000

Non-dominated Sorting Genetic Algorithm II Population P with size N; Evaluate P Use mutation to get P´ size N; Evaluate P´ Calculate non-dominated fronts of {P È P´} size 2N New population size N from highest fronts of {P È P´}

Evolve Game AI Game where opponents have multiple objectives Inflict damage as a group Avoid damage individually Stay alive individually Objectives are contradictory and distinct Opponents take damage from bat Player is knocked back by NPC

Intelligent Baiting Behavior

How to avoid stagnation Some trade-offs are too easy to reach Focus on difficult objectives TUG: Targeting Unachieved Goals Avoids need for incremental evolution Evolution Hard Objectives

Smaller Team w/ Expert Timing

Multitask Domains Perform separate tasks Predator/Prey Prey: run away Pred: prevent escape Front/Back Ramming Attack with ram on front Attack with ram on back

Multimodal Networks One network, multiple policies Multitask [4] = one mode per task Mode mutation = network chooses mode to use Multitask Mode Mutation Two tasks, two modes Start with one mode, mutation adds another Appropriate mode used for task Preference neurons control mode choice [4] R. A. Caruana, "Multitask learning: A knowledge-based source of inductive bias" ICML 1993

Multimodal Predator/Prey Behavior Learned with Mode Mutation Runs away in Prey task Corralling behavior in Predator task

Multimodal Front/Back Ramming Behavior Learned with Multitask Efficient front ramming Immediately turn around to attack with back ram

What about “real” domains? Unreal Tournament 2004 Commercial video game Basis for BotPrize competition: Bot Turing Test Placed 2nd with our bot: UT^2

UT^2 Behavior/Judging Game

Summary Neural networks can represent complex behavior Neuroevolution = way to discover this behavior Multiobjective evolution needed in complex domains Success in challenging designed/commercial domains

Questions. E-mail: schrum2@cs. utexas. edu Webpage: http://www. cs

Auxiliary Slides Empirical results

Differences for Alternating and Chasing significant with p < .05