1. Evolutionary Computation for Creativity and Intelligence
By Lauren Gillespie, Gabby Gonzalez, and Alex Rollins

2. Neuroevolution: an overview
Neural Networks - Brains
Nodes – Neurons
Links – Synapses
Genotype - Phenotype
Evolutionary Algorithm - Abstraction of evolution
Asexual reproduction (mutation)
Sexual reproduction (crossover)
Survival of the fittest (selection)
Neural Networks + Evolution = Neuroevolution
Neuroevolution of Augmenting Topologies (NEAT)
Our group focuses on neural evolution using neural networks
Neural Networks are an abstraction a brain
1.) Neural Networks are made up of nodes – which are synonymous neurons,
Connected by links, which are like synapses between neurons.
2.) Neuroevolution also mimics biology in that a genotype of a network is used to create a phenotype, called an artificial neural network, which is then used to play tetris etc. However, it is the genotypes that are evolved, like in biology using an evolutionary algorithm
3.) Neuroevolution a computational abstraction of biological evolution, mimics Darwin’s evolution, descent with modification
a. NE mimics asexual reproduction through the mutation of genotypes
b. Also, mimics sexual reproduction via crossover of genotypes
c. It too mimics survival of the fittest using selection based on fitness score, aka only the best networks are chosen to reproduce
Using these evolutionary algorithms, over time the population of networks evolves to learn intelligent behaviors, such as Alex’s work in the torus world domain
4.) So now you can see where we get this idea of neuroevolution from, it is simply evolution applied to artificial neural networks! Neural Networks plus evolution, ergo neuroevolution!!

3. Compositional Pattern Producing Networks (CPPN)
(H, S, B)
(R, G, B) X Y D
Bias H S B
Actual Network
(X, Y, D, Bias)
distance
One special kind of network we use is a CPPN
Basic idea: network(as seen actual network) takes a location of a pixel and paints a color there. Coordinates to color.
Detailed: feed a neural network the x-y coordinates, the distance from the center and a bias
Values pass through network to produce corresponding color outputs
Takes coordinates of an input space (graph)
And then sends inputs (x, y, distance, bias) through the neural network
To produce color outputs that can then be painted at the corresponding pixel, as seen in diagram
CPPNs allow networks to learn spacial intelligence, which lead to some pretty interesting results…

4. As you can see, these networks can be evolved to produce a wide diversity of images.
These cool images you see evolved here were evolved using a program we created called picbreeder!

5. Picbreeder Demo
Basics: So the basic idea of picbreeder is that you take pictures and ‘breed’ them!
1.)This small population of images form a population that will be evolved
2.) I pick the most ‘fit’ (which is in this case the coolest looking pictures) and those are used to create the next generation
3.) the networks that encoded the chosen images are then ‘bred’ via crossover and mutation to create the next generation
Ergo, the next generation will look similar to the parents
But, occasionally you’ll get an outlier that looks different, has an adaptation
I like that one so I click on it and it populates the next generation, ergo evolution

6. Evolved Behavior Three blue predators Two green prey
World is torus shaped (wraps around)
Predators evolved to catch the prey
Evolved Behavior

7. Fitness Score Average Number of Prey Caught Generation
Multi-Objective Predators with Proximity Sensors
Fitness Functions Used:
Maximize Number of Prey Caught
Minimize Distance to the Prey
Minimize Elapsed Game Time
Explain: 1000 generations. 100 agents per population, one population per generation. Each agent was evaluated 20 times (twenty trials) and the average number of prey caught is found for each agent. This average number of prey caught is the fitness score, and the agents with the highest fitness scores survive into the next generation and new agents are constructed with evolutionary mutations based off of the traits and the genotypes of these “best” agents.
Generation

8. Fitness Functions Prey Predator Minimize Prey Survival Time
Minimize Distance
Maximize Number of Prey Caught
Explain the main three fitness functions/concepts (distance, speed, number caught)
Explain why the three fitnesses are not enough on their own:
Distance: gets close to the prey then goes into an infinite loop as close as possible so that it ends close but doesn’t care about catching prey
Speed: Based on minimizing the amount of time that the prey are alive, so never learns to catch prey in the first place because the randomization isn’t enough to catch the static prey
Number Caught: Never learns to catch prey in the first place because the randomization isn’t enough to catch the static prey
Single fitness functions can be extremely complex and can involve all of these same factors, but when creating these and adding all factors into one objective together, it is almost impossible to weight each individual piece of the objective perfectly.

9. Coevolution Evolved Predator Vs Static Prey
Controller
Prey Agent
Population
Grid World
Network
Predator Agent
Evolved Predator Vs Static Prey
Cooperative Coevolution of Predators
Competitive Coevolution (Homogenous Teams)
Competitive and Cooperative Coevolution
Explain: The research process that I am going through for SCOPE (finishing up with evolved predator and static prey, and beginning cooperative, heading towards competitive). Explain how the networks or static controllers are given to each agent according to the type of task being used. Explain that I will be implementing multi-Objective based fitness along with coevolution.

10. Evolved Tetris Player Checks each possible move it can make.
Uses neural network to rate each move
Picks the best action (maximum utility)
Evolved Tetris Player

11. Tetris Features
Based on features from Bertsekas, D. P. and Tsitsiklis, J. N. (1996). Neuro-Dynamic Programming. Athena Scientific

12. Fitness Score Evolved Tetris Player Game Score Generation
Tetris movie here too
Game Score
Evolved Tetris Player
Generation

13. Hybercube-based NEAT (HyperNEAT)
One way to evolve with the raw screen inputs, is that we use an extension of NEAT called HyperNEAT
It uses a CPPN, like from my earlier slide, but instead of painting a cool picture, it ‘paints’ a network onto a hyper-dimensional space
Advantages of hyperNEAT:
1.) allows the network to take advantage of the geometry of the domain space
2.) Actually uses another network, the CPPN, to encode another network, which then controls the agent
This is called indirect encoding and it allows us to create very complex, extremely large neural networks, like this one
Stanley, Kenneth O.; B. D'Ambrosio, David; Gauci, Jason (2009). "A Hypercube-Based Encoding for Evolving Large-Scale Neural Networks". Artificial Life 15: 185–212.

14. Tetris with Raw Feature Inputs (HyperNEAT)
Utility

15. Classic Game “Doom” (VizDoom)
- Use raw screen pixels to make decisions
- Try using single row in initial experiments

16. Using Full Screen (HyperNEAT)

17. Questions? Dr. Jacob Schrum Southwestern University
HHMI-Inquiry Initiative
Howard Hughes Medical Institute
Questions?

18. Auxiliary slides

19. How Neuroevolution works
Biological evolution
A population of creatures has slightly different traits based on DNA differences
Environment exerts pressure on population
Natural selection occurs, fittest members survive
Survivors reproduce both sexually and asexually
Different networks encode different phenotypes
Phenotypes compete in task
Networks evaluated on phenotype score
Mutation and crossover modify best networks
So, an ANN encodes a phenotype, just like your DNA encodes you (aka your phenotype)
These agents form a population, like all y’all here in the audience
They then agents perform their task, like if all y’all had to do as many pushups as possible
Then after task is done, the agents are evaluated based on their fitness scores, like if we crowned the top 5 people with the best number of pushups
Finally, the networks that encoded the best agents are mutated and crossed over to populate the next generation, like if we had those best 5 people reproduce to create next year’s group of SCOPE students!
Obviously it’s a little more complicated than that (plus would never make y’all do pushups because that’s just cruel) but that is the gist of the idea
Using these ideas, our research group is taking neuroevolution and training networks how to do cool stuff
Also, on the bottom you can see the evolution of computer scientists

20. Neuro-Evolution of Augmenting Topologies (NEAT)
Evolutionary Algorithm
Complex agents evolved from simple networks
Complexity built up via mutation and mating
Add a little more detail in case of questions
AUXILLARY SLIDE
Best friend – NEAT
Is an evolutionary algorithm
A way to evolve a population over time to tackle highly sophisticated, complex tasks
Follows the same line of thought as evolution (point to phylogenetic tree of life) in that complexity can emerge from a population of simple agents, like we arose from bacteria many moons ago
NEAT e

21. The Network and the Sensors
Network for a Single Agent
Inputs and Outputs for a Single Agent
Do Nothing
Explain: Inputs are the distances to that agent from this agent (this agent being the agent that is being defined by this network) along the X or the Y axis. Proximity sensors reorder the agents from closest agent to furthest agent then provides the inputs to the network instead of just simply listing the distances to the agents. This specific agent was given proximity sensors and was able to sense his teammates. Output is chosen based on the maximum of the sum of the weight of each edge going to it multiplied by the input value connected to that edge. The output is the next movement that the evolved agent will take. There are also hidden nodes between the inputs and the outputs which provide new edges with new weights in order to re-assess the value for that given set of inputs based upon the hidden node’s edge weight. At the beginning there are no hidden nodes and the edge weights are randomized, and the agent will add hidden nodes/edges and re-define the edge weights according to behaviors that will move that agent towards a better fitness score.

22. Multi-Objective Optimization
Imagine game with two objectives:
Minimize Distance
Maximize Number of Prey Caught
A dominates B if and only if A is strictly better in one objective and at least as good in others
Population of points not dominated are best: Pareto Front
Picture of pareto front thing for multiObjective
Why multiObjective can be and has shown to be better (more simple, efficient, effective)
It is much better, easier and more efficient to define them simply in a multi-Objective setup and let evolution figure out what works best situationally and overall.
Explain why the multiObjectives uses these three fitness functions:
Distance: Teaches the predators to get as close as possible to the prey so that they will begin to catch each prey
Speed: Once prey have begun being caught, predator agents that catch the prey quicker are preferred over other predators, so whatever actions inspired the speed survive to the next generation
Number Caught: Once prey have begun being caught, predator agents that catch a higher number of prey will be preferred