1

2 overview Background on video games Background on video games Neural networks Neural networks NE NE NEAT NEAT rtNEAT rtNEAT NERO NERO

3 background The global video game market, according to analysts, will increase from $25.4 billion in revenue in 2004 to almost $55 billion in 2009, larger than even that of Hollywood. Techniques in artificial intelligence can potentially both increase the longevity of video games and decrease their production costs.

4 background In most games character behavior is scripted. In most games character behavior is scripted. no matter how many times the player exploits a weakness, that weakness is never repaired. So what to do?

5 The solution Machine learning can keep video games interesting by allowing agents to change and adapt. The agents can be trained to perform complex behaviors. if they would be trained offline (out-game learning) and then “ freeze ” and put in the game, they will not adapt and change in response to players in the game.

6 The solution That ’ s why agents should adapt and change in real-time, and a powerful and reliable machine learning method is needed. Prior examples in the MLG genre include the Tamagotchi virtual pet and the video “ God game ” Black & White.Black & White

7 The solution Here I introduce the machine learning : rtNEAT – real time Neuroevolution Of Augmenting Topologies. And the game that implement the rtNEAT: NERO – neuroEvolving robotic operatives.

8 Neural Networks way. A try to somehow mimic the neural network of a human being in a simplistic way. The network is a collection of simple units that mimic a neuron: The network is a collection of simple units that mimic a neuron: weightsinputWeighted weights Computing unit Weighted weights threshold output Σ

9 Neural Networks The network adapts as follows: change the weight by an amount proportional to the difference between the desired output and the actual output: The network adapts as follows: change the weight by an amount proportional to the difference between the desired output and the actual output: where η is the learning rate, D is the desired output, and Y is the actual output where η is the learning rate, D is the desired output, and Y is the actual output

10 NE – Neuroevolution The artificial evolution of neural networks using an evolutionary algorithm. The network The network can compute arbitrarily complex functions, can both learn and perform in the presence of noisy inputs, and generalize its behavior to previously unseen inputs.

11 Demands of video games AI 1) Large state/action space: high dimensional space and having to check the value of every possible action on every game tick for every agent in the game. - In NE agents are evolved to output only a single requested action per game tick

12 Demands of video games AI 2) Diverse behaviors: Agents should not all converge to the same Behavior - A homogeneous population would make the game boring. - In NE Diverse populations can be explicitly maintained through speciation

13 Demands of video games AI 3) Consistent individual behaviors: Characters should take a random action in order to explore new behaviors, but not periodically making odd moves. - In NE The behavior of an agent does not change because it always chooses actions from the same network.

14 Demands of video games AI 4) Fast adaptation and sophisticated behaviors: not waiting hours for agents to adapt require simple presentation but that results in simple behaviors. - In NE A representation of the solution can be evolved, allowing simple behaviors in the beginning and complexified later.

15 Demands of video games AI 5) Memory of past states: in order to react more convincingly to the present situation requires keeping track of more than the current state. - In NE Recurrent neural networks can be evolved, what gives us a memory of past situations.

16 NEAT A technique for evolving neural networks using an evolutionary algorithm. The correct topology not needed to be known prior to evolution. Unique in that it begins with minimal networks and adds connections and nodes to them over generations, allowing complex problems to be solved based upon simple ones.

17 Genetic encoding - Genome

18 Genetic encoding - Mutation

19 Genetic encoding - Mutation

20 Genetic encoding - results The splitting inserting into the system a nonlinearity. Old behaviors that were in the pre existing networks are not vanished and their quality stay almost the same, with the opportunity to refine those behaviors.

21 Cross over – historical markings When a mutation occurs a global innovation number is incremented and assigned. When cross over happens the offspring will inherit the innovation numbers. Thus the historical origin of every gene is known throughout the evolution.

22 Crossover – historical markings With innovation numbers we don ’ t need to do a topological analysis. Two genes with the same historical origin represent the same structure since they were both derived from the same ancestral gene at some point in the past

23 Crossover – historical markings

24 Speciation NEAT divides the population into species. The individuals compete primarily with their own niches. this way topological innovations are protected and have time to optimize their structure before competing with other niches, like new ideas that reach a potential before elimination.

25 Speciation Another advantage that Species with smaller genomes survive as long as their fitness is competitive, ensuring that small networks are not replaced by larger ones unnecessarily.

26 Speciation

27 Speciation To examine the similarity of networks a distance is formulated: E = number of excess, D = number of disjoint, W = average weight differences of matching genes, N = num of genes in the larger Genome (a normalizing factor) E = number of excess, D = number of disjoint, W = average weight differences of matching genes, N = num of genes in the larger Genome (a normalizing factor) c1,c2 and c3 defines the importance of the three factors. c1,c2 and c3 defines the importance of the three factors.

28 Speciation if a genome ’ s distance to a randomly chosen member of the species is less than δt, a compatibility threshold, the genome is placed into this species. δt can be dynamic, be raised if there are too many species and lowered if too few. If a genome is not compatible with any existing species, a new species is created.

29 Speciation – adjusted fitness The fitness of an organism is also defined by its niche: Where Where And you sum over all the distances from the organisms in the population resulting in the number of organisms in the same species as organism i. And you sum over all the distances from the organisms in the population resulting in the number of organisms in the same species as organism i.

30 Speciation - offspring to each species k is: The number of offspring distributed to each species k is: Where F k is the average fitness of species k, F tot = Σ k F k is the total of all species fitness average, and |P| is the size of the population Where F k is the average fitness of species k, F tot = Σ k F k is the total of all species fitness average, and |P| is the size of the population

31 Speciation - offspring First we eliminate the lowest performing members of the population. First we eliminate the lowest performing members of the population. The whole generation is then replaced by the new offspring. The whole generation is then replaced by the new offspring.

32 NEAT – minimization and complexification NEAT begins with uniform simple networks with no hidden nodes, differing only by their initial random weights. NEAT begins with uniform simple networks with no hidden nodes, differing only by their initial random weights. It adds connections and nodes incrementally. It adds connections and nodes incrementally. Only the most fitted structures remains. Only the most fitted structures remains. NEAT searches for the optimal topology by complexifying existing network. NEAT searches for the optimal topology by complexifying existing network.

33 NEAT - performance In experiments it has been shown that all three main components of NEAT are essential for it to work – the historical markings, speciation, and starting from minimal structure. In experiments it has been shown that all three main components of NEAT are essential for it to work – the historical markings, speciation, and starting from minimal structure. It was also shown that NEAT outperform other NE methods, especially those that are fixed topology evolution because it find faster complex solutions by starting with simple networks and expanding only when beneficial. It was also shown that NEAT outperform other NE methods, especially those that are fixed topology evolution because it find faster complex solutions by starting with simple networks and expanding only when beneficial.

34 rtNEAT In order for players to interact with evolving agents in real time NEAT was improved to rtNEAT. In order for players to interact with evolving agents in real time NEAT was improved to rtNEAT. Now fitness statistics are collected constantly as the game is played, and the agents evolved continuously as well. Now fitness statistics are collected constantly as the game is played, and the agents evolved continuously as well.

35 rtNEAT – replacement cycle Every few game ticks the worst individual is replaced by an offspring of parents chosen from among the best. Every few game ticks the worst individual is replaced by an offspring of parents chosen from among the best.

36 rtNEAT - algorithm Because rtNEAT performs in real time it cannot produce the whole generation at once like NEAT, and so the algorithm loop must change. Because rtNEAT performs in real time it cannot produce the whole generation at once like NEAT, and so the algorithm loop must change.

37 rtNEAT - algorithm

38 rtNEAT - algorithm 1)Calculating adjusted fitness: Where |S| is the number of individuals in the species. Where |S| is the number of individuals in the species.

39 rtNEAT - algorithm 2)Removing the worst Agent: a) if we remove the worst unadjusted fitness agent, innovation preservation will be damaged, because new small species will be eliminated as soon as they appear. That ’ s why the worst adjusted fitness agent will be removed.

40 rtNEAT - algorithm 2)Removing the worst Agent: a) if we remove the worst unadjusted fitness agent, innovation preservation will be damaged, because new small species will be eliminated as soon as they appear. That ’ s why the worst adjusted fitness agent will be removed.

41 rtNEAT - algorithm 2)Removing the worst Agent: b) agents must also have time to evaluated sufficiently before removed, because unlike NEAT that each individual live the same amount of time, in rtNEAT different agents have been around for different amount of time. That ’ s why rtNEAT only removes agents that have played for more than the minimum amount of time m.

42 rtNEAT - algorithm 2)Removing the worst Agent: c) we must re-estimate the average fitness F, because a specie now have one less member, and it ’ s average most likely been changed.

43 rtNEAT - algorithm 3)Creating offspring: the probability to choose a parent specie is proportional to it ’ s average fitness compared with the total of all species ’ average fitnesses: And a single new offspring is created by recombining two individuals from the parent species

44 rtNEAT - algorithm 4)Reassigning agents to species: because minimizing the number of species is important in real time environment, when CPU time is not allocated all to evolution, the threshold δt must be dynamically adjusted. But also the entire population must be assigned again to species.

45 rtNEAT - algorithm 4)Replacing the old agent with the new one: replacing depends on the game. You may just replace the neural network in the body of the removed agent. Or you may kill it and replace a new agent instead.

46 The loop interval should be every n ticks. For n to be chosen a “ law of eligibility ” is formulated: Where m is the minimum time alive, n is the number of ticks between replacement (the loop interval), |p| is the population size, and I is the fraction of the population that is too young to be evaluated. rtNEAT – loop interval

47 rtNEAT – loop interval And I left for the user to determine, because its most critical for performance.

48 the idea of the game is to train a team of agents by designing a training program. In order for the agents to learn, the learning algorithm is of course rtNEAT.

49 NERO - fitness In the training there are sliders that lets you configure what attributes you want your agents to have. In the training there are sliders that lets you configure what attributes you want your agents to have.

50 NERO - fitness fitness is the sum of all the components c i multiply by the sliders value v i : fitness is the sum of all the components c i multiply by the sliders value v i : To put in also the rate of forgetting r into the fitness this formula is given: To put in also the rate of forgetting r into the fitness this formula is given: Where f t is the current fitness. Where f t is the current fitness.

51 NERO - sensors Agents have several types of sensors. Agents have several types of sensors. Their output is: the direction of movement, and whether or not to fire. Their output is: the direction of movement, and whether or not to fire.

52 NERO - sensors Enemy radar: divides the 360 º area around the agent into slices. Each slice activates a sensor in proportion to how close the enemy in that slice. Enemy radar: divides the 360 º area around the agent into slices. Each slice activates a sensor in proportion to how close the enemy in that slice.

53 NERO - sensors Range finder: projects ray at several angles, and the distance the ray travels before it hits something is returned as the sensor value. Range finder: projects ray at several angles, and the distance the ray travels before it hits something is returned as the sensor value.

54 NERO - sensors On-target sensor: returns full activation only if a ray projected along the front heading of the agents hits an enemy. This sensor tells the agent whether it should attempt to shoot On-target sensor: returns full activation only if a ray projected along the front heading of the agents hits an enemy. This sensor tells the agent whether it should attempt to shoot

55 NERO - sensors Line-of-fire sensor: detect where a bullet from the closest enemy is heading. Thus, can be used to avoid fire. They work by computing where the line of fire intersects rays projecting from the agent. Line-of-fire sensor: detect where a bullet from the closest enemy is heading. Thus, can be used to avoid fire. They work by computing where the line of fire intersects rays projecting from the agent.

56 NERO – evolved networks Seekers VS Wall-fighters Seekers VS Wall-fighters

57