1. Robot Intelligence Technology Lab. 8. Competitive co-evolution From ‘Evolutionary Robotics’ Presented by Jeongki, Yoo `07.01.23

2. 2 Robot Intelligence Technology Lab. Contents Introduction Co-evolutionary complications Co-evolving predator and prey robots Evolution of predator and prey robots : a basic experiment Testing individuals against all discovered solutions How the length of “arms race”may vary in different conditions The role of ontogenetic plasticity in co-evolution Conclusions

3. 3 Robot Intelligence Technology Lab. 8.1 Introduction Competitive co-evolution : the evolution of two or more competing populations with coupled fitness May produce increasingly complex evolving challenges  Dawkins and Krebs (1979)  “ arms race “  Ex) predators and prey model Incremental Training Good to learn a complex task which cannot be learned at once  Also efficient for artificial evolution like neural networks  By gradually modifying the fitness function and/or the characteristics of the environment  Need careful attention and planning by the experimenter The variety of tasks forced by every single individual.  Over generations, individuals face variety of opponents(cases) to be more general solution

4. 4 Robot Intelligence Technology Lab. 8.2 Co-evolutionary complications Co-evolution could spontaneously produce a form of incremental evolution capable to select more complex competencies “arm races” Difficulty in competitive co-evolution : Continuous increase in complexity is not guaranteed Co-evolving populations might drive one another along twisting pathways  Each new solutions is just good enough to counter balance the current strategies  Not necessarily more complex than earlier solutions  May cycle between alternative classes of strategies

5. 5 Robot Intelligence Technology Lab. 8.2 Co-evolutionary complications A>B : A giving it an advantage over population B According to left condition of advantage, the cycling of the same strategies will be repeated over and over again  Difficulty in competitive co-evolution : Cycling may cancel out several of the previously described advantages -Evolution might rediscover previously selected strategies that could be adopted with only a few genetic changes

6. 6 Robot Intelligence Technology Lab. 8.2 Co-evolutionary complications Difficulty in competitive co-evolution : Changes in the fitness landscape and the external assessment of progress Changes in one species  affect the other species Particular combination of genetic traits evolved over some generations might quickly become ineffective in later generations if the opponent species changes its behaviors  modification of the fitness landscape

7. 7 Robot Intelligence Technology Lab. 8.2 Co-evolutionary complications Red Queen effect ( van Valen 1973;Ridley 1993) Makes it hard to monitor progress by using conventional indicators (ex. Average population fitness or fitness of the best individual) Need appropriate measuring techniques Using CIAO data ( Cliff and Miller(1995) )  CIAO data (current individual vs. ancestral opponents)  By testing the performance of the best individual at each generation against all the best competing ancestors Master Tournament ( Floreano and Nolfi (1997) )  By testing the performance of the best individual at each generation against the best competitors of all generations Above two methods : measure the average performance of one strategy against other discovered strategies  Measure the performance of co-evolutionary progress

8. 8 Robot Intelligence Technology Lab. 8.3 Co-evolving predator and prey robots Experimental Framework Used two Khepera robots  One for predator  with vision module  Another for prey  Twice faster than predator  Both : eight infrared proximity sensors  Six on the front,two on the back Environment  Infrared sensors can detect a wall at a distance of approximately 3cm  Could detect the other robot at only half of 3cm.  Evolved within a square arena of 47x47  Predator could always see the prey if within the visual angle.  Two robots are connected to a desktop workstation.

9. 9 Robot Intelligence Technology Lab. 8.3 Co-evolving predator and prey robots Robots were fitted with a conductive metallic ring around their base  to detect the collision against each other Vision module of the predator  One dimensionar array of 64 photoreceptors  provide linear image of 64 pixels of 256 grey levels (36 degrees range) Controllers Two simple neural networks with recurrent connections at the output layer  Having eight input units ( clamped to eight infrared sensors ) and two output units of two motors Neural controller of predator  Having additional five units that received information from the vision module

10. 10 Robot Intelligence Technology Lab. 8.3 Co-evolving predator and prey robots Two genetic algorithms were run in parallel  On the workstation CPU, but each newly decoded neurocontroller was downloaded through the serial line into the microcontroller of the Khepera robots. Input/output cycles : 15Hz for both prey and predator Image acquisition and low level visual preprocessing : 68HC11 processor Fitness:time of collision  Larger is better (0~499 internal clocks)

11. 11 Robot Intelligence Technology Lab. 8.4 Evolution of predator and prey robots : a basic experiment Initial set of exploratory experiments : from simulations The average population fitness the fitness of the best individual at each generation Pr : predator; py = prey. The fitness for the prey always tended to generate highter peaks due to position advantage  Not a good measure of progress Just mean the relative performance of the two species.  Prey is quickly encounter balanced by progress in the competing species Not telling whether evolutionary time corresponds to true progress or how to choose the best prey and the best predator

12. 12 Robot Intelligence Technology Lab. 8.4 Evolution of predator and prey robots : a basic experiment For measuring absolute performance Organize a master tournament where the best individuals for each generation are tested against the best competitors from all generations Individuals of later generations are not necessarily better than those from pervious ones The result of real experiment was similar to the simulated one

13. 13 Robot Intelligence Technology Lab. 8.4 Evolution of predator and prey robots : a basic experiment Characteristics Very quickly the two scores become closer and closer until 15 generations. After that, they diverge again in below two graphs. 25 generations are sufficient to display one oscillatory cycle.

14. 14 Robot Intelligence Technology Lab. 8.4 Evolution of predator and prey robots : a basic experiment The variety and complexity of behavioral strategies Below are the path of predator and prey for 13,20 and 22 generations For 13 generations: prey moves quickly around the environment and the predator attacks only when the prey is at a certain distance For 20 generations: the prey spins in place and, when the predator gets close, it rapidly avoids it. Predator draws a circle to find a prey.

15. 15 Robot Intelligence Technology Lab. 8.4 Evolution of predator and prey robots : a basic experiment Cycling(A for Predator, B for Prey) A1 : track the prey and try to catch it by approaching it A2 : wait the prey while remaining far, attacking the prey only on very special occasions B1 : stay still or “hidden” near a wall, escape after detecting predator B2 : move fast in the environment, avoiding both the predator and the walls

16. 16 Robot Intelligence Technology Lab. 8.5 Testing individuals against all discovered solutions To avoid “arms race” and cycling  save and use as competitors all the best individuals of previous generations (“hall of fame”) Could be implausible from a biological point of view New experiments,”hall of fame” selection regime Each individual is tested against 10 opponents randomly selected from all previous generations

17. 17 Robot Intelligence Technology Lab. 8.5 Testing individuals against all discovered solutions Left : An ideal situation in which predators are able to catch all prey of previous generations and the prey are able to escape all predators of previous generations Center : Typical result of a control simulation Right : Typical result of the “hall of fame” simulation in which each individual is tested against 10 opponents randomly selected from all previous generations

18. 18 Robot Intelligence Technology Lab. 8.6 How the length of “arms race”may vary in different conditions The structure of the sensory system can indeed affect the course of the co-evolutionary process and the length of “arms races” Prey has a limited sensory system ( limited distance, no direction difference) The prey cannot improve its strategy above a certain level because of this limitation Just suddenly changing behavior when predators adopt an effective strategy If the prey`s sensory system is improved, the other effective behavior sets could be expected.

19. 19 Robot Intelligence Technology Lab. 8.6 How the length of “arms races”may vary in different conditions Inserted Wide Vision sensor to prey 240 degrees of view range  wider than predator 150 photorecepters of 256 grey level Both predator and prey could see their competitors as a black spot against a white background Standard co-evolution was used  Individuals were tested against the best competitors of the 10 previous generations  Cf. In “ hall of fame” tested against all previous generations Prey is more successful than predators

20. 20 Robot Intelligence Technology Lab. 8.6 How the length of “arms race”may vary in different conditions Individuals obtained with “standard” co-evolution tend to outperform individuals obtained with “hall of fame” co-evolution Prob. of “hall of fame” prey defeating “hall of fame” predator > defeating “standard” predator Prob. of “hall of fame” predators are more likely to defeat “standard” than “hall of fame” prey  High variability between different replications “hall of fame” : low probability to fall into limit cycle. But, Not necessarily produce more general solutions Big change after changing the sensory-motor system  big change in the course of the evolutionary process  Importance of real experiment. Simulation simplifies sensory-motor system.

21. 21 Robot Intelligence Technology Lab. 8.7 How co-evolution can enhance the adaptive power of artificial evolution Can co-evolution really enhance the adaptive power of artificial evolution??? “Arms race” means adaptive power of evolutionary process Reasons for hypothesizing this assumption :  Individuals evolving in a co-evolutionary framework experiences a larger number of different environmental events  Co-evolution may produce an evolutionary “arms race” Evolving prey against the best predator (from co-evolution), predator against the best prey (from co-evolution) No cameras, only 8 ambient light sensors per each robot 16 sensory units ( 8 for infrared sensors, 8 for the ambient light sensors)

22. 22 Robot Intelligence Technology Lab. 8.7 How co-evolution can enhance the adaptive power of artificial evolution Standard selection regime used case 8 in 10 cases failed to select predators capable of catching the co-evolved prey Predators of the first generations are almost always unable to catch the prey and therefore the selection could not operate properly Co-evolution could produce more complex challenges than simple selection regime

23. 23 Robot Intelligence Technology Lab. 8.8 The role of ontogenetic plasticity in co- evolution If completely general solutions do not exist in some cases,  cycling problem  co-evolutionary dynamics lead to a limit cycle of solutions  also have to be treated as optimal The best can be done : to select the appropriate strategy for the current counterstrategy, which is what actually happens when co-evolutionary dynamics end in a limit cycle. If complete general solutions exists( successful against all strategies of previous opponents), co-evolution will lead to a progressive increase in complexity. If complete general solutions do not exist or are hard to be found  cycling dynamics  1) appropriate to the current strategy of the co-evolving opponents  2) easily modifiable so as to defeat new strategy of the co-evolving opponent Cycling dynamics : ideal situation for ontogenetic adaptation In limiting cycle  high selective pressure for ontogenetic adaptation

24. 24 Robot Intelligence Technology Lab. 8.8 The role of ontogenetic plasticity in co- evolution Full general vs. Plastic general individuals Full general individual : have a single strategy for one situation  More effective than plastic case. Because this is independent on past history of individual  Too difficult to find. Plastic general individual : can have several strategy according to the current or past situations  Might be difficult to obtain  Individual have to be capable of displaying a variety of different strategies  Also have to be capable of selecting the right strategy at each moment.

25. 25 Robot Intelligence Technology Lab. 8.9 Conclusions Competitive co-evolution allows the study of adaptation in an ever changing environment  much larger variety of behaviors Might produce incremental evolution without requiring additional supervison Cycling problem of co-evolution (limit cycles) Prevented by “hall of fame” Drawbacks of “hall of fame” : less and less pressure to discover strategies of current generation and increasing pressure of adapting past solutions. Full general vs Plastic general individuals “arms race” conditions are also sensitive to race conditions like sensory system.