Coevolution of Human-Competitive Robocode Tanks Using Genetic Programming with Exogenous Fitness Jason Owens & Ron Bowers

Why? Interesting GP and coevolution are particularly intriguing aspects of EC. We are using EC to generate an entertaining product. Challenging GP and coevolution are difficult to use. To date, no one has succeeded with coevolution in Robocode. Relevant Our employer is developing adaptive, agent-based combat simulations. We aren't working on them.

Coevolution Angeline and Pollock [1993] argue that competitive fitness functions are more effective than independent fitness functions. A competitive fitness function is dependent on the current state of the population, rather than relative to a global optimum. Generally, competitive fitness is measured through competition betweenpopulation members. These competitions can range from testing each member against each other to a carefully structured tournament Floreano, Nolfi and Mondada [1999] found that competitive coevolution can generate efficient chasing and escaping strategies in a predator/prey environment. This capability evolved even more quickly than "simple" wall avoidance They postulate that competitive coevolution generates solutions to difficult problems by generating a series of challenges of increasing difficulty. Uchibe and Asasa [2006] were able to use coevolution to evolve both competitive and cooperative behaviors in soccer playing bots.

Difficulties with Coevolution Mediocre Stable State Ficici and Pollack [1998] note additional difficulties. Maintaining balance in the arms race. The environment should neither be too easy nor too difficult: Open-ended coevolution is not necessarily synonymous with a purely competitive framework. Instead, it's necessary to provide a mixture of competitive and cooperative pressures in order to avoid the mediocre stable state.

Previous Work with Robocode Eisenstein [2003] Used a GA to evolve a subsumption architecture. Was successful in developing bots that could fight a specific adversary given a specific starting condition Attempted to use coevolution but after several generations I found the populations rife with catatonics Hong and Cho [2004] Used a GA that consisted of 6 chromosomes, representing the bot behavior in the main loop and in 5 of the event handlers. Each chromosome consisted of six genes, corresponding to actions such as move or shoot. Each action could be one of 2 or more hand-coded implementations. Were successful in consistently defeating 3 of the standard bots.

Previous Work With Robocode Shichel, Ziserman, and Sipper [2005] Used Koza-style GP Limited investigation to Haiku Bot (4 lines of code) Evolved bots were entered into a Haiku Bot tournament where they placed third out of 27. Also attempted coevolution, but could not produce competitive bots.

Hypothesis We hypothesize that by using genetic programming and coevolution with an exogenous fitness function we can evolve Robocode agents that can compete successfully against human-coded bots. But...

Initial Results We did not entirely succeed. We did not produce any competitive agents in time to report in the paper. We have continued our efforts!

Algorithm Configuration Strongly-typed tree-based GP Linear-rank selection using stochastic universal sampling 80% crossover, 20% mutation (whoa!) Elitism (one individual) Initial tree depth of 5

Fitness Functions simple (shoot and dodge) movement, enemy sensing, wall avoidance re-proportioned movement value final, emphasis on damage with firing/scanning efficiencies

Analysis of Initial Results Unearned dodging credit overblown scan credit Generally a rapid rise, then decay. Trees appear non-robust (brittle). Behavior arises from the coupling of trees and state. Different event histories lead to different states and thus radically different behavior.

Mistakes and Deficiencies Using a Single Population Using instantaneous fitness. Not monitoring the progress to determine if the population had started to degress. Threading issues (Heisenbugs) in Robocode.

Updated Results

Open Questions Can coevolution in general succeed in multi-dimensional one-population competitive environments without outside influence? Does this environment need the insertion of external human-coded bots to drive fitness adjustments? How do we modify the fitness function to improve behavior?

Future Work Continue experimenting with the exogenous fitness function, the node set, population size, and the reproduction parameters in an attempt to succeed with the coevolutionary approach. Incorporate the results from the progress measurement rounds into the selection process. Replace the single population model with a dual population model and investigate the efficacy of the evaluation approaches specified in De Jong [2004] and Bader-Natal [2005] Modify the driver program to limit the size of the trees.