1. A Colony of Robots Using Vision Sensing and Evolved Neural Controllers
A. L. Nelson, E. Grant, G. Barlow
Center for Robotics and Intelligent Machines
North Carolina State University
Raleigh, NC
T. C. Henderson
School of Computing
University of Utah
Salt Lake City, UT
My name is…
Today I’ll be presenting work conducted by Eddie Grant, Greg Barlow, Tom Henderson, and myself
The research involves (read)
11/21/2018
IROS 2003

2. Evolutionary Robotics
Evolutionary Robotics (ER)
Automat controller synthesis based on robot-environment interaction
Address complex behavioral controller domains in which designers have insufficient knowledge to develop a knowledge-based controller
Study behavior acquisition in robotic systems
ER is an emerging sub-field under the general heading of Behavioral robotics
Research in this field is focused on generating controllers for fully autonomous robots using evolutionary computing methods
In the general case robot controllers are trained (or evolved) to perform a task, or negotiate an environment…
And training is based on reinforcement feedback gained during robot interaction with their environment.
A main motivation behind ER is to develop fully automated methods of controller synthesis for autonomous robots
And a main benefit would be that intelligent or behavioral controllers could be developed for tasks that weren’t well understood by human designers
Most of the ER work done to date has involved very simple behavioral tasks such as phototaxis and has used a minimum of simple sensors
The work I be presenting today applies ER to a more complicated behavior, team game playing, and relies completely on a vision based sensor system that requires an order of magnitude more sensor inputs than other work in the field.
11/21/2018
IROS 2003

3. Overview
Artificial evolution was applied to evolve neural networks to control autonomous mobile robots
The robot controllers were evolved to play a competitive team game: Capture the Flag
Robots used vision based range-finding sensors
Selection during evolution was based on the results of robot-robot competition in a game environment
Controllers were evolved in simulation and transferred to real robots
Evolved controllers were tested in competition against a knowledge-based controller
Let me give you and overview of the research I’ll be presenting today.
(read directly)
11/21/2018
IROS 2003

4. Population-based Artificial Evolution in ER
Initialization P
(k=0)
Performance of
controllers ( p in P )
instantiated in robots
in an Environment 1 4 2 3
Fitness Evaluation
of each p in P
Based on
Performance in
Environment F( 1
) = # . n N
Re-order P
based
on fitness values p 1 2 3 4 5 6 N
(k)
{F(
) > F(
) > … }
Propagate P(k) to
P(K+1) using a
Genetic Algorithm (GA)
(mutation/crossover) p N 4 5 2 ' 1 6 3
2.6 P
(k)
(k+1) P ( k + 1 ) = p , 2 … N
Here is an overview of the evolutionary process in a typical ER application.
This is a cyclic process where each cycle is a generation (during evolution)
{step through}
11/21/2018
IROS 2003

5. Population-based Artificial Evolution in ER
Initialization P
(k=0)
Performance of
controllers ( p in P )
instantiated in robots
in an Environment 1 4 2 3
Fitness Evaluation
of each p in P
Based on
Performance in
Environment F( 1
) = # . n N
Re-order P
based
on fitness values p 1 2 3 4 5 6 N
(k)
{F(
) > F(
) > … }
Propagate P(k) to
P(K+1) using a
Genetic Algorithm (GA)
(mutation/crossover) p N 4 5 2 ' 1 6 3
2.6 P
(k)
(k+1) P ( k + 1 ) = p , 2 … N
Here is an overview of the evolutionary process in a typical ER application.
This is a cyclic process where each cycle is a generation (during evolution)
{step through}
11/21/2018
IROS 2003

6. CRIM Research Robots: The EvBots
EvBot with tactile sensors
Here are several pictures of robots used in the CRIM for evolutionary robotics experiments.
These robots were developed in the CRIM and are call EvBots.
Here is an EvBot fitted with a tactile sensor array.
These EvBots are fitted with colored shells that are used in conjunction with their color vision system.
Here is a picture of the next generation EvBot, the EvBotII developed by Leonardo Mattos.
The research Ill present today used Teams of EvBots as shown in this configuration {center}.
EvBots with cameras and colored shells
EvBot II
11/21/2018
IROS 2003

7. CRIM ER Test-Bed: Environment
The CRIM robot colony test-bed includes a physical reconfigurable maze arena
Here, Robots have been fitted with colored shells and are engaged in a searching and navigation behavior
11/21/2018
IROS 2003

8. Robotic Capture the Flag
Populations of robot controllers evolved to play Capture the Flag
Defend own goal (“flag”) while searching for opponent's goal
Games played in maze environments
Populations of robot controllers were evolved to play the competitive team game: Capture the Flag.
In the game, robots start near a stationary goal object (a home goal or flag)
During the course of the game robots must search the environment for their opponent's goal
Robots on each team must try to come in contact with their opponent’s goal while defending their own goal
These games are played in maze worlds.
11/21/2018
IROS 2003

9. CRIM ER Test-Bed: Video Range Emulation Sensors
Raw Image
Color Identification 50
100
150
Calculated Range Data
Walls
Red Robot
Green Robot
Red Goal
Horizontal Position (pixels)
Green Goal
Raw Image
Color Identification 50
100
150
Calculated Range Data
Walls
Red Robot
Green Robot
Red Goal
Horizontal Position (pixels)
Green Goal
Raw Image
Color Identification 50
100
150
Calculated Range Data
Walls
Red Robot
Green Robot
Red Goal
Horizontal Position (pixels)
Green Goal
The robots used a vision-base range-emulation sensor system.
Here I’ll some examples of robot-eye-views of the maze environment and how those images are converted into range data
{1st picture} This is a maze wall and {2nd picture} here are robots and objects in the back ground.
The vision system takes advantage of geometric and color constants in the environment to calculate object range and material-type.
The images are processed sequentially.
First colors are identified, then object-type, –based on color and geometry…
Then ranges are calculated for each object type by summing vertical pixel columns.
The resulting processed data are in the form of five range data vectors, one for each type of object that the robots can detect.
These are red and green robots, red and green goals.
(shoe next examples)
11/21/2018
IROS 2003

10. CRIM ER Test-Bed: Real vs. Simulation
These two panels compare real sensor data generated by a real robot, to that generated by a simulated robot in a simulated environment of the same configuration.
The real sensor data have been use to generate a graphic similar to that displayed in simulation. This has been superimposed onto an overhead view of the real maze for comparison.
Real sensors
Simulated sensors
11/21/2018
IROS 2003

11. Monolithic Neural Controllers
Here is an example of a ANN controller structure
This is a matrix representations of the network
And here is a graphical representation of the same network
This networks are fully evolvable and support arbitrary connectivity, mixed neuron types and can vary in size during evolution
The network shown is a small example network. The actual evolved networks are much larger and have 150 inputs and 5000 or more weighed connections
ANN Matrix Representation
ANN Graphic Representation
11/21/2018
IROS 2003

12. Controller Overview Artificial Neural Net: 150 Inputs 2 Outputs Robot50
100
150
Calculated Range Data
Walls
Red Robot
Green Robot
Red Goal
Horizontal Position (pixels)
Green Goal
Real-Valued
Numerical Arrays
Video Image
Sensor Input
Color
Identification
Artificial
Neural Net:
150 Inputs
2 Outputs G R L
Drive Motors
(Differential
Steering)
Robot
Treaded Base
Let me walk through a complete outline of the sensor and control structure we used
11/21/2018
IROS 2003

13. Controller Overview Artificial Neural Net: 150 Inputs 2 Outputs Robot50
100
150
Calculated Range Data
Walls
Red Robot
Green Robot
Red Goal
Horizontal Position (pixels)
Green Goal
Real-Valued
Numerical Arrays
Video Image
Sensor Input
Color
Identification
Artificial
Neural Net:
150 Inputs
2 Outputs G R L
Drive Motors
(Differential
Steering)
Robot
Treaded Base
11/21/2018
IROS 2003

14. The Bimodal Fitness Function
Accommodates sub-minimally competent initial populations (the Bootstrap Problem)
Relaxes human bias based selection in minimally competent populations
Makes use of the Red Queen Affect to evolve complexity
Fitness F(p) of an individual p in an evolving population P takes the general form:
I won’t go through this slid in detail as the main focus of this talk is on the sensor an evolutionary neural network portions of this research.
The evolutionary process is driven by a fitness selection function
In this work, fitness was calculated based on competition to win the game.
During a generation, controllers competed against one another it a tournament
Those winning more games were deemed fitter and selected for propagation
When evolving controllers for a given behavioral task, it is desirable to limit the amount of human bias in selection.
Any type of bias can potentially curtail a controller search space.
More importantly, we want methods that can evolve robot controllers to do do tasks that humans do not know how to perform, or perform well,
Hence fitness functions that require human knowledge and bias to formulate are not truly consistent with the motivation behind ER.
In order to generate truly bias free selection, in the ideal case, controllers would be selected based only on weather they could complete a given complex task or not. All fitness evaluation can then be aggregated into a function of a single binary success/failure term.
In the case of competitive relative fitness selection, success/failure evaluation may be able to be used to evolve increasingly complex behavior because it provides a continually increasing task difficulty due to the increasing competence of the other competing controllers.
That really cannot be said for any other methods of fitness selection used in ER.
Unfortunately, for most complex tasks, randomly initialized controllers have no detectable ability to complete the overall task,
This is referred to as the Bootstrap problem
Randomly initialized populations of controllers that display no delectable ability to accomplish an overall complex behavior are called sub-minimally competent.
To overcome the bootstrap problem, it is likely necessary to inject human bias into the evolving system in one form or another
In order to accommodate the bootstrap problem but still allow for un-fettered intra-population competitive aggregate selection, This research used a fitness function with two modes: a bimodal fitness function
This is given by F(p),
In the equation, Fmode_1 is the initial minimal-competence mode and Fmode_2 is the purely success/failure based mode.
Here OX indicates dependant exclusive-or: if the success/failure based mode’s value is non-zero, it is used and any value from Fmode_1 is discarded. Otherwise fitness is based on the output of Fmode_1
11/21/2018
IROS 2003

15. Evolved controllers in a simulated maze-world Winner: Red
This slide shows a movie sequence with the evolved controllers.
11/21/2018
IROS 2003

16. Movie: Evolved controllers in the real world
Here is a movie of game sequences played between evolved neural controllers transferred to real robots.
11/21/2018
IROS 2003

17. Post-Evolution Testing
ANN Wins
Rule-base Wins
Draws 10 20 30 40 50 60 70 80 90
100
110
120
130
Wins per Tournament
Population 1
Population 2
Controller population evolutions replicated
Evolved controllers tested in competition with knowledge-based controller
Evolved controllers competitive with knowledge-based Test controller
The evolution process was replicated several times and with a number of parameter variation.
In order to measure the quality of the evolved controllers, extensive tournaments of games were played (240 games) between evolved controllers and a hand coded knowledge-based controller of know abilities.
The bar graph shows the outcomes of two such tournaments using the best-evolved controllers from two separate evolutionary runs
Each triplet of clustered bars shows the number of ANN wins, the number of rule-base wins, and the number of draws for a given tournament.
Each type of evolution was repeated twice. The first two tournaments involved all-in-one evolutions
While the third and fourth tournaments involved incremental evolutions.
These data show that both types of evolutions produces controllers that could play competitively against the rule-based controller.
Additionally the first and fourth tournaments were won by the evolved ANN controllers.
11/21/2018
IROS 2003

18. Summary of Results
Artificial evolution was used to evolve robot controllers
Competitive team game behavior: Capture the Flag
Relative competitive Fitness Selection Function
Large networks using color vision based sensors were evolved
Evolved controllers were tested on real robots in a physical maze environment
Fully evolved controllers were tested in competition against a knowledge-based controller
Artificial evolution was used to evolve mobile robot controllers to play the game Capture the Flag
A bimodal fitness function was defined and used to drive the artificial evolution process.
The bimodal fitness function accommodates the bootstrap problem by selecting individuals in a population for minimal competence…
but it allows for competitive aggregate selection in controllers that have attained minimal level of competence.
Competitive selection is more powerful than other selection methods because it allows for the continual ramping-up of effective task difficulty.
Neural networks evolved to use processed video images for sensing of their environment.
This work used far more sensor inputs than other work in the field…
and is the first ER work to rely completely on processed color vision.
Very large complex Networks with a general architecture that allows for arbitrary connectivity and variable size were evolved.
These networks accommodated the large number of processed video inputs,…
The size and complexity of the networks is also relevant to the further development of ER:
For truly complex behaviors, it is likely that large complicated networks will need to be evolved.
Finally, fully evolved networks were tested in competition with a controller of well-defined abilities.
This served as a metric for fully-evolved controller quality outside the context of evolution.
and produced information was not evident or easily extractible from the data generated during evolution.
The best controllers from different populations were compared to each other by comparing their performances against the knowledge-based controller.
In addition, the acquisition of task skill at various points during evolution was measure using competition with the knowledge-based controller, and also the best controller from the final generation of a population.
11/21/2018
IROS 2003