1. Producing Artificial Neural Networks using a Simple Embryogeny Chris Bowers School of Computer Science, University of Birmingham cpb@cs.bham.ac.uk White space- No Structure/Cell Gray- Substrate/Dead/Non-functioning cells Red- Output neuron Green- Hidden neuron Light green- Developing hidden neuron Yellow- Stem cell Light blue- Developing input neuron Blue- Output neuron Black lines-Axons projected from cells Gray lines-Completed axon-dendrite connection between two cells Figure 2 - Example of individual phenotype The model described here has highlighted many problem areas in the use of an embryological system. Mainly that of the huge and rugged genotype landscape which makes evolution towards an optimal ANN extremely hard. Directions for further work are mainly concerned with improving the evolvability of the systems described here: Ensure the representation used has the flexibility to exhibit safe mutation and allow development processes to be initiated at different times in the development? In natural development these are know as canalisation and heterochrony respectively and seem to be extremely important in allowing efficient evolution. The capabilities of a genotype representation can be expanded by adding more genes to the genotype. However this results in variable genome length which introduces a new set of problems for evolutionary computation. Nature overcomes these problems using gene duplication. Artificial Neural Networks (ANN’s) can be extremely powerful and this power is mainly a result of the hugely parallel structure exhibited by these systems. However predetermining the structure of such complex networks is entirely problem dependent and so there is no strong theory behind how to construct an ANN given a particular problem. The fact that the structure of a successful ANN cannot be predetermined has resulted in limited success in producing networks which can solve large complex problems. This is due to an almost exponential increase in the numbers of parameters describing a network as the size of a network is increased. In order to overcome this problem of scalability, neural networks have been integrated into evolutionary systems (EANN’s) so that optimal neural networks for more complex problems can be found. This approach has resulted in novel architectures and networks never achieved by hand coded methods*. However, this has had limited success at producing large scale complex networks since the size of the genetic search space is directly related to the number of free parameters in the ANN. So for larger networks this results in an enormous search space. Therefore, most EANN implementations still result in a lack of scalability when considering large ANN’s. In recent years, the use of an even more biologically inspired method for EANN’s has been suggested. Nature solves this problem of scalability by evolving the rules upon which biological neural networks are grown. These growth rules are applied at the cellular level during development in a process known as embryogeny. This means that the size of the genetic search space is no longer dependent upon the size of the neural networks but upon the amount of linearity, repetition and modularity exhibited within the resultant network. The work shown on this poster is an example of how a simple model of embryogeny can be used to grow neural networks and identifies some of the difficulties highlighted by this approach. Introduction Further Work Problems with an embrogeny approach Since an embryogeny introduces a complex mapping into the evolutionary process this effectively results in two separate search spaces. a genotype search space which consists of all possible growth rules and is entirely dependent upon the representation used for these growth rules. a phenotype search space which consists of all possible neural networks, many of which would produce identical results and many of which are totally invalid neural networks (this space may be infinite in size). Considering that the size of the phenotype space is likely to be much larger than that of the genotype space and that also there is likely to be a many to one mapping between the genotype space and phenotype space means that the genotype space will only map to certain locations in the phenotype space. Therefore the representation used will limit the number of possible ANN architectures that can be produced (figure 3). It is imperative in this case that the representation used exhibit the following two characteristics when considering the search space. produce a fitness landscape in the genotypic space which is conducive to evolutionary search, i.e. as smooth and with as few local optima as possible. only map to areas in the phenotype space that consists of valid solutions and to keep neutrality in the fitness landscape of this areas in the phenotype space to a minimum. Process of mapping genotype to phenotype Determining the fitness of a genotype Genotype structure The genotypic structure is based upon a recurrent version of the Cartesian Genetic Programming (CGP) approach** originally developed for Boolean function learning. The genotype consists of a set of nodes represented in grid formation (figure 1). The number of rows in the grid is dependent upon the number of binary outputs required from the Boolean function. The number of columns is dependent upon the complexity of the Boolean function required. Each nodes defines a Boolean operation, such as NAND or NOT, and a set of binary inputs upon which the Boolean function operates. These inputs are either direct inputs to the CGP or from the output of other nodes. In this manor Boolean operations can be connected together in a graph to form a Boolean function which operates on a binary string. The genotypes are mapped to phenotypes using a growth process. This growth process is performed on a grid which defines an environment where, within each location, a cell can exist. In order to produce a growth step the state of each cell in the grid is encoded into a binary string upon which the genotype operates. The result of this operation is the new state which the cell must take. This is done in parallel on each cell in the grid. The state of a cell consists of: the current cell type. The cell type of surrounding cells. The level of chemical in the surrounding area. Axon direction and length This allows each cell to perform operations such as die, divide, move, differentiate and produce and axon. Figure 1 - An example of the genetic representation of CGP Genome 0205250270302431222710524066011825667051415331014312140813141951601881 7316713117125081517421221171901487022017613251271619261154219176352641 732801601153512421321183523012222341417221342271301480 Phenotype space Genotype space Complex mappings Figure 3 - Visualisation of genotype and phenotype mapping The aim of this work is not to produce a model of neural cell growth for developmental biologists or neuroscience but to try to understand the basic forces in development that allow it to be so powerful in expressing complex systems in comparatively simple genotypes. It is hoped that in future work it will be shown that development can be used to produce more complex systems than can currently be produced using a traditional approach without restrictions from issues of scalability. Only preliminary testing has been performed in order to determine the capabilities of this model. Initially a simple XOR network was found quite easily and several solutions have been found for various sized n-parity problems. However, in order to really test the scalability of this embryogeny the task was extended to the harder problem of finding a genome which solves all n-parity problems in n growth steps. This has proved extremely difficult. There are a number of important problems which have not been dealt with so far in this work. Since a simple back propagation method was used to determine the weights in the network the training performance was still dependent upon the size of the network and so network size is still limited in this model by computational power required to obtain optimal weights. The genotype search space can still be quite large for more complex genotypes and it is likely that the representation used here results in a rather rugged search space which is difficult for evolution to navigate. Discussion * X. Yao, Evolving artificial neural networks, Proceedings of the IEEE, 87(9):1423-1447, September 1999 ** J. F. Miller, P. Thomson, Cartesian Genetic Programming, Third European Conference on Genetic Programming, Edinburgh, April 15-16, 2000, Proceedings published as Lecture Notes in Computer Science, Vol. 1802, pp. 121-132 1.Does the network consist of the correct number of inputs and outputs. 2.Does the network connectivity result in a valid network. 3.How well does the network perform when trained on a set of training patterns taking an average of a number of randomly initialised networks. The fitness of a given genotype is dependent upon the fitness of its resultant phenotype. Since the phenotype defines a grid occupied by an arrangement of cells with various axons protruding from cells then an interpretation algorithm is required to produce a ANN structure form this phenotype. The process consists of first determining each cell type and then determining its connectivity with other cells in the network. Connectivity between two cells is dependent upon the distance of the axon head of one cell to the body of the other (figure 2). Once this is completed the network can be evaluated which requires a three stage process.