ELeaRNT: Evolutionary Learning of Rich Neural Network Topologies Authors: Slobodan Miletic 3078/2010 Nikola Jovanovic 3077/2010

Introduction Genetic algorithm Genetic algorithm mimics natural evolution mimics natural evolution candidate solution candidate solution mutation mutation Neural network Neural network based on biological neurons based on biological neurons network consists of neurons grouped in layers network consists of neurons grouped in layers 2/16 Slobodan Miletić Nikola

Problem definition Current design methods are manual and inefficient Current design methods are manual and inefficient Hard to define number of neurons and connections Hard to define number of neurons and connections No automated design method No automated design method for specific optimal topology for specific optimal topology 3/16 Slobodan Miletić Nikola

Problem importance Neural networks have a large use area Neural networks have a large use area Creating new neural networks Creating new neural networks takes money and time 4/16 Slobodan Miletić Nikola

Problem trend Computers are getting more powerful Computers are getting more powerful New neural network uses are found New neural network uses are found If not solved, this problem would slow down the evolution of neural networks If not solved, this problem would slow down the evolution of neural networks 5/16 Slobodan Miletić Nikola

Existing solutions Trial & Error Trial & Error Manual algorithm Manual algorithm Few parameters for optimization Few parameters for optimization Long, costly, and not very efficient Long, costly, and not very efficient 6/16 Slobodan Miletić Nikola

Existing solutions Destructive Algorithm Destructive Algorithm Starts with very big networks Starts with very big networks Gets results by pruning the initial network Gets results by pruning the initial network A lot of time is spent on unnecessary training A lot of time is spent on unnecessary training of big networks 7/16 Slobodan Miletić Nikola

Existing solutions Constructive Algorithm Constructive Algorithm Starts with a small neural network Starts with a small neural network Adds nodes and connections Adds nodes and connections Uses input/error rate to form new nodes Uses input/error rate to form new nodes Can miss optimal solution Can miss optimal solution 8/16 Slobodan Miletić Nikola

Proposed solution What’s better? What’s better? General algorithm – fitness function change enables generation General algorithm – fitness function change enables generation of different neural networks types Created neural networks outperform Created neural networks outperform neural network models designed by hand neural network models designed by hand Besides the best network, it creates several suboptimal networks Besides the best network, it creates several suboptimal networks that can also be used as a solutions 9/16 Slobodan Miletić Nikola

Proposed solution What’s new? What’s new? No similar general algorithm on the market No similar general algorithm on the market Original set of genetic algorithm mutations Original set of genetic algorithm mutations 10/16 Slobodan Miletić Nikola

Proposed solution What’s its future? What’s its future? Computer power and parallelism are increasing Computer power and parallelism are increasing which allows more complex neural networks Automated neural network generation algorithm Automated neural network generation algorithm like this will allow creation of complex neural networks 11/16 Slobodan Miletić Nikola

f 2 g 5 p 2 h 7 f 3 r 2 No changeAdd node a 4 b 3 c 7 d 2 q 5 g 5 h 7 f 3 a 4 b 3 d 2 f 2 c 7 p 2 r 2 f 2 p 2 r 2 g 5 h 7 f 3 c 7 a 4 b 3 d 2 f 3 g 5 a 4 b 3 c 7 d 2 f 2 g 2 p 2 h 7 f 3 r 2 h 7 r 2 a 4 b 3 c 7 f 2 p 2 d 2 Drop nodeActivation Function h 7 r 2 a 4 b 3 c 7 f 2 f 5 p 2 d 2 f 2 f 2 f 2 f 2 f 2 f 2 f 2 f 2 g 5 p 2 h 7 f 3 r 2 f 2 a 3 a 3 f 2 f 2 g 2 h 2 f 1 a 3 b 4 c 5 c 3 c 4 c 5 f 2 d 2 a 3 q 3 r 2 h 1 h 1 h 1 h 1 f 3 h 2 f 3 g 3 g 3 h 2 a 4 b 3 c 7 d 2 Solution details Number of Neurons a 4 b 1 c 7 d 2 a 4 b 3 c 7 d 2 3 f 2 g 5 p 2 h 7 f 3 r 2 Drop Connection a 4 b 3 c 7 d 2 Add Connection f 2 c 7 p 2 r 2 12/16 Slobodan Miletić Nikola 1 point Crossover 2 point Crossover

a 4 b 3 c 7 d 2 q 5 Solution details h 7 r 2 a 4 b 3 c 7 f 2 f 5 p 2 d 2 g 5 h 7 f 3 a 4 b 3 d 2 a 4 b 1 c 7 d 2 f 2 g 5 p 2 h 7 f 3 r 2 a 4 b 3 c 7 d 2 f 2 c 7 p 2 r 2 a 4 b 3 g 5 h 7 f 3 d 2 a 4 b 3 c 7 d 2 q 5 a 4 b 3 d 2 q 5 g 5 h 7 f 3 d 2 a 4 b 3 c 7 g 5 h 7 f 3 d 2 a 4 b 3 c 7 a 1 Add nodeNo change a 4 b 3 c 7 d 2 q 5 Drop connection a 4 b 3 c 7 d 2 q 5 g 5 a 4 b 3 h 7 f 3 d 2 a 4 b 3 c 7 d 2 q 5 h 7 f 3 d 2 a 4 g 5 a 4 b 3 b 3 c 7 d 2 q 5 a 4 b 3 c 7 d 2 q 5 g 5 h 7 f 3 a 4 b 3 d 2 g 5 h 7 b 3 f 3 a 4 d 2 a 4 c 7 d 2 b 3 q 5 a 4 c 7 d 2 g 5 h 7 b 3 f 3 a 4 d 2 b 3 q 5 13/16 Slobodan Miletić Nikola 1 point Crossover 2 point Crossover

f 3 a 4 d 2 b 3 q 5 g 5 h 7 f 3 d 2 a 4 b 3 c 7 a 1 f 3 a 4 d 2 b 3 q 5 Solution details a 4 b 3 c 7 d 2 q 5 a 4 b 3 c 7 d 2 q 5 a 4 c 7 d 2 g 5 h 7 b 3 f 3 a 4 d 2 b 3 q 5 h 7 f 3 d 2 a 4 g 5 a 4 b 3 b 3 c 7 d 2 q 5 a 4 b 3 d 2 q 5 g 5 h 7 f 3 d 2 a 4 b 3 c 7 a 1 g b Activation function f1 5 h 7 f 3 d 2 a 4 b 3 c 7 a 1 f 3 a 4 d 2 a 3 q 5 5 Number of Neurons g 10 h 7 f 3 d 2 a 4 b 3 c 7 a 1 g 5 a 4 d 2 f 3 b 3 q 5 a 1 f 3 b 3 q 5 h 7 f 3 d 2 a 4 b 3 c 7 a 4 d 2 g 5 Drop node f 3 b 3 q 5 h 7 f 3 d 2 a 4 b 3 c 7 Add connection a 4 d 2 g 5 h 7 f 3 d 2 a 4 b 3 c 7 a 1 Drop connection g 5 h 7 f 3 d 2 a 4 b 3 c 7 a 1 Add node f 3 a 4 d 2 b 3 q 5 q 5 Add connection f 3 a 4 d 2 b 3 q 5 q 5 14/16 Slobodan Miletić Nikola 2 point Crossover

Conclusion New way to create neural networks New way to create neural networks Results fully comparable Results fully comparable with hand designed networks Space for further improvement Space for further improvement More then one created network can be used More then one created network can be used 15/16 Slobodan Miletić Nikola

QUESTIONS? 16/16 Slobodan Miletić Nikola