1. Mutation Operator Evolution for EA-Based Neural Networks By Ryan Meuth

2. Reinforcement Learning State Action Reward Environment Agent State Value Estimate Action Policy

3. Reinforcement Learning Good for On-Line learning where little is known about environment Easy to Implement in Discrete Environments Value estimate can be stored for each state Value estimate can be stored for each state In infinite time, optimal policy guaranteed. In infinite time, optimal policy guaranteed. Hard to Implement in Continuous Environments Infinite States! Must estimate Value Function. Infinite States! Must estimate Value Function. Neural Networks Can be used for function approximation. Neural Networks Can be used for function approximation.

4. Neural Network Overview Feed Forward Neural Network Based on biological theories of neuron operation Based on biological theories of neuron operation

5. Feed-Forward Neural Network

6. Recurrent Neural Network

7. Neural Network Overview Traditionally used with Error Back- Propagation BP uses Samples to Generalize to Problem BP uses Samples to Generalize to Problem Few “Unsupervised” Learning Methods Few “Unsupervised” Learning Methods Problems with No Samples: On-Line Learning Conjugate Reinforcement Back Propagation

8. EA-NN Both Supervised and Unsupervised Learning Method. Uses weight set as genome of individual Fitness Function is Mean-Squared Error over target function. Mutation Operator is a sample from a Gaussian Distribution. Possible that mutation operator might not be best. Possible that mutation operator might not be best.

9. Uh… Why? Could improve EA-NN efficiency Faster Online Learning Faster Online Learning Revamped tool for Reinforcment Learning Revamped tool for Reinforcment Learning Smarter Robots. Smarter Robots. Why Use an EA? Knowledge – Independent Knowledge – Independent

10. Experimental Implementation First Tier – Genetic Programming Individual is Parse-tree representing Mutation operator Individual is Parse-tree representing Mutation operator Fitness is Inverse of sum of MSE’s from EA Testbed Fitness is Inverse of sum of MSE’s from EA Testbed Second Tier – EA Testbed 4 EA’s, spanning 2 classes of problems 4 EA’s, spanning 2 classes of problems 2 Feed-Forward Non-Linear Approximations 2 Feed-Forward Non-Linear Approximations 1 High-Order, 1 Low-Order 1 High-Order, 1 Low-Order 2 Recurrent Time Series Predictions 2 Recurrent Time Series Predictions 1 Will be Time-Delayed, 1 Not Time-Delayed

11. GP Implementation Functional Set: {+,-,*,/} Terminal Set: Weight to be Modified Weight to be Modified Random Constant Random Constant Uniform Random Variable Uniform Random Variable Over-Selection: 80% of Parents from top 32% Rank-Based Survival Initialized by Grow Method (Max Depth of 8) Fitness: 1000/(AvgMSE) – num_nodes P(Recomb) = 0.5; P(Mutation) = 0.5; Repair Function 5 runs, 100 generations each. Steady State: Population of 1000 individuals, 20 children per generation.

12. EA-NN Implementation Recombination: Multi-Point Crossover Mutation: Provided by GP Fitness: MSE over test function (minimize) P(Recomb) = 0.5; P(Mutation) = 0.5; Non-Generational: Population of 10 individuals, 10 children per generation 50 Runs of 50 Generations.

13. Results This is where results would go. Single Uniform Random Variable: ~380 Observed Individuals: ~600 Improvement! Just have to Wait and See…

14. Conclusions I don’t know anything yet.

15. Questions? Thank You!