1. Added After Talk
Looking for a review paper on evolving plastic networks? Here is a recent one from Andrea Soltoggio, Sebastian Risi, and Kenneth Stanley:

2. Evolving to Learn through Synaptic Plasticity
Kenneth O. Stanley
Uber AI Labs
And
Evolutionary Complexity Research Group,
Department of Computer Science,
University of Central Florida

3. Evolution and Plasticity
Brains in nature are evolved
But not static: brains learn over their lifetime
If neuroevolution is about solving a problem, static might be okay
If neuroevolution is about evolving a brain
Plasticity may be essential
(Though recurrence alone is also a theoretical option)

4. Evolution Can Discover the Delta Rule (1990)…
Chalmers, David J. "The evolution of learning: An experiment in genetic connectionism."
In Proceedings of the 1990 connectionist models summer school, pp San Mateo, CA, 1990.

5. But “Local Learning Rules” Are Perhaps More Interesting
The delta rule and backprop are already known
Not entirely clear whether backprop is biologically plausible
Biggest one: Domain specific learning rules are likely more efficient though less generic

6. But “Local Learning Rules” Are Perhaps More Interesting
The delta rule and backprop are already known
Not entirely clear whether backprop is biologically plausible
Biggest one: Domain specific learning rules are likely more efficient though less generic
Hinton 2014 slide on backprop in the cortex

7. But “Local Learning Rules” Are Perhaps More Interesting
The delta rule and backprop are already known
Not entirely clear whether backprop is biologically plausible
Biggest one: Domain specific learning rules are likely more efficient though less generic

8. What Can Happen at a Synapse?
(from Dr. George Johnson at )

9. What Can Happen at a Synapse?
Weighted signal transmission
But also:
Strengthening
Weakening
Sensitization
Habituation
Hebbian learning
Neuromodulation (Soltoggio et. al 2008)

10. How Should Weights Change? (Blynel and Floreano 2002)
Plain Hebb Rule:
Postsynaptic rule:
Weakens synapse if postsynaptic node fires alone
Presynaptic rule:
Covariance rule:
Strengthens when correlated, weakens when not

11. Floreano’s Genetic Encoding

12. Experiment: Light-switching
Fully Recurrent Network
Task: Go to black area to turn on light, then go to area under light
Requires a policy change in mid-task: Reconfigure weights for new policy
Blynel, J. and Floreano, D. (2002) Levels of Dynamics and Adaptive Behavior in Evolutionary Neural Controllers. In B. Hallam, D. Floreano, J. Hallam, G. Hayes, and J.-A. Meyer, editors. From Animals to Animats 7: Proceedings of the Seventh International Conference on Simulation on Adaptive Behavior, MIT Press.

13. Results
Adaptive synapse networks evolved straighter and faster trajectories
Rapid and appropriate weight modifications occur at the moment of change
However, other early experiments (e.g. dangerous food foraging with NEAT) showed recurrence alone doing better
Still, almost surely plasticity matters

14. Soltoggio et al. (2008): Neuromodulated Plasticity
Regular activation
Modulatory activation
Plasticity term
(can be any learning rule)
Weight change
Advantage: Knowing when to change
Magnitude of change modulated by external neuron
Moving towards RL-like capabilities

15. Neuromodulation Experiment: T-maze
Double T-maze
Position of reward can change across trials
Modulatory plastic networks perform better than simple plastic networks on harder T-maze
Memory “lock-in” happens

16. Yes, given some unusual ingredients:
Interesting New Idea (Soltoggio and Stanley 2012): Reconfigure-and-saturate Hebbian Plasticity
Is it possible to make a Hebbian network learn new behaviors based on reward or penalty signals?
Yes, given some unusual ingredients:
A modulation signal represents the reward
Weight saturation
Neural noise
Stronger weight (from random start) will win
Positive modulation yields Hebbian plasticity,
But negative modulation yields anti-Hebbian (decreases strengh of pathway by reducing weight difference)

17. Reconfigure-and-saturate Dynamics
Neural Noise determines the winner during Hebbian phases
Insight: Noise is driving exploration
Saturation allows stability

18. Result: R&S Intelligent Navigation Experiment
Learns over 2,000 timesteps to navigate intelligently
Relearns after
reward switch
What’s next: Temporal association learning through eligibility traces

19. Plasticity through HyperNEAT
Adaptive HyperNEAT (Risi and Stanley 2012): Indirect encoding compactly generates pattern of rules across NN
Imagine a pattern of rules spread across the brain as complex as this picture

20. And See Tomorrow: Backpropagated plasticity: learning to learn with gradient descent in large plastic neural networks Thomas Miconi, Jeff Clune, Kenneth Stanley
At the Workshop on Meta-Learning on Saturday: Poster Spotlights start 9:40am
Idea: Plasticity parameters optimized by gradient descent between “lifetimes”
Weights then adjusted according to plasticity rules within life
Results: Plastic networks with millions of parameters become better at image reconstruction task than plain RNN or LSTM

21. And See Tomorrow: Backpropagated plasticity: learning to learn with gradient descent in large plastic neural networks Thomas Miconi, Jeff Clune, Kenneth Stanley
At the Workshop on Meta-Learning on Saturday: Poster Spotlights start 9:40am
Learned plasticity coefficients

22. Conclusion Wide scope for creativity
Endless kinds of plasticity can be evolved
Domain-specific learning mechanisms are less studied than domain-general
Could be important in some domains, e.g. learning to walk quickly on new terrain or in new gravity
Cross-pollination between NE and DL

23. More information My Homepage: http://www.cs.ucf.edu/~kstanley
Uber AI Labs:
Evolutionary Complexity Research Group: