1. The Neuronal Replicator Hypothesis Chrisantha Fernando & Eors Szathmary CUNY, December 2009 1 Collegium Budapest (Institute for Advanced Study), Budapest, Hungary 2 Centre for Computational Neuroscience and Robotics, Sussex University, UK 3 MRC National Institute for Medical Research, Mill Hill, London, UK 4 Parmenides Foundation, Kardinal-Faulhaber-Strase 14a, D-80333 Munich, Germany 5 Institute of Biology, Eötvös University, Pázmány Péter sétány 1/c, H-1117 Budapest, Hungary

2. Visiting Fellow MRC National Institute for Medical Research London Post-Doc Center for Computational Neuroscience and Robotics Sussex University Marie Curie Fellow Collegium Budapest (Institute for Advanced Study) Hungary

3. The Hypothesis Evolution by natural selection takes place in the brain at rapid timescales and contributes to solving cognitive/behavioural search problems. Our background is in evolutionary biology/the origin of non-enzymatic template replication/evolutionary robotics/computational neuroscience.

4. Outline Limitations of some proposed search algorithms, e.g. Reward biased stochastic search Reinforcement Learning How copying/replication of neuronal data structures can alleviate these limitations.

5. Mechanisms of neuronal replication Applications and future work

6. Simple Search Tasks Behavioural and neuropsychological learning tasks can be solved by stochastic-hill climbing Stroop Task Wisconsin Card Sorting Task (WCST) Instrumental Conditioning in Spiking Neural Networks Simple inverse kinematics problem

7. Stochastic Hill- Climbing Initially P(x i = 1) = 0.5, Initial reward = 0 Make random change to P Generate M examples of binary strings Calculate reward If r(t) > r(t-1), keep changes of P, else revert to previous P values. One solution, change solution, keep good changes, loose bad changes. 0.5 0.80.5 0.40.5

8. Can get stuck on local optima

9. Stroop Task Green Red Blue Purple Blue Purple Blue Purple Red Green Purple Green Name the colour of the words.

10. Dehaene et al, 1998 dW = Reward x pre x post Decreased reward -> Instability in workspace

11. WCST Each card has several “features”. Subjects must sort cards according to a feature (color, number, shape, size).

12. Rougier et al 2005. PFC weights stabilised if expected reward obtained, destabilised if expected reward not obtained, i.e. TD learning

13. Instrumental Conditioning

14. In a spiking neural net Izhikevich 2007 Simple spiking model Random connections STDP Delayed reward Eligibility traces Synapse selected

15. Simple spiking model

16. STDP

19. Time t pre

20. Time t post

21. Interval = T post - T pre

22. Time t post

26. Time t pre

27. Interval = T post - T pre

29. A simple 2D inverse kinematics problem

30. Reinforcement Learning For large problems a tabular representation of state- action pairs is not possible. How does compression of state representation occur? Function approximation Domain-specific knowledge provided by the designer, e.g. TD-Gammon was dependent on Tesauro’s skillful design of a non-linear multilayered neural network, used for value function approximation in the Backgammon domain consisting of approximately 10 20 states” p20 [51].

31. So far… SHC works on simple problems RL is a sophisticated kind of SHC In order for RL/SHC to work, action/value representations must fit the problem domain. RL doesn’t explain how appropriate data-structures/representations arise.

32. Large search space so random search or exhaustive search not possible. Representation critical local optima. Requires internal sub-goals, no explicit reward. What neural mechanisms underlie complex search?

33. What is natural selection? Some hereditary traits affect survival and/or fertility 1.multiplication 2.heredity 3.variability

34. Natural selection reinvented itself

35. Evolutionary Computation Solving problems by EC also requires decisions about genetic representations And about fitness functions For example, we use EC to solve the 10 coins problem

37. Fitness function Convolution of desired inverted triangle over grid Instant fitness = number of coins occupying he inverted triangle template An important question is how such fitness functions (subgoals/goals) could themselves be bootstrapped in cognition.

39. Michael Ollinger, Parmenides Foundation, Munich

40. Structuring Phenotypic Variation Natural Selection can act on genetic representations variability properties (genetic operators, e.g mutation rates)

41. A Variation in Variability Improvement of representations for free…

42. B

43. Non-trivial Neutrality g1g1 g2g2 p ed 1 ed 2 Adapted from Toussaint 2003

44. Population Search Natural selection allows redistribution of search resources between multiple solutions. We propose that multiple (possibly interacting) solutions to a search problem exist at the same time in the neuronal substrate.

45. A A B C D A A B C D ABCD ABCD

46. ABCD A D’D’’ D’’’ D C D A B ABCD A A B C D ABCD D’D’’D’’’D Waste

47. Can units of selection exist in the brain? We propose 3 possible mechanisms Copying of connectivity patterns Copying of bistable activity patterns Copying of spatio-temporal spike patterns & explicit rules

48. Copying of connectivity patterns

49. How to copy small neuronal circuits DNA neuronal network

50. STDP and causal inference

52. With error correction and sparse activation

53. 1 + 1 Evolution Stratergy

54. Copying of bistable activity patterns

56. 1 bit copy

57. Hebbian Learning can Structure Exploration Distributions

66. - Search in biased towards previous local optima

70. The Origin of Heredity in Neuronal Networks. Phenotype 2 Phenotype 1 M2M2 M1M1 C Genotype 1 Genotype 2 CM 2 = M 1 C = M 2 -1 M 1

71. Non-local, e.g. requires ATA

72. Stochastic hill climbing can select for neuronal template replication M2M2 M1M1 C Genotype 1 Genotype 2 E E Error

74. Copying of Spatiotemporal Spike Patterns & Explicit Rules

75. Spatiotemporal spike patterns

80. ABA vs ABB DD vs DS Visual shift-invariance mechanisms applied to linguistics.

81. APPLICATIONS

82. Evolution of Predictors (Feed-forward Models/Emulators/Bayesian Causal Networks). First derivative of predictability Evolution of Linguistic Construction Evolution of controllers for robot hand- manipulation Evolution of Productions in ACT- R/Copycat Evolution of representations and search for insight problem solving.

84. Operations to construct a BN Larranaga et al, 1996. Structure Learning of Bayesian Networks by Genetic Algorithms. Kemp & Tenenbaum, 2008. The discovery of structural form.

85. Luc Steels et al, Sony Labs

86. Istvan Zacher Collegium Budapest (Institute for Advanced Study)

87. K(v) S(p)C(p) 0 1 KSC K01 S C

88. K(v) S(p)C(p) 0 1 KSC K01 S C Rules

89. K(v) S(p)C(p) 0 1 KSC K01 S C Rules

90. K(v) S(p)C(p) 0 1 KSC K01 S C Rules

91. K(v) S(p)C(p) 0 1 KSC K01 S C Rules KC

92. K(v) S(p)C(p) 0 1 KSC K01 S C Rules KC S

93. KSC K S C Rules KC S

94. KSC K01 S C Rules KC S K(v) S(p)C(p) 0 1

95. Helge Ritter, Bielefeld, Germany

96. Thanks to Richard Goldstein Richard Watson Dan Bush Eugine Izhikevich Phil Husbands Luc Steels K.K. Karishma Anna Fedor, Zoltan Szatmary, Szabolcs Szamado, Istvan Zachar Anil Seth