1. Adaptive Cooperative Systems Chapter 8 Synaptic Plasticity 8.11 ~ 8.13 Summary by Byoung-Hee Kim Biointelligence Lab School of Computer Sci. & Eng. Seoul National University

2. (C) 2009 SNU CSE Biointelligence Lab Contents 8.11 Principal component neurons  Introductory remarks  Principal components and constrained optimization  Hebbian learning and synaptic constraints  Oja’s solution / Linsker’s model 8.12 Synaptic and phenomenological spin models  Phenomenological spin models  Synaptic models in the common input approximation 8.13 Objective function formulation of BCM theory  Projection pursuit  Objective function formulation of BCM theory 2

3. Goals and Contents Goal: the information-processing functions of model neurons in the visual system Contents  Principal component neurons  Special class of synaptic modification models  Relation to phenomenological spin models  Objective function formulation of BCM theory (C) 2009 SNU CSE Biointelligence Lab3

4. Introductory Remarks Images are highly organized spatial structures – some common statistical properties Development of the visual system is influenced by the statistical properties of the images  knowledge of the statistical properties of natural scenes ~ understanding the behavior of cells in the visual system (C) 2009 SNU CSE Biointelligence Lab4

5. Scale Invariance in Natural Images Studies of image statistics reveal non-preferrence of angular scale  Decimation procedure with the grey-valued pixels of the image assuming the role of the spins  p.d.f. of image constrasts and image gradients are unchanged  (Field 1987), (Ruderman and Bialek 1994), (Ruderman 1994) Representing the scale invariance through the covariance matrix  Gives a constraint on the form of the covariance matrix  Starting point for the PCA  (Hancock, et al. 1992), (Liu and Shouval 1995), (Liu and Shouval 1996) (C) 2009 SNU CSE Biointelligence Lab5

6. Principal Components We are rotating the coordinate system in order to find projections with desirable statistical properties Projections: maximally preserve information content while compressing the data into a few leading components (C) 2009 SNU CSE Biointelligence Lab6 Variance of the data projected onto the axis is maximal

7. Principal Components and Constrained Optimization (C) 2009 SNU CSE Biointelligence Lab7 n-component random vector correlation matrix If =0, then the covariance matrix Introducing a fixed vector that satisfies the normalization condition use this to help us find interesting projections Variance after operation: Optimization problem: find the vector a that satisfies the normalization condition, and maximizes the variance The variance is equal to the eigenvalue The maximum variance is given by the largest root

8. Hebbian Learning and Synaptic Constraints (C) 2009 SNU CSE Biointelligence Lab8 The simplest form Hebb’s rule for synaptic modification [Problem] Unstable. The synaptic weights would undergo unbounded growth c: output activity m: synaptic weight vector d: input activity vector On reaching a fixed point m is an eigenvector of the input correlation matrix with eigenvalue equal to zero

9. Solutions for the Unbounded Growth Problem Oja’s solution Linsker’s model (C) 2009 SNU CSE Biointelligence Lab9 On reaching a fixed point Results in a synaptic vector m for which the projection of the input activity has a maximum variance The synaptic system may be characterized as performing a principal component analysis of the input data constraint on the total synaptic strength Clipping - The sum of the synaptic weights are kept constant - each synaptic weight lies within a set range E Q : the variance in the input activity E k : constraint

10. Properties of the Linsker’s Model Stability corresponds to a global near minimum of the energy function Equivalent to the maximum in the input variance subject to the constraint Dynamics of the model system  In different regimes for the parameters k 1 and k 2, different receptive field structures dominate  As k 1 and k 2 are varied, particular eigenvectors other than the principal one gain in relative importance (C) 2009 SNU CSE Biointelligence Lab10

11. Synaptic and Phenomenological Spin Models Theory on synaptic modification  Model to explain the emergence of these highly ordered repeating structures (C) 2009 SNU CSE Biointelligence Lab11 Phenomena  Cells in the primate visual cortex self-organize onto ocular dominance columns and iso-orientation patches  The patterns observed experimentally are highly ordered

12. Phenomenological Spin Models 2D Ising lattice of eye-specificity encoding spints (Cowan and Friedman 1991)  Coupling strengths  If we take with, this type of coupling generates a short-range attraction plus a long-range repulsion between terminals from the same eye Hamiltonian for iso-orientation (C) 2009 SNU CSE Biointelligence Lab12

13. (C) 2009 SNU CSE Biointelligence Lab13

14. Synaptic Models in the Common Input Approximation Consider an LGN-cortico-cortico network with modifiable geniculocortico synapses and fixed cortico-cortico-connections Design of an energy function s.t. the fixed point of the network correspond to the minima of the energy function The common input model by Shouval and Cooper  hamiltonian in this model: (C) 2009 SNU CSE Biointelligence Lab14 general form correlational hamiltonian

15. Information-processing Activities by Common Input Neurons For exclusive excitatory connections  symmetry breaking does not occur  all receptive fields have the same orientation selectivity Inhibition  affects both the organization and structure of the receptive fields  If there is sufficient inhibition, the network will develop orientation selective receptive fields The cortical cells self-organize into iso-orientation patches with pinwheel singularities (C) 2009 SNU CSE Biointelligence Lab15

16. Objective Function Formulation of BCM Theory - Intro Distinguishment between information preservation (variance maximization) and classification (multimodality) (C) 2009 SNU CSE Biointelligence Lab16

17. Projection Pursuit Projection pursuit  a method for finding the most interesting low-dimensional features of high-dimensional data sets  The objective is to find orthogonal projections that reveal interesting structure in the data  PCA is a particular case with the proportion of total variance as the index of interestingness  Why is it needed? High-dimensional spaces are inherently sparse, or “curse of dimensionality” For classification purpose  Interesting projection is one that departs from normalcy (C) 2009 SNU CSE Biointelligence Lab17

18. Objective Function Formulation of BCM Theory (1/3) In the objective (energy) function formulation of BCM theory, a feature is associated with each projection direction A one-dimensional projection may be interpreted as a single feature extraction Goal: to find an objective (loss) function whose minimization produces a one-dimensional projection that is far from normal (C) 2009 SNU CSE Biointelligence Lab18

19. Objective Function Formulation of BCM Theory (2/3) (C) 2009 SNU CSE Biointelligence Lab19 Redefining the threshold function Synaptic modification functions How? Introduce a loss function With some assumptions

20. Objective Function Formulation of BCM Theory (3/3) (C) 2009 SNU CSE Biointelligence Lab20 The risk, or expected value of the loss, which is continously differentiable We are able to minimize the risk by means of gradient descent w.r.t. m i Slightly modified, deterministic version of the stochastic BCM modification equation A BCM neuron is extracting third-order statistical correlates of the data This would be a natural extension of principal component processing in the retina

21. Take-Home Message (Tomasso Poggio, NIPS 2007 tutorial) (C) 2009 SNU CSE Biointelligence Lab21