2. On Natural Scenes Analysis, Sparsity and Coding Efficiency Redwood Center for Theoretical Neuroscience University of California, Berkeley Mind, Brain & Computation Stanford University Vivienne Ming Adapted by J. McClelland for PDP class, March 1, 2013

3. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Two Proposals Natural Scene Analysis  Neural/cognitive computation can only be fully understood in “naturalistic” contexts Efficient (Sparse) Coding Theory  Neural computation should follow information theoretic principles

4. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Classical Physiology

5. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Classical Physiology +

6. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Classical Physiology +

7. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Reverse Correlation Jones and Palmer (1987)

8. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Limits of Classical Physiology Assumes units (neurons) are linear  so known nonlinearities are "added on" to the models Contrast sensitivity “Non-classical receptive fields” Two-tone inhibition ETC. Assumes that units operate independently  activity of one cell doesn't depend on the activity of others  i.e., characterizing cell-by-cell equivalent to characterizing the whole population  of evolution and development, drifting gratings and white noise are very "unnatural“  Is it possible that our sensory systems are functionally adapted to the statistics of “natural” (evolutionarily relevant) signals?  Would this adaptation affect our characterization of cells?

9. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Response to Natural Movie Classical Receptive Field Response Response in “Context”

10. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Limits of Classical Physiology Assumes units (neurons) are linear  so known nonlinearities are "added on" to the models Contrast sensitivity “Non-classical receptive fields” Two-tone inhibition ETC. Assumes that units operate independently  activity of one cell doesn't depend on the activity of others  i.e., characterizing cell-by-cell equivalent to characterizing the whole population Finally, in terms of evolution and development, drifting gratings and white noise seem very "unnatural“  Is it possible that our sensory systems are functionally adapted to the statistics of “natural” (evolutionarily relevant) signals?  Would this adaptation affect our characterization of cells?  How can we test this?

11. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Efficient Coding Theory Barlow (1961); Attneave (1954) Natural images are redundant  Statistical dependencies amongst pixel values in space and time An efficient visual system should reduce redundancy  Removing statistical dependencies

12. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Information Theory Shannon (1949) Optimally efficient codes reflect the statistics of target signals

13. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Natural Scenes Analysis: First-Order Statistics Naïve Models

14. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Natural Scenes Analysis: First-Order Statistics Histogram Equalization Intensity Histogram

15. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Natural Scenes Analysis: Second-Order Statistics

16. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Natural Scenes Analysis: Second-Order Statistics

17. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Natural Scenes Analysis: Second-Order Statistics

18. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Spatial Correlations Compare intensity at this pixel To the intensity at this neighbor

19. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Spatial Correlations

20. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. The Ubiquitous. Flat (White) Power Spectrum

21. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Example: synthetic 1 / f signals

22. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Natural Scenes Analysis: Principal Components Analysis PCARotationWhitening Information theory says this is an ideal code. No redundancy

23. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. PCA vs. Center Surround

24. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Natural Scenes Analysis: Higher-Order Statistics PCARotationWhitening Principle dimensions of variation don’t align with data’s intrinsic structure

25. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Natural Scenes Analysis: Higher-Order Statistics Need a more powerful learning algorithm Independent Component Analysis (ICA)

26. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Which are the independent components in the scene below?

27. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. +_______ +=

28. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. The Model x = s  + n Overcomplete: #(s) >> #(x) Factorial: p(s) =  i p(s i ) Sparse: p(s i ) = exp(g(s i ))  Where g(.) is some non-Gaussian distribution e.g., Laplacian: g(s) = −|s|  e.g., Cauchy: g(s) = −log(2 + s 2 ) The noise is assumed to be additive Gaussian  n ~ N(0,  2 I) Goal: find dictionary of functions, , such that coefficients, s, are as sparse and statistically independent as possible Information Theory demands sparseness

29. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Learning log likelihood L(  ) = Learning rule: Basically the delta rule:  =  (x −  s)s T  Impose constraint to encourage the variances of each s to be approximately equal to prevent trivial solutions Usually whiten the inputs before learning  Forces network to find structure beyond second-order  Increases stability

30. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Sparsity

31. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. ?

32. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Efficient Auditory Coding Smith & Lewicki (2006) Extend Olshausen (2002) to deal with time-varying signals  e.g., sounds or movies Train the network on “Natural” sounds  Environmental Transients  Environmental Ambients  Animal Vocalizations

33. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D.

34. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Cat ANF Revcor Filters

35. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Efficient Kernels

36. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Population Coding

37. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Population Coding

38. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Population Coding

39. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Population Coding

40. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Population Coding

41. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Speech

42. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Speech

43. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Speech

44. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Efficient Coding Literature Empirical  Weliky, Fiser, Hunt & Wagner (2003)  Vinje & Gallant (2002)  DeWeese, Wehr & Zador (2003)  Laurent (2002)  Theunissen (2003) Theoretical  Field (1987)  van Hateren (1992)  Simoncelli & Olshausen (2001)  Olshausen & Field (1996)  Bell & Sejnowski (1997)  Hyvarinen & Hoyer (2000)  Smith & Lewicki (2006)  Doi & Lewicki (2006)

45. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Hierarchical Structure? Can we identify interesting structure in the world by looking at higher order statistics of the activations of the linear features discovered by the first-order model?  Karklin and Lewicki (2005) looked for patterns at the level of the variances of the linear features.  Karklin and Lewicki (2009) looked for patterns at the level of the covariances of the linear features.

46. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Looking at Hierarchical Structure

47. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Looking at Hierarchical Structure

48. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Looking at Hierarchical Structure

49. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Looking at Hierarchical Structure

50. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Looking at Hierarchical Structure

51. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Generalizing the standard ICA model

52. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Generalizing the standard ICA model Instead of: we now have units u and v such that

53. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Independent density components

54. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Karklin & Lewicki (2009) The model tries to find the values of the y j ’s that lead to a combined covariance matrix C that matches the covariance of the data represented by activities across first-level filters. The learning process involves a search for vectors b k and weights w jk that allow the model to fit the data while keeping the y j ’s sparse and independent.

55. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D.

56. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D.

57. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D.

58. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Responses of Cell to Gratings

59. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D.

60. Rev jlm 3/5/2010Natural Scenes Analysis Vivienne Ming, Ph.D. Efficient Coding Summary StatisticComputation Algorithm Example Biological Example Reference 1 st -order Contrast gain control Histogram equalization Retina or H1 adaptation Fairhall et al. (2001) 2 nd -orderWhiteningPCA Retinal/ Thalamic coding Atick (1992) Higher-order Sparse Coding ICA / Sparsenet V1 coding Olshausen & Field (1996) Time-varying Shift- invariance Efficient Spike Coding Cochlear coding Smith & Lewicki 2006 Hierarchical Conditional Independence Hierarchical coding ? Karklin & Lewicki ’05,’09