1. Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009

2. What does neural activity represent? In Bayesian models: probabilities Direction of motion: single, static variable Accumulated evidence in area LIP Shadlen and Newsome (2001)

3. What does neural activity represent? In Bayesian models: probabilities Direction of motion: single, static variable What about multi-dimensional, dynamic quantities? Accumulated evidence in area LIP Shadlen and Newsome (2001)

4. Foveal vision and fixational drift

5. - between micro-saccades - ~20 receptive fields Image from: X. Pitkow - between spikes (100 Hz) - ~2-4 receptive fields ! Fixational drift is large in the fovea: cone separation: 0.5 arcmin

6. Foveal vision and fixational drift - between micro-saccades - ~20 receptive fields Image from: X. Pitkow - between spikes (100 Hz) - ~2-4 receptive fields ! Downstream areas require knowledge of trajectory to interpret spikes Fixational drift is large in the fovea: cone separation: 0.5 arcmin

7. Joint decoding of image and position Bayesian: Discrimination task: vs. X. Pitkow et al, Plos Biology (2007) N x 2 probabilities # positions

8. Bayesian: Discrimination task: vs. X. Pitkow et al, Plos Biology (2007) N x 2 probabilities Unconstrained image 30 x 30 binary pixels # positions N x 2 900 probabilities Joint decoding of image and position

9. Bayesian: Discrimination task: vs. X. Pitkow et al, Plos Biology (2007) N x 2 probabilities Unconstrained image 30 x 30 binary pixels # positions N x 2 900 probabilities Can the brain apply a Bayesian approach to this problem? Joint decoding of image and position

10. Can the brain apply a Bayesian approach to this problem? Decoding strategy Performance in parameter space What are the biological implications?

11. Can the brain apply a Bayesian approach to this problem? Decoding strategy Performance in parameter space What are the biological implications?

12. Decoding strategy Discards information about correlations Factorized representation:

13. Decoding strategy Discards information about correlations minimize D KL Factorized representation: Exact if trajectory is known. evidence, diffusion Update dynamics:

14. Decoding strategy Discards information about correlations minimize D KL Factorized representation: Exact if trajectory is known. evidence, diffusion evidence - Poisson spiking (rate λ 1 for on pixels, λ 0 for off) diffusion - Random walk (diffusion coefficient D) Retinal encoding model: Update dynamics:

15. Decoding strategy Discards information about correlations Neural Implementation - Two populations: where, what For 30 x 30 pixels: N × 2 900 → N + 900 quantities. Factorized representation:

16. Update rules Update of what neurons: multiplicative gating Ganglion cells What Where nonlinearity

17. Update rules Update of what neurons: Update of where neurons: multiplicative gating Ganglion cells What Where What multiplicative gating Ganglion cells + diffusion nonlinearity

18. Demo image retina m x m binary pixels 2d diffusion (D) Poisson spikes: 100 Hz (on), 10 Hz (off) Decoder

19. Demo

20. Decoding strategy Performance in parameter space What are the biological implications? Can the brain apply a Bayesian approach to this problem?

21. Performance DD Convergence time [s] accuracy Performance degrades with larger D (and smaller λ)

22. Performance DD Convergence time [s] Faster and more accurate for larger images m = 5, 10, 30, 50, 100 accuracy

23. Demo

24. Performance DD Convergence time [s] Faster and more accurate for larger images accuracy m = 5, 10, 30, 50, 100

25. Performance DD Convergence time [s] Faster and more accurate for larger images accuracy m = 5, 10, 30, 50, 100

26. Performance DD Convergence time [s] Faster and more accurate for larger images accuracy m = 5, 10, 30, 50, 100

27. Performance D/m Convergence time [s] accuracy scales with linear image size m m x m pixels

28. Performance D/m Convergence time [s] accuracy scales with linear image size m Analytical scaling: D* m x m pixels

29. Performance Performance improves with image size. Success for images 10 x 10 or larger Prediction for psychophysics: Degradation in high acuity tasks when visual scene contains little background detail.

30. Temporal response of Ganglion cells Common view: fixational motion important to activate cells, due to biphasic response f(t) t Temporal response makes decoding much more difficult. 50 ms Need history Non-Markovian:

31. Temporal response of Ganglion cells Approach: Choose decoder that is Bayes optimal if the trajectory is known. What Ganglion “filtered trajectory” Where history dependent decoder / naive decoder Convergence time [s] accuracy D D

32. Temporal response of Ganglion cells Is fixational motion beneficial? Known trajectory, perfect inhibitory balance Convergence time [s] D Optimal D - order of magnitude smaller than biological value

33. Can the brain apply a Bayesian approach to this problem? Decoding strategy Performance in parameter space What are the biological implications?

34. Network architecture Each ganglion cell innervates multiple what & where cells (spread: ~10 arcmin) WhereWhat Ganglion Reciprocal, multiplicative gating

35. Activity: What neurons Slow dynamics, evidence accumulation Where neurons Fewer. Highly dynamic activity Tonic, sparse in retinal stabilization conditions.

36. Activity: What neurons Slow dynamics, evidence accumulation Where neurons Fewer. Highly dynamic activity Tonic, sparse in retinal stabilization conditions. Where in the brain? Monocular LGN? V1? If so, suggests LGN or V1 Modulatory inputs to relay cells (gating?) Lateral connectivity in where network, Increase in number of neurons.

37. Summary Strategy for stabilization of foveal vision Factorized Bayesian approach to multi-dimensional inference

38. Summary Strategy for stabilization of foveal vision Explicit representation of stabilized image “What” and “where” populations Factorized Bayesian approach to multi-dimensional inference

39. Summary Strategy for stabilization of foveal vision Explicit representation of stabilized image “What” and “where” populations Good performance at 1 arcmin resolution Problem is easier for large images, for coarser reconstruction Factorized Bayesian approach to multi-dimensional inference

40. Summary Strategy for stabilization of foveal vision Explicit representation of stabilized image “What” and “where” populations Good performance at 1 arcmin resolution Problem is easier for large images, for coarser reconstruction Factorized Bayesian approach to multi-dimensional inference Network architecture: Many-to-one inputs from retina, multiplicative gating (what/where)

41. Uri Rokni Haim Sompolinsky Markus Meister Special thanks - the Swartz foundation Acknowledgments