1. 16.899A: Physiology (contd) Lavanya Sharan January 24 th, 2011

2. Before we start, a few caveats A lot is not known about how the human visual system works. We (Alyosha + Lavanya) don’t know a lot about physiology. But, before you worry, a few lines from Marr…

3. Slide source: Nancy Kanwisher & Jim DiCarlo

4. We care about ‘big picture’ In this class, we are interested in the underlying software/algorithm/computations Not in specifics of the `particular hardware’ Want back pocket models for various components of the human visual system – Very few of these exist. Our closest cousins: computational neuroscientists/cognitive scientists/psychophysicists

5. Overview of `particular hardware’ Retina, LGN What are visual areas? Tools for studying human visual system Area V1 Beyond area V1 What and where pathways Summary of what (and how little) we know

6. Primary Visual Pathway 1.Retina 2.Thalamus –Lateral Geniculate Nucleus (LGN) divided into magno and parvo layers 3.Primary visual cortex (V1) 4.Extrastriate visual areas Each visual hemifield projects to the opposite hemisphere Slide source: Jody Culham

7. 7 Slide source: Nancy Kanwisher & Jim DiCarlo

9. Primary Visual Pathway 1.Retina 2.Thalamus –Lateral Geniculate Nucleus (LGN) divided into magno and parvo layers 3.Primary visual cortex (V1) 4.Extrastriate visual areas Each visual hemifield projects to the opposite hemisphere Slide source: Jody Culham

10. What is a Visual Area? 1.Function –an area has a unique pattern of responses to different stimuli 2.Architecture –different brain areas show differences between cortical properties (e.g., thickness of different layers, sensitivity to various dyes) 3.Connectivity –Different areas have different patterns of connections with other areas 4.Topography –many sensory areas show topography (retinotopy, somatotopy, tonotopy) –boundaries between topographic maps can indicate boundaries between areas (e.g., separate maps of visual space in visual areas V1 and V2 Slide source: Jody Culham

11. Why are there so many visual areas? Source: Felleman & Van Essen, 1991Source: Mapping the MInd cover image MAGNO quick and dirty PARVO slow and detailed Slide source: Jody Culham

12. More brain, more visual areas Slide source: Jody Culham

13. Why not one really big visual area? V1 Slide source: Jody Culham

14. Why not a really big visual area? As areas become larger, longer interconnections are required Limits on cortical thickness and connections may constrain max area size Slide source: Jody Culham

15. Parallel processing is more efficient Teach neural network to identify “what” and “where” One neural network with 18 nodes (~neurons) devoted to both tasks versus One neural networks with two streams of 9 nodes each (total = 18) After 300 training trials, the two stream model outperformed the single-system model Rueckl, Cave & Kosslyn, 1989 Slide source: Jody Culham

16. Different Tasks Require Different Information different regions may need to use different coding systems ventral stream: object-centred dorsal stream: viewer-centred Slide source: Jody Culham

17. Wiring Constraints Source: Van Essen, 1997 David Van Essen proposes that as the brain develops, areas that are richly interconnected will be pulled together to form a gyrus (and those that are weakly interconnected form sulci). Slide source: Jody Culham

18. Sulcal Formation: V1-V2 Source: Van Essen, 1997 The V1/V2 border provides one example of two richly interconnected areas that form a gyrus. This arrangement also explains why maps in V1 and V2 are mirror images of each other! calcarine sulcus Slide source: Jody Culham

19. Optimized Connections Multidimensional Scaling strength of connections can be used to infer spatial layout expected layout of visual areas matches anatomy amazingly well Occipital Parietal Temporal Malcolm Young Slide source: Jody Culham

20. Tools for mapping human areas Neuropsychological Lesions Temporary Disruption transcranial magnetic stimulation (TMS) Electrical and magnetic signals electroencephalography (EEG) magnetoencephalography (MEG) Brain Imaging positron emission tomography (PET) functional magnetic resonance imaging (fMRI) Slide source: Jody Culham

21. Slide source: Nancy Kanwisher & Jim DiCarlo

22. Overview of `particular hardware’ Retina, LGN What are visual areas? Tools for studying human visual system Area V1 Beyond area V1 What and where pathways Summary of what (and how little) we know

23. Cortical Receptive Fields Single-cell recording from visual cortex David Hubel & Thorston Wiesel © Stephen E. Palmer, 2002

24. Cortical Receptive Fields Single-cell recording from visual cortex © Stephen E. Palmer, 2002

25. Cortical Receptive Fields Three classes of cells in V1 Simple cells Complex cells Hypercomplex cells © Stephen E. Palmer, 2002

26. Cortical Receptive Fields Simple Cells: “Line Detectors” © Stephen E. Palmer, 2002

27. Cortical Receptive Fields Simple Cells: “Edge Detectors” © Stephen E. Palmer, 2002

28. Cortical Receptive Fields Constructing a line detector © Stephen E. Palmer, 2002

29. Cortical Receptive Fields Complex Cells 0o0o © Stephen E. Palmer, 2002

30. Cortical Receptive Fields Complex Cells 60 o © Stephen E. Palmer, 2002

31. Cortical Receptive Fields Complex Cells 90 o © Stephen E. Palmer, 2002

32. Cortical Receptive Fields Complex Cells 120 o © Stephen E. Palmer, 2002

33. Cortical Receptive Fields Constructing a Complex Cell © Stephen E. Palmer, 2002

34. Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002

35. Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002

36. Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002

37. Cortical Receptive Fields Hypercomplex Cells “End-stopped” Cells © Stephen E. Palmer, 2002

38. Cortical Receptive Fields “End-stopped” Simple Cells © Stephen E. Palmer, 2002

39. Cortical Receptive Fields Constructing a Hypercomplex Cell © Stephen E. Palmer, 2002

40. Overview of `particular hardware’ Retina, LGN What are visual areas? Tools for studying human visual system Area V1 Beyond area V1 What and where pathways Summary of what (and how little) we know

41. Logothetis 1999; from http://psychology.uwo.ca/fMRI4Newbies/RetinotopicandEarlyVisualAreas.html Overview of visual areas

42. Macaque & human visual areas are similar Tootell et al. 2003; from http://psychology.uwo.ca/fMRI4Newbies/RetinotopicandEarlyVisualAreas.html

43. Slide source: Nancy Kanwisher & Jim DiCarlo

44. Retinotopy (Tootell et al. 1982) Adjacent parts of visual field are mapped to adjacent parts of cortex. Not all visual areas have retinotopy, may be graded. Slide source: Nancy Kanwisher & Jim DiCarlo

45. Slide source: Nancy Kanwisher, Jim DiCarlo, David Heeger

47. Why edges? So, why “edge-like” structures in the Plenoptic Function?

48. Two visual pathways The two visual processing streams for different visual percepts: “What” (ventral stream)- object recognition main input from “slow and detailed” parvo system “Where” or “How” (dorsal stream) - spatial perception, motor planning main input from “quick and dirty” magno system Source: Mishkin & Ungerleider, 1982 Slide source: Jody Culham

49. Two visual pathways The two visual processing streams for different visual percepts: “What” (ventral stream)- object recognition main input from “slow and detailed” parvo system “Where” or “How” (dorsal stream) - spatial perception, motor planning main input from “quick and dirty” magno system Source: Mishkin & Ungerleider, 1982 Slide source: Jody Culham

50. The “What” Pathway body motionfacesplacesbodiesobjects Other Visual Areas contain more complex receptive fields Temporal Lobe contains many specialized areas for recognizing various things Slide source: Jody Culham

51. The “Where” or “How” Pathway eye movements grasping and reaching motion perception Parietal Lobe contains many specialized areas for using vision to guide actions in space head movements attention Slide source: Jody Culham

52. Slide source: Nancy Kanwisher & Jim DiCarlo

53. Summary Low-level areas Filter banks, SIFT, HOG for color, orientation, spatial frequencies, motion… High-level areas Desired output from computer vision systems e.g., segmentation, robust object/scene/texture recognition, motion understanding and planning… Middle-level area Where the magic happens – No one (neuroscientists, psychologists, computer scientists, etc.) really understands this stage of processing. For more, come find us for pointers to papers/books/readings and people to talk to.