1. Computational Theories & Low-level Pixels To Percepts A. Efros, CMU, Spring 2009

2. Four Stages of Visual Perception © Stephen E. Palmer, 2002 Ceramic cup on a table David Marr, 1982

3. Four Stages of Visual Perception © Stephen E. Palmer, 2002 The Retinal Image An Image (blowup) Receptor Output

4. Four Stages of Visual Perception © Stephen E. Palmer, 2002 Image-based Representation Primal Sketch (Marr) An Image (Line Drawing) Retinal Image Image- based processes Edges Lines Blobs etc.

5. We likely throw away a lot

6. line drawings are universal

7. Four Stages of Visual Perception © Stephen E. Palmer, 2002 Surface-based Representation Primal Sketch 2.5-D Sketch Image-based Representation Surface- based processes Stereo Shading Motion etc.

8. Single Surface (Koenderink’s trick)

9. Four Stages of Visual Perception © Stephen E. Palmer, 2002 Surface-based Representation Primal Sketch 2.5-D Sketch Image-based Representation Surface- based processes Stereo Shading Motion etc.

10. Figure/Ground Organization  A contour belongs to one of the two (but not both) abutting regions. Figure (face) Ground (shapeless) Figure (Goblet) Ground (Shapeless) Important for the perception of shape

11. © Stephen E. Palmer, 2002 Properties of figures vs. grounds 15.18 FigureGround Thing-likeNot thing-like CloserFarther ShapedExtends behind Figure-Ground Organization

12. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Surroundedness 15.19 Figure-Ground Organization Surrounded region --> Figure Surrounding region --> Ground

13. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Size 15.20 Figure-Ground Organization Smaller region --> Figure Larger region --> Ground

14. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Orientation 15.21 Figure-Ground Organization Horizontal/vertical region --> Figure Oblique region --> Ground

15. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Contrast 15.22 Figure-Ground Organization Higher contrast region --> Figure Lower contrast region --> Ground

16. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Symmetry 15.23 Figure-Ground Organization Symmetrical region --> Figure Asymmetrical region --> Ground

17. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Convexity 15.24 Figure-Ground Organization More convex region --> Figure Less convex region --> Ground

18. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Parallelism 15.25 Figure-Ground Organization More parallel region --> Figure Less parallel region --> Ground

19. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Lower region 15.26 Figure-Ground Organization Lower region --> Figure Upper region --> Ground

20. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Meaningfulness 15.27 Figure-Ground Organization More meaningful region --> Figure Less meaningful region --> Ground

21. © Stephen E. Palmer, 2002 Relation to Depth Factors 15.28 Figure-Ground Organization Figure-ground organization as edge assignment: To which side does the edge belong? Depth cues can also be figure-ground factors and Figure-ground factors can be depth cues. To the closer side. This fact connects figure-ground organization with depth perception.

22. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Occlusion 15.29 Figure-Ground Organization Occluding region --> Figure Occluded region --> Ground

23. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Cast Shadows 15.30 Figure-Ground Organization Shadowing region --> Figure Shadowed region --> Ground

24. © Stephen E. Palmer, 2002 Principles of figure-ground organization: Shading 15.32 Figure-Ground Organization Shaded region --> Figure Nonshaded region --> Ground

25. Line Labeling > : contour direction + : convex edge - : concave edge possible junctions (constraints) Constraint Propagation [Clowes 1971, Huffman 1971; Waltz 1972; Malik 1986]

26. 26

27. Line Labeling

28. Four Stages of Visual Perception © Stephen E. Palmer, 2002 Object-based Representation Object- based processes Grouping Parsing Completion etc. Surface-based Representation 2.5-D Sketch Volumetric Sketch

29. Geons (Biederman '87)

30. Four Stages of Visual Perception © Stephen E. Palmer, 2002 Category-based Representation Category- based processes Pattern- Recognition Spatial- description Object-based Representation Volumetric Sketch Basic-level Category Category: cup Color: light-gray Size: 6” Location: table

31. We likely throw away a lot

32. line drawings are universal

33. However, things are not so simple… ● Problems with feed-forward model of processing…

34. Junctions in Real Images

35. Are Junctions local evidence? J McDermott, 2004

36. © Stephen E. Palmer, 2002 14.38 Is grouping an early or late process? Early vs. Late Grouping ????

37. © Stephen E. Palmer, 2002 14.39 Before or after stereoscopic depth? (Rock & Brosgole, 1964) Early vs. Late Grouping

38. © Stephen E. Palmer, 2002 14.40 Before or after lightness constancy? (Rock, Nijhawan, Palmer & Tudor, 1992) Early vs. Late Grouping Opaque paper strip

39. © Stephen E. Palmer, 2002 14.41 Before or after visual completion? (Palmer, Neff & Beck, 1996) Early vs. Late Grouping

40. © Stephen E. Palmer, 2002 14.42 Before or after illusory contours? (Palmer & Nelson, 2000) ? Early vs. Late Grouping

41. © Stephen E. Palmer, 2002 14.43 Conclusion: Grouping can occur “late” Question: Can grouping also occur “early” (Palmer & Brooks, in preparation) Early vs. Late Grouping

42. © Stephen E. Palmer, 2002 14.44 Grouping affects shape constancy (Palmer & Brooks, in preparation) Ambiguous Flat oval Circle in depth Early vs. Late Grouping

43. © Stephen E. Palmer, 2002 14.45 Proximity effects Biased toward oval Biased toward circle Early vs. Late Grouping

44. © Stephen E. Palmer, 2002 14.46 Color similarity effects Biased toward ovalBiased toward circle Early vs. Late Grouping

45. © Stephen E. Palmer, 2002 14.47 Common fate effects Biased toward ovalBiased toward circle Early vs. Late Grouping

46. © Stephen E. Palmer, 2002 14.48 Conclusion: Grouping occurs both “early” and “late” -- possibly everywhere! Grouping Early vs. Late Grouping

47. two-tone images

50. hair (not shadow!) inferred external contours “attached shadow” contour “cast shadow” contour

52. Finding 3D structure in two-tone images requires distinguishing cast shadows, attached shadows, and areas of low reflectivity The images do not contain this information a priori (at low level) Cavanagh's argument

53. A Classical View of Vision Grouping / Segmentation Figure/Ground Organization Object and Scene Recognition pixels, features, edges, etc. Low-level Mid-level High-level

54. A Contemporary View of Vision Figure/Ground Organization Grouping / Segmentation Object and Scene Recognition pixels, features, edges, etc. Low-level Mid-level High-level But where we draw this line?

55. Question #1: What (if anything) should be done at the “Low-Level”? N.B. I have already told you everything that is known. From now on, there aren’t any answers.. Only questions…

56. Who cares? Why not just use pixels? Pixel differences vs. Perceptual differences

57. Eye is not a photometer! "Every light is a shade, compared to the higher lights, till you come to the sun; and every shade is a light, compared to the deeper shades, till you come to the night." — John Ruskin, 1879

58. Cornsweet Illusion

59. Campbell-Robson contrast sensitivity curve Sine wave

60. Metamers

61. Question #1: What (if anything) should be done at the “Low-Level”? i.e. What input stimulus should we be invariant to?

62. Invariant to: Brightness / Color changes? small brightness / color changes low-frequency changes But one can be too invariant

63. Invariant to: Edge contrast / reversal? I shouldn’t care what background I am on! but be careful of exaggerating noise

64. Representation choices Raw Pixels Gradients: Gradient Magnitude: Thresholded gradients (edge + sign): Thresholded gradient mag. (edges):

65. Spatial invariance Rotation, Translation, Scale Yes, but not too much… In brain: complex cells – partial invariance In Comp. Vision: histogram-binning methods (SIFT, GIST, Shape Context, etc) or, equivalently, blurring (e.g. Geometric Blur) -- will discuss later

66. Many lives of a boundary

67. Often, context-dependent… inputcanny human Maybe low-level is never enough?

68. 1/f amplitude spectra for natural images (Field 1987) There are statistical regularities in the natural world, and image statistics reflect that. (Burton & Moorehead 1987; Field 1987; Tolhurst et al. 1992)

69. Why 1/f? Scale invariance Edges have 1/f structure Object distribution in real world (Ruderman 1997; Lee & Mumford 1999) (Image source: smokiesguidebook.com Slide content: Simoncelli & Olshausen 2001)

70. A closer look at amplitude spectra (Torralba & Oliva 2003)

71. Do natural image statistics matter? Sensory coding might exploit statistical regularities of our world according to various criteria: Representational efficiency Decorrelate input responses, make them independent, sparse, information theoretic metrics etc. Metabolic efficiency Spike efficiency, minimal wiring. Learning efficiency Sparseness, invariance, over completeness etc. Lots and lots of work; see reviews Graham & Field (2007), Simoncelli & Olshausen (2001)