1. Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts Institute of Technology

2. Matte Glossy Mirrored

3. Henry Moore

4.  Visual system estimates surface orientation from image intensity Classical Shape from Shading reflectance mapimage

5.  Visual system estimates surface orientation from image intensity Classical Shape from Shading reflectance map  Problems:  Intensities are ambiguous  Reflectance map is unknown  No principled way to predict successes vs. failures of shape perception

6. Surface reflectance  A parametric space of glossy plastic materials (using Ward model) Diffuse Reflectance, d Specular Reflectance, s

7.  Don’t use image intensity !  Use the kinds of image measurements the visual system employs at the front end Alternative approach reflectance mapimage

8.  Don’t use image intensity !  Use the kinds of image measurements the visual system employs at the front end Alternative approach image  What can these measurements tell us about 3D shape ?  Can filter responses predict human shape perception ?

9. highly curved Curvatures determine distortions

10. slightly curved Anisotropies in surface curvature lead to powerful distortions of the reflected world Curvatures determine distortions

11. Population codes

15. Statistics Illuminations Shapes  Render many images:  50 Shapes  12 Illuminations  5 Reflectances  Measure the distribution of orientations (i.e. filter population response) for every point in every image  Look for regularities

16. Orientation fields Ground truth

17. Orientation fields Error (estimate - ground truth)

18. Surface reflectance Diffuse Reflectance, d Specular Reflectance, s

19.  If the visual system relies on these measurements then: 1: Shape perception should be stable across changes that do not affect these measurements 2: Perceived shape should vary systematically when scene or image modifications do affect these measurements Predictions

20.  Perceived shape should be extremely stable across changes in surface glossiness. Prediction 1 Nefs, Koenderink & Kappers, 2006 “We found no evidence that the perceived shapes of glossy objects are different from the perceived shapes of matte objects...”

21. Experiment Specular reflection Diffuse reflection

22. Experiment Specular reflection Diffuse reflection

23. Orientation fields ground truth

24. Orientation fields ground truth

25.  For shaded surfaces, perceived shape should undergo (subtle) changes across variations in illumination Prediction 2

26.  For shaded surfaces, perceived shape should undergo (subtle) changes across variations in illumination Prediction 2

27.  For shaded surfaces, perceived shape should undergo (subtle) changes across variations in illumination Prediction 2 Todd, Norman, Koenderink & Kappers (1997) report little effect of illumination. But that was with additional cues. Koenderink, van Doorn, Christou & Lappin (1996) Nefs, Koenderink & Kappers, 2006

28.  For shaded surfaces, perceived shape should undergo (subtle) changes across variations in illumination Prediction 2 Caniard & Fleming, 2007

29.  If the visual system relies on these measurements then: 1: Shape perception should be stable across changes that do not affect these measurements. Even when these changes are not natural. 2: Perceived shape should vary systematically when scene or image modifications do affect these measurements Predictions

30.  Test improbable combination of lighting and reflectance  Decouple intensity from image orientation non-linear intensity transfer function normal shading ‘weird’ shading “Weird” Shading

31. normal shading ‘weird’ shading “Weird” shading

32. normal shading ‘weird’ shading “Weird” shading perceived tilt perceived slant normal shading “weird” shading S1

33. normal shading ‘weird’ shading “Weird” shading perceived tilt perceived slant normal shading “weird” shading S2

34. normal shading ‘weird’ shading “Weird” shading  Pooled data across 6 shapes tiltslant normal shading r 2 = 0.93r 2 = 0.88

35. Affine Transformation Shear: - does affect first derivatives - does NOT affect second derivatives

36. Shear: - does affect first derivatives - does NOT affect second derivatives Affine Transformation

37. Shear: - does affect first derivatives - does NOT affect second derivatives Affine Transformation

38. Shear: - does affect first derivatives - does NOT affect second derivatives Affine Transformation

39. Matching Task  Subject adjusts shear of match until it appears to be same shape as test testmatch

40. Matching Task  Subject adjusts shear of match until it appears to be same shape as test testmatch

41. Matching Task  Subject adjusts shear of match until it appears to be same shape as test testmatch

42. Matching Task  Subject adjusts shear of match until it appears to be same shape as test testmatch

43. Predictions test shear match shear veridical image statistics prediction

44. Results test shear match shear

45.  If the visual system relies on these measurements then: 1: Shape perception should be stable across changes that do not affect these measurements. 2: Perceived shape should vary systematically when scene or image modifications do affect these measurements. Even when these changes are not natural. Predictions

46. Illusory distortions of shape Inspired by Todd & Thaler VSS 05

47. Illusory distortions of shape Inspired by Todd & Thaler VSS 05

48. Illusory distortions of shape Inspired by Todd & Thaler VSS 05

49. Illusory distortions of shape

51. Experiment

52. Results veridicalstimulus

53. Results predictedstimulus

54. Results resultsstimulus

55.  Dot product between subject’s data and predictions Results

56. Results

57. Results “veridical” prediction “orientation field” prediction

58.  Dot product between subject’s data and predictions Results “veridical” prediction “orientation field” prediction

59. Conclusions  Useful shape cues can be derived from relatively simple image measurements at the front end of vision  In some cases these measurements are surprisingly robust across variations in other scene properties (e.g. illumination, reflectance).  Scale and orientation measurements can predict certain successes and failures of human 3D shape perception across a range of natural and unnatural stimuli.

60. Thank you Funding RF supported by DFG FL 624/1-1

61.  Generative space of all possible combinations of surface curvature and local orientation in the reflectance map Expected errors

62. Reflectance as Illumination a(f) = 1 / f   = 0  = 0.4  = 0.8  = 1.2  = 1.6  = 2.0  = 4.0  = 8.0

63. Cues to 3D Shape specularitiesshadingtexture Conventional wisdom: different cues have different physical causes  must be processed differently by visual system (‘modules’)

64. specularitiesshadingtexture Goal: Find commonalities between cues. Cues to 3D Shape

66. Fleming, Torralba, Adelson Todd and colleagues Mingolla and Grossberg Koenderink and van Doorn Zucker and colleagues Zaidi and Li Malik and Rosenholtz

67. Rendering with Reflectance maps  Reflectance map is a lookup-table that specifies image intensity for all surface normals  Surface normals are indices for accessing values from the reflectance map  Within a local patch of surface, the normal changes smoothly  This maps a small patch of the reflectance map “texture” into the image  The rate at which the indices sweep through the reflectance map determines the warping transformation that is applied to the texture patch during the mapping

68. Hierarchy of shape attributes  We often refer to “stereo” or “texture”, or “shading” as “cues” to shape.  Traditional definition of shape cue: a physical property that can inform us about shape, e.g. “stereo”, or “texture”, or “shading”  New definition of cue: a specific image measurement that provides statistically reliable information about a specific property of the scene.  Any given cue on its own may be highly ambiguous, specifying some abstract, high level scene property that does not uniquely specify the object

69. Hierarchy of shape attributes  Easily measurable image statistics that can inform us about any property of shape

70. It is remarkable that we can recover 3D shape:  No motion  No stereo  No shading  No texture image consists of nothing more than a distorted reflection of the world surrounding the object Ideal mirrored surface Fleming et al. (2004). JOV Shape from Specularities

71. As the object moves from scene to scene, the image changes dramatically. Yet, somehow we are able to recover the 3D shape. Shape from Specularities

72. Approach I: inverse optics  Estimate shape by inverting the physics of mirror reflections. Image from Savarese and Perona  Make an explicit model of the environment  Make assumptions about specific environmental features (e.g. ‘lines are straight’)

73.  Estimate shape directly from the image  Collect image measurements that are reliable across ‘typical’ environments Approach II: direct perception  No need to estimate the environment  ust use the pattern of distortions in the image

74. Pattern of compressions and rarefactions across the image indicates something about the 3D shape. Shape from Texture

75.  Real-world illumination is highly structured  Specular reflections of the real world are a bit like texture  Can we solve the 3D shape of mirrors using shape-from- texture ? Shape from Texture ?

76.  Slant distorts texture but not reflections Image distortions

79.  Curvature distorts reflections but not texture Image distortions

80. Shape-from-texture and shape-from-specularity follow different rules  For texture, image compression depends on surface slant  first derivative of surface  For reflections, image compression depends on surface curvature properties  second derivatives of surface

81. Local analysis: banding patterns

82. Gauge Figure Task  Subject adjusts 3D orientation of “gauge figure” to match local orientation of surface

83. Slant and Tilt Image from Palmer, 1999

84. Results I objective tilt subjective tilt Tilt objective slant subjective slant Slant objective tilt subjective tilt objective slant subjective slant TiltSlant

85. Results II objective tilt subjective tilt Tilt objective slant subjective slant Slant objective tilt subjective tilt objective slant subjective slant TiltSlant

86. Is it just the occluding contour? No, it is not

87. Interpreting distorted reflections

88. Effects of compression

89. 3D shape appears to be conveyed by the continuously varying patterns of orientation across the image of a surface

90. Beyond specularity Specular reflection Diffuse reflection

91. Differences between diffuse and specular reflection

94. Shiny Painted

95. Beyond specularity Specular reflection Diffuse reflection

96. Latent orientation structure

97. Orientation fields in shading

99. highly curved

100. slightly curved Anisotropies in surface curvature lead to anisotropies in the image.

101. Texture Anisotropic compression of texture depends on surface slant

102. Texture

103. Orientation fields in texture

106.  No need for visual system to estimate reflectance or illumination explicitly.  Classical shape from shading uses the reflectance map to estimate surface normals from image intensities  Reflectance map is usually unknown and ambiguous Potential of Orientation Fields

107.  Visual system estimates surface orientation from image intensity Classical Shape from Shading reflectance mapimage

108.  Stable across albedo discontinuities. Breton and Zucker (1996), Huggins and Zucker (2001) Potential of Orientation Fields

109.  Uses biologically plausible measurements Orientation selectivity maps in primary visual cortex of tree shrew. After Bosking et al. (1997). Potential of Orientation Fields

110.  May explain how images with no obvious BRDF interpretation nevertheless yield 3D percepts Potential of Orientation Fields Ohad Ben-Shahar

111. Converting between cues input image Todd & Oomes 2004 ( ) 2 Latent shading

112. ( ) 2 Converting between cues input image Todd & Oomes 2004 Latent shading

113. Matte vs. Shiny  Same generative statistics, different mappings Mapped as texture Mapped as reflection Mapped as texture Mapped as reflection

114. Texture vs. Reflectance

118. Conclusions  Orientation fields are potentially a very powerful source of information about 3D shape  For the early stages of 3D shape processing, seemingly different cues may have more in common than previously thought

119. Todd’s Blobs

121. What still needs to be explained?  For Lambertian materials (or blurry illuminations), the reflectance map is so smooth that it is significantly anisotropic.  Therefore shading orientation fields vary considerably with changes in illumination. sidefronttop

122. What still needs to be explained?  Note analogy to textures of different orientations Todd et al. (2004)

123. Two possibilities I.Change in orientation field predicts (subtle) changes in perceived 3D shape II.There are higher-order invariants in the orientation fields sidefronttop

124. Eigenvectors of Hessian matrix Intrinsic principal curvatures

126. Matte dark grey Rough metal Glossy light grey

127. Plastics (a) Mirror(b) Smooth plastic(c) Rough plastic

128. When the world is anisotropic Brushed horizontallyBrushed vertically

129. Stripy world

130. Matte vs. Shiny  Same generative statistics, different mappings Mapped as texture Mapped as reflection Mapped as texture Mapped as reflection

131. Hypothesis: the way the reflections are distorted is systematically related to properties of the 3D shape Shape from Specularities