1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics

2 Cues to 3D Shape specularitiesshadingtexture Conventional wisdom: different cues have different physical causes must be processed differently by visual system (modules)

3 specularitiesshadingtexture Goal: Find commonalities between cues. Cues to 3D Shape

4

5 Fleming, Torralba, Adelson Todd and colleagues Mingolla and Grossberg Koenderink and van Doorn Zucker and colleagues Zaidi and Li Malik and Rosenholtz

6 It is remarkable that we can recover 3D shape: No motion No stereo No shading No texture image consists of nothing more than a distorted reflection of the world surrounding the object Ideal mirrored surface Fleming et al. (2004). JOV Shape from Specularities

7 As the object moves from scene to scene, the image changes dramatically. Yet, somehow we are able to recover the 3D shape. Shape from Specularities

8 Curvatures determine distortions highly curved

9 Curvatures determine distortions slightly curved Anisotropies in surface curvature lead to powerful distortions of the reflected world

10 Interpreting distorted reflections

11 Orientation fields Ground truth

12 3D shape appears to be conveyed by the continuously varying patterns of orientation across the image of a surface

13 Beyond specularity Specular reflection Diffuse reflection

14 Differences between diffuse and specular reflection

15 Differences between diffuse and specular reflection

16 Differences between diffuse and specular reflection

17 Shiny Painted

18 Beyond specularity Specular reflection Diffuse reflection

19 Latent orientation structure

20 Orientation fields in shading

21 Orientation fields in shading

22 Reflectance as Illumination a(f) = 1 / f = 0 = 0.4 = 0.8 = 1.2 = 1.6 = 2.0 = 4.0 = 8.0

23 highly curved

24 slightly curved Anisotropies in surface curvature lead to anisotropies in the image.

25 Stability across changes in surface reflectance A parametric space of glossy plastic materials (using Ward model) Diffuse Reflectance, d Specular Reflectance, s

26 Idea: Experiment 1 Rationale: measure stability of 3D shape across changes in surface reflectance Method: gauge figure task? Problem: costly to do full depth reconstruction for many shapes and materials Solution? Compare sparse gauge measurement? Alternative task?: locate depth extrema along given raster line (2D task)

27 Texture Anisotropic compression of texture depends on surface slant

28 Texture Anisotropic compression of texture depends on surface slant

29 Orientation fields in texture

30 Orientation fields in texture

31 Orientation fields in texture

32 Affine Transformation Shear: - does affect first derivatives - does NOT affect second derivatives

33 Shear: - does affect first derivatives - does NOT affect second derivatives Affine Transformation

34 Shear: - does affect first derivatives - does NOT affect second derivatives Affine Transformation

35 Shear: - does affect first derivatives - does NOT affect second derivatives Affine Transformation

36 Shear: - does affect first derivatives - does NOT affect second derivatives Affine Transformation

37 Shear: - does affect first derivatives - does NOT affect second derivatives Affine Transformation

38 Idea: Experiment 2 Rationale: use orientation fields to predict misperceptions of 3D shape Possible methods Gauge figure task? Matching task: subject adjusts shear of a textured object until it appears to match the shaded version of the same object Subject adjusts shear of one oject (shaded or textured) until it appears to match the degree of shear of another object? Sounds too strange?

39 Illusory distortions of shape Inspired by Todd & Thaler VSS 05

40 Illusory distortions of shape Inspired by Todd & Thaler VSS 05

41 Idea: Experiment 3 Rationale: use orientation fields to predict misperceptions of 3D shape Possible methods gauge figure task to reconstruct full 3D shape. Again, this is costly, but perhaps a few shapes are enough depth extrema task: locate depth extrema along raster line (this is what Todd and Thaler did). Potentially we could predict the locus directly from the orientation field

42 Idea: Experiment 3 Compare small and large changes in orientation field by using texture stretching along the line of sight Advantage: same infringement of isotropy assumption, different change in apparent 3D shape Unstretched Stretched 2:1 along line of sight

43 Uses biologically plausible measurements Orientation selectivity maps in primary visual cortex of tree shrew. After Bosking et al. (1997). Potential of Orientation Fields

44 No need for visual system to estimate reflectance or illumination explicitly. Classical shape from shading uses the reflectance map to estimate surface normals from image intensities Reflectance map is usually unknown and ambiguous Potential of Orientation Fields

45 Stable across albedo discontinuities. Breton and Zucker (1996), Huggins and Zucker (2001) Potential of Orientation Fields

46 Handle improbable combinations of reflectance and illumination. non-linear intensity transfer function normal shading weird shading Potential of Orientation Fields

47 We could measures shape estimates with these types of stimuli as well. non-linear intensity transfer function normal shading weird shading Link back to experiment 1 ?

48 May explain how images with no obvious BRDF interpretation nevertheless yield 3D percepts Potential of Orientation Fields Ohad Ben-Shahar

49 Converting between cues input image Todd & Oomes 2004 ( ) 2 Latent shading

50 ( ) 2 Converting between cues input image Todd & Oomes 2004 Latent shading

51 Conclusions Orientation fields are potentially a very powerful source of information about 3D shape For the early stages of 3D shape processing, seemingly different cues may have more in common than previously thought

52 Thank you Collaborators Ted Adelson Antonio Torralba Funding RF supported by DFG FL 624/1-1

53 What still needs to be explained? For Lambertian materials (or blurry illuminations), the reflectance map is so smooth that it is significantly anisotropic. Therefore shading orientation fields vary considerably with changes in illumination. sidefronttop

54 What still needs to be explained? Surprising prediction: 3D shape should actually be less stable across changes in illumination for diffuse than for specular materials. We found evidence for changes in 3D shape with changes in illumination Alternative: higher order invariants establish an equivalence between different orientation fields. Example: joint measures of orientation at different locations. sidefronttop

55 Note analogy to textures of different orientations Todd et al. (2004) What still needs to be explained?

56 Matte dark grey Rough metal Glossy light grey

57 Plastics (a) Mirror(b) Smooth plastic(c) Rough plastic

58 When the world is anisotropic Brushed horizontallyBrushed vertically

59 Stability across changes in surface reflectance A parametric space of glossy plastic materials (using Ward model) Diffuse Reflectance, d Specular Reflectance, s