1. Computer Graphics Psychophysics Heinrich H. Bülthoff Max-Planck-Institute for Biological Cybernetics Tübingen, Germany Heinrich H. Bülthoff Max-Planck-Institute for Biological Cybernetics Tübingen, Germany

2. What is psychophysics? A research strategy to understand perception and performance by testing the relationship between the psychic (what an observer sees and reports) and the physical (patterns of light entering the eye).

3. Method Show and tell : –static images –video animations –vision + haptics –virtual environments Show and tell : –static images –video animations –vision + haptics –virtual environments

4. Basic Question: What image information does the brain use and what does it ignore ? Applied Question: How can we render images that convey only the information that the brain uses? Basic Question: What image information does the brain use and what does it ignore ? Applied Question: How can we render images that convey only the information that the brain uses? Main questions

5. Examples of vision problems – Recognition – Depth perception – Navigation –(Haptics) – Recognition – Depth perception – Navigation –(Haptics)

6. Computer Graphics Psychophysics at the MPI Tuebingen RecognitionRecognition Shape PerceptionShape Perception HapticsHaptics DrivingDriving NavigationNavigation RecognitionRecognition Shape PerceptionShape Perception HapticsHaptics DrivingDriving NavigationNavigation

7. Object recognition – how does it work? Naïve theory : First, the brain computes an object’s 3-D shape. Then the brain matches the 3-D shape to previously seen shapes that are stored in memory. image z-buffer 3-D shape memory image z-buffer 3-D shape memory Naïve theory : First, the brain computes an object’s 3-D shape. Then the brain matches the 3-D shape to previously seen shapes that are stored in memory. image z-buffer 3-D shape memory image z-buffer 3-D shape memory

8. Image-based Recognition Alternative Theory: The visual system ignores depth perception when recognizing an object. Rather, the visual system recognizes an object directly from its image. z-buffer (depth perception) z-buffer (depth perception) image image image memory (recognition) image memory (recognition) Alternative Theory: The visual system ignores depth perception when recognizing an object. Rather, the visual system recognizes an object directly from its image. z-buffer (depth perception) z-buffer (depth perception) image image image memory (recognition) image memory (recognition)

9. Recognition of Biological Motion Recognition of point-like walker

10. Biological Motion Perception Johansson (1911-1998) Walter: insert orig. johannson movie here 30 sec max Walter: insert orig. johannson movie here 30 sec max

11. Depth perception from stereo is ignored Bülthoff, Bülthoff and Sinha (Nature Neuroscience 1998) Recognition is unaffected by scrambling the depth structure 2D motion pattern and not 3D structure is used for recognition Z y z x

12. Image-based Recognition Evidence from: Psychophysics (Buelthoff, MPI Tuebingen)Psychophysics (Buelthoff, MPI Tuebingen) –limited generalization (30°) despite full 3D information Psychophysics (Buelthoff, MPI Tuebingen)Psychophysics (Buelthoff, MPI Tuebingen) –limited generalization (30°) despite full 3D information Physiology (Logothetis, MPI Tuebingen)Physiology (Logothetis, MPI Tuebingen) –image-specific neurons in trained monkeys Physiology (Logothetis, MPI Tuebingen)Physiology (Logothetis, MPI Tuebingen) –image-specific neurons in trained monkeys Theory (Poggio, MIT)Theory (Poggio, MIT) –image-interpolation networks Theory (Poggio, MIT)Theory (Poggio, MIT) –image-interpolation networks Rendering Application (Blanz & Vetter, MPI Tuebingen)Rendering Application (Blanz & Vetter, MPI Tuebingen) –image-based face synthesis Rendering Application (Blanz & Vetter, MPI Tuebingen)Rendering Application (Blanz & Vetter, MPI Tuebingen) –image-based face synthesis

13. One Object - Two Interpretations Markus Raetz

14. Man or Hare ? Markus Raetz

15. Why does the brain make assumptions? Images are ambiguous. Any image can be explained by several combinations of 3-D shape, material and lighting.

16. e.g. The Necker Cube

17. view from aboveview from below

18. e.g. Depth-reversal ambiguity in shading A valley illuminated from the right looks the same as a hill illuminated from the left. hillvalley

19. Prior assumption about light source

20. Mould of a footprint ?

21. Assumption that light source is stationary

23. Prior assumptions about shape The Hollow Mask Illusion

24. 3 prior assumptions 1. light from above 2. viewpoint from above 3. shape is convex 1. light from above 2. viewpoint from above 3. shape is convex

25. Example: failure of all 3 prior assumptions shape is concave viewpoint from below light from below shape is concave viewpoint from below light from below

26. “ Measuring Visual Shape using Computer Graphics Psychophysics” (see Workshop proceedings) convex concave convex concave (face) (mask) (face) (mask) convex concave convex concave (face) (mask) (face) (mask)

27. Procedure

29. Task: hill or valley ?

30. Assumption 1 : light direction lightfromabovelightfrombelow lightfromabovelightfrombelow

31. Assumption 2 : viewpoint direction (case 1 - convex shape) view from above view from above view from below view from below view from above view from above view from below view from below

32. Assumption 2 : viewpoint direction (case 2 – concave shape) view from below view from below view from above view from above view from below view from below view from above view from above

33. Data : per cent correct scores (see workshop proceedings for more details) 87 (best) 15 (worst) 64 39

34. Lessons for rendering Illumination and viewpoint should be from above, especially if the surface is concave. Otherwise : -Shading will look weird. -Perceived shape will be incorrect. Illumination and viewpoint should be from above, especially if the surface is concave. Otherwise : -Shading will look weird. -Perceived shape will be incorrect.

35. Weird shading Frankenstein monster illuminated from below.

36. Face Recognition Cyberware Scanner Texture Map Shape Map Cyberware Scanner Texture Map Shape Map

37. MPI 3-D Face Database

38. View-synthesis from a single image Volker Blanz and Thomas Vetter (SIGGRAPH ‘99) Input Output W 2 * + W 1 * W 3 * + W 4 * + +….. = = W 2 * + W 1 * W 3 * + W 4 * +

39. All views from a single image

40. Synthetic Actors Volker Blanz und Thomas Vetter (SIGGRAPH 1999)

41. Audrey Hepburn (1929-1993) New Hat + New Illumination Original New View

42. Do you recognize her ? Mona Lisa New View

43. Virtual Tuebingen

44. Summary (Visual Recogntion) Visual recognition is based on images, not on 3-D geometry.Visual recognition is based on images, not on 3-D geometry. –More texture memory not polygons Better to apply a texture map from a previously seen image than to re-render an object or scene from a new viewpoint or under a new lighting condition. Visual recognition is based on images, not on 3-D geometry.Visual recognition is based on images, not on 3-D geometry. –More texture memory not polygons Better to apply a texture map from a previously seen image than to re-render an object or scene from a new viewpoint or under a new lighting condition.

45. Summary Does the brain use a depth buffer? – –Yes: for shape perception (shape from shading) – –No: for recognition (biological motion) Stereo goggles not necessary for recognition. Does the brain use a depth buffer? – –Yes: for shape perception (shape from shading) – –No: for recognition (biological motion) Stereo goggles not necessary for recognition.

46. Summary (3-D Shape Perception) Shape from shading is strongly affected by prior assumptions hard wired into the brain.Shape from shading is strongly affected by prior assumptions hard wired into the brain. Photorealistic renderings will look weird if the scene model or viewing parameters are inconsistent with these assumptions. Photorealistic renderings will look weird if the scene model or viewing parameters are inconsistent with these assumptions. Shape from shading is strongly affected by prior assumptions hard wired into the brain.Shape from shading is strongly affected by prior assumptions hard wired into the brain. Photorealistic renderings will look weird if the scene model or viewing parameters are inconsistent with these assumptions. Photorealistic renderings will look weird if the scene model or viewing parameters are inconsistent with these assumptions.

47. 100 100 light-from-above light-from-above light-from-below light-from-belowPercent correct 50 view-from-above view-from-below view-from-below convex convex concave concave Data (see workshop proceedings) 0