1. Tal Amir Advanced Topics in Computer Vision May 29 th, 2015 COUPLED MOTION- LIGHTING ANALYSIS

2. 1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

3. SURFACE RECONSTRUCTION: OVERVIEW

5. Structure from motion:  Images are taken from different locations  Scene remains static  Seek dense correspondence SURFACE RECONSTRUCTION: OVERVIEW

6. Structure from motion:  Images are taken from different locations  Scene remains static  Seeking dense correspondence  Brightness constancy assumption SURFACE RECONSTRUCTION: OVERVIEW

7. Structure from motion:  Images are taken from different locations  Scene remains static  Seek dense correspondence  Use it to recover depths  Brightness constancy assumption SURFACE RECONSTRUCTION: OVERVIEW

8. Structure from motion:  Images are taken from different locations  Scene remains static  Seek dense correspondence  Use it to recover depths  Brightness constancy assumption SURFACE RECONSTRUCTION: OVERVIEW Very simplistic assumption Requires Lambertian reflectance

9. Photometric reconstruction (stereo):  Images of the scene are taken with different illuminations  Scene and camera remain static  Uses a light scattering model SURFACE RECONSTRUCTION: OVERVIEW

10. Basri, R., Jacobs, D., & Kemelmacher, I. (2007). Photometric stereo with general, unknown lighting. International Journal of Computer Vision, 72(3), 239-257.

11. Photometric reconstruction (stereo):  Few images of the scene are taken with different illuminations  Scene and camera remain static  Uses a light scattering model What about reconstruction with…  Object motion?  Camera motion? SURFACE RECONSTRUCTION: OVERVIEW

12. Coupled motion-lighting analysis  Uses a light scattering model  Takes object / camera motion into account  Assumes small (differential) motion, which is known SURFACE RECONSTRUCTION: OVERVIEW

13. Why assume differential motion? If the motion is large… SURFACE RECONSTRUCTION: OVERVIEW

14. Why assume differential motion? …we need point correspondences. SURFACE RECONSTRUCTION: OVERVIEW

15. Why assume differential motion? If we have a correspondence for all points, we’ve already solved the problem. Therefore, we assume the camera rotation and translation are small. SURFACE RECONSTRUCTION: OVERVIEW

16. 1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

17. 1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

18. What happens when a beam of light hits a surface? LIGHT SCATTERING MODELS

19. What happens when a beam of light hits a surface? LIGHT SCATTERING MODELS Specular reflection

20. What happens when a beam of light hits a surface? LIGHT SCATTERING MODELS Specular reflection Diffuse reflections

21. What happens when a beam of light hits a surface? LIGHT SCATTERING MODELS Specular reflection Diffuse reflections Subsurface scattering There are other, more complex phenomena, such as interreflections and shadow casting.

22. LIGHT SCATTERING MODELS

24. Bidirectional Reflectance Distribution Function (BRDF): LIGHT SCATTERING MODELS

25. Bidirectional Reflectance Distribution Function (BRDF):  Generalizes specular and Lambertian reflections.  A narrower model: Symmetric BRDF.  Rotation invariant. Depends only on the angles between the three vectors. LIGHT SCATTERING MODELS

26. Bidirectional Reflectance Distribution Function (BRDF): LIGHT SCATTERING MODELS

27. Bidirectional Reflectance Distribution Function (BRDF):  Generalizes specular and Lambertian reflections.  A narrower model: Symmetric BRDF.  Rotation invariant. Depends only on the angles between the three vectors.  Does not account for subsurface scattering.  Only good for opaque surfaces. LIGHT SCATTERING MODELS

28. Bidirectional Reflectance Distribution Function (BRDF): LIGHT SCATTERING MODELS

29. Bidirectional Reflectance Distribution Function (BRDF): LIGHT SCATTERING MODELS

30. 1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

31. 1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

32. Basri, R., & Frolova, D. (2008): A two-frame theory of motion, lighting and shape Assumptions:  Orthographic projection  Known (differential) object motion  Lambertian reflectance  Possibly non-uniform albedo  Known directional lighting SHAPE FROM OBJECT MOTION Basri, R., & Frolova, D. (2008, June). A two-frame theory of motion, lighting and shape. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on (pp. 1-7). IEEE.

33. SHAPE FROM OBJECT MOTION

48. We need boundary conditions. At the boundary of the silhouette, we know the direction of the surface normal. SHAPE FROM OBJECT MOTION

51. Results SHAPE FROM OBJECT MOTION

55. 1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

56. 1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

57. Chandraker, M., Reddy, D., Wang, Y., & Ramamoorthi, R. (2013): What object motion reveals about shape with unknown BRDF and lighting A general framework for surface reconstruction from camera motion. GENERALIZING TO UNKNOWN LIGHTING AND BRDF Chandraker, M., Reddy, D., Wang, Y., & Ramamoorthi, R. (2013, June). What object motion reveals about shape with unknown BRDF and lighting. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on (pp. 2523-2530). IEEE.

58.  Tells us when we can or cannot reconstruct surface.  Treats both orthographic and perspective projections.  Generalizes to complex unknown illumination.  Does not assume Lambertian reflectance. Limiting assumptions:  Known differential motion  Symmetric BRDF  Object is distant from camera and light sources GENERALIZING TO UNKNOWN LIGHTING AND BRDF

67. Results GENERALIZING TO UNKNOWN LIGHTING AND BRDF

68. Lighting is co-located with the camera: Characteristic curves Initial values GENERALIZING TO UNKNOWN LIGHTING AND BRDF

69. Unknown lighting: Characteristic curves Initial values GENERALIZING TO UNKNOWN LIGHTING AND BRDF

71. Results GENERALIZING TO UNKNOWN LIGHTING AND BRDF

72. Synthetic images: GENERALIZING TO UNKNOWN LIGHTING AND BRDF

73. Real images: GENERALIZING TO UNKNOWN LIGHTING AND BRDF

74. Real images: GENERALIZING TO UNKNOWN LIGHTING AND BRDF

75. 1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

76. 1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

77. Chandraker, M., (2014): What camera motion reveals about shape with unknown BRDF  Handles the case of camera motion, while object and lighting remain static.  This task is more complex than object motion. SHAPE FROM CAMERA MOTION Chandraker, M. (2014, June). What camera motion reveals about shape with unknown BRDF. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on (pp. 2179-2186). IEEE.

78. For orthographic projection:  Depth cannot be recovered purely from camera motion.  A quasi-linear PDE can be derived using 3 images, if we make some restrictive assumptions on the BRDF. SHAPE FROM CAMERA MOTION

80. For orthographic projection:  Depth cannot be recovered purely from camera motion.  A quasi-linear PDE can be derived using 3 images, if we make some restrictive assumptions on the BRDF. For perspective projection:  Depth can be directly recovered from 4 images.  An additional linear PDE can be recovered if we make some restrictive assumptions on the BRDF. SHAPE FROM CAMERA MOTION

81. Results GENERALIZING TO UNKNOWN LIGHTING AND BRDF

82. Orthographic projection (synthetic images): GENERALIZING TO UNKNOWN LIGHTING AND BRDF

83. Perspective projection (synthetic images): GENERALIZING TO UNKNOWN LIGHTING AND BRDF

84. Perspective projection (real images): GENERALIZING TO UNKNOWN LIGHTING AND BRDF

85. 1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

86. 1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

87. We have seen:  Surface reconstruction from two frames  Known directional lighting  Lambertian reflectance  Orthographic projection  General framework for object motion  Unknown illumination  Symmetric BRDF  Orthographic / perspective projection  General framework for camera motion CONCLUSION