Tal Amir Advanced Topics in Computer Vision May 29 th, 2015 COUPLED MOTION- LIGHTING ANALYSIS

1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

SURFACE RECONSTRUCTION: OVERVIEW

Structure from motion:  Images are taken from different locations  Scene remains static  Seek dense correspondence SURFACE RECONSTRUCTION: OVERVIEW

Structure from motion:  Images are taken from different locations  Scene remains static  Seeking dense correspondence  Brightness constancy assumption SURFACE RECONSTRUCTION: OVERVIEW

Structure from motion:  Images are taken from different locations  Scene remains static  Seek dense correspondence  Use it to recover depths  Brightness constancy assumption SURFACE RECONSTRUCTION: OVERVIEW

Structure from motion:  Images are taken from different locations  Scene remains static  Seek dense correspondence  Use it to recover depths  Brightness constancy assumption SURFACE RECONSTRUCTION: OVERVIEW Very simplistic assumption Requires Lambertian reflectance

Photometric reconstruction (stereo):  Images of the scene are taken with different illuminations  Scene and camera remain static  Uses a light scattering model SURFACE RECONSTRUCTION: OVERVIEW

Basri, R., Jacobs, D., & Kemelmacher, I. (2007). Photometric stereo with general, unknown lighting. International Journal of Computer Vision, 72(3),

Photometric reconstruction (stereo):  Few images of the scene are taken with different illuminations  Scene and camera remain static  Uses a light scattering model What about reconstruction with…  Object motion?  Camera motion? SURFACE RECONSTRUCTION: OVERVIEW

Coupled motion-lighting analysis  Uses a light scattering model  Takes object / camera motion into account  Assumes small (differential) motion, which is known SURFACE RECONSTRUCTION: OVERVIEW

Why assume differential motion? If the motion is large… SURFACE RECONSTRUCTION: OVERVIEW

Why assume differential motion? …we need point correspondences. SURFACE RECONSTRUCTION: OVERVIEW

Why assume differential motion? If we have a correspondence for all points, we’ve already solved the problem. Therefore, we assume the camera rotation and translation are small. SURFACE RECONSTRUCTION: OVERVIEW

1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

What happens when a beam of light hits a surface? LIGHT SCATTERING MODELS

What happens when a beam of light hits a surface? LIGHT SCATTERING MODELS Specular reflection

What happens when a beam of light hits a surface? LIGHT SCATTERING MODELS Specular reflection Diffuse reflections

What happens when a beam of light hits a surface? LIGHT SCATTERING MODELS Specular reflection Diffuse reflections Subsurface scattering There are other, more complex phenomena, such as interreflections and shadow casting.

LIGHT SCATTERING MODELS

Bidirectional Reflectance Distribution Function (BRDF): LIGHT SCATTERING MODELS

Bidirectional Reflectance Distribution Function (BRDF):  Generalizes specular and Lambertian reflections.  A narrower model: Symmetric BRDF.  Rotation invariant. Depends only on the angles between the three vectors. LIGHT SCATTERING MODELS

Bidirectional Reflectance Distribution Function (BRDF): LIGHT SCATTERING MODELS

Bidirectional Reflectance Distribution Function (BRDF):  Generalizes specular and Lambertian reflections.  A narrower model: Symmetric BRDF.  Rotation invariant. Depends only on the angles between the three vectors.  Does not account for subsurface scattering.  Only good for opaque surfaces. LIGHT SCATTERING MODELS

Bidirectional Reflectance Distribution Function (BRDF): LIGHT SCATTERING MODELS

Bidirectional Reflectance Distribution Function (BRDF): LIGHT SCATTERING MODELS

1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

Basri, R., & Frolova, D. (2008): A two-frame theory of motion, lighting and shape Assumptions:  Orthographic projection  Known (differential) object motion  Lambertian reflectance  Possibly non-uniform albedo  Known directional lighting SHAPE FROM OBJECT MOTION Basri, R., & Frolova, D. (2008, June). A two-frame theory of motion, lighting and shape. In Computer Vision and Pattern Recognition, CVPR IEEE Conference on (pp. 1-7). IEEE.

SHAPE FROM OBJECT MOTION

We need boundary conditions. At the boundary of the silhouette, we know the direction of the surface normal. SHAPE FROM OBJECT MOTION

Results SHAPE FROM OBJECT MOTION

1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

Chandraker, M., Reddy, D., Wang, Y., & Ramamoorthi, R. (2013): What object motion reveals about shape with unknown BRDF and lighting A general framework for surface reconstruction from camera motion. GENERALIZING TO UNKNOWN LIGHTING AND BRDF Chandraker, M., Reddy, D., Wang, Y., & Ramamoorthi, R. (2013, June). What object motion reveals about shape with unknown BRDF and lighting. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on (pp ). IEEE.

 Tells us when we can or cannot reconstruct surface.  Treats both orthographic and perspective projections.  Generalizes to complex unknown illumination.  Does not assume Lambertian reflectance. Limiting assumptions:  Known differential motion  Symmetric BRDF  Object is distant from camera and light sources GENERALIZING TO UNKNOWN LIGHTING AND BRDF

Results GENERALIZING TO UNKNOWN LIGHTING AND BRDF

Lighting is co-located with the camera: Characteristic curves Initial values GENERALIZING TO UNKNOWN LIGHTING AND BRDF

Unknown lighting: Characteristic curves Initial values GENERALIZING TO UNKNOWN LIGHTING AND BRDF

Results GENERALIZING TO UNKNOWN LIGHTING AND BRDF

Synthetic images: GENERALIZING TO UNKNOWN LIGHTING AND BRDF

Real images: GENERALIZING TO UNKNOWN LIGHTING AND BRDF

Real images: GENERALIZING TO UNKNOWN LIGHTING AND BRDF

1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

Chandraker, M., (2014): What camera motion reveals about shape with unknown BRDF  Handles the case of camera motion, while object and lighting remain static.  This task is more complex than object motion. SHAPE FROM CAMERA MOTION Chandraker, M. (2014, June). What camera motion reveals about shape with unknown BRDF. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on (pp ). IEEE.

For orthographic projection:  Depth cannot be recovered purely from camera motion.  A quasi-linear PDE can be derived using 3 images, if we make some restrictive assumptions on the BRDF. SHAPE FROM CAMERA MOTION

For orthographic projection:  Depth cannot be recovered purely from camera motion.  A quasi-linear PDE can be derived using 3 images, if we make some restrictive assumptions on the BRDF. For perspective projection:  Depth can be directly recovered from 4 images.  An additional linear PDE can be recovered if we make some restrictive assumptions on the BRDF. SHAPE FROM CAMERA MOTION

Results GENERALIZING TO UNKNOWN LIGHTING AND BRDF

Orthographic projection (synthetic images): GENERALIZING TO UNKNOWN LIGHTING AND BRDF

Perspective projection (synthetic images): GENERALIZING TO UNKNOWN LIGHTING AND BRDF

Perspective projection (real images): GENERALIZING TO UNKNOWN LIGHTING AND BRDF

1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

1.Surface reconstruction: Overview  Structure from Motion  Photometric Reconstruction  Coupled Motion-Lighting Analysis 2.Light scattering models  Lambertian reflectance  BRDF 3.Shape from object motion 4.Generalizing to unknown lighting and BRDF 5.Shape from camera motion 6.Conclusion OUTLINE

We have seen:  Surface reconstruction from two frames  Known directional lighting  Lambertian reflectance  Orthographic projection  General framework for object motion  Unknown illumination  Symmetric BRDF  Orthographic / perspective projection  General framework for camera motion CONCLUSION