1. Understanding the effect of lighting in images Ronen Basri

2. Light field Rays travel from sources to objects There they are either absorbed or reflected Energy decreases with distance and number of bounces Camera captures the set of rays that travel through the focal center

3. Specular reflectance (mirror) When a surface is smooth light reflects in the opposite direction of the surface normal

4. Specular reflectance When a surface is slightly rough the reflected light will fall off around the specular direction

5. Diffuse reflectance When the surface is very rough light may be reflected equally in all directions

6. Diffuse reflectance When the surface is very rough light may be reflected equally in all directions

7. BRDF

8. Lambertian reflectance

9. Lambert’s law

10. Preliminaries

11. Photometric stereo Given several images of the a lambertian object under varying lighting Assuming single directional source

12. Photometric stereo We can solve for S if L is known (Woodham) If L is unknown we can use SVD factorization (Hayakawa)

13. Factorization Use SVD to find a rank 3 approximation Define Factorization is not unique, since invertible To reduce ambiguity we impose integrability – up to generalized bas relief transformation (Belhumeur et al.)

14. Integrability

15. Shape from shading (SFS)

16. Shape from shading

18. Distance transform

19. Solution

20. Example

21. Illumination cone = 0.5*+0.2*+0.3*

22. Illumination cone Cone characterization is generic, holds also with specularities, shadows and inter- reflections Unfortunately, representing the cone is complicated (infinite degrees of freedom) Cone is “thin” for Lambertian objects; indeed the illumination cone of many objects can be represented with few PCA vectors (Yuille et al.)

23. Lambertian reflectance is smooth lighting reflectance (Basri & Jacobs; Ramamoorthi & Hanrahan)

24. Reflectance obtained with convolution   

25.   

26. Spherical harmonics  ZYX XZYZXY Positive values Negative values

27. Harmonic approximation

28. Harmonic transform of kernel

29. “Harmonic faces” Positive values Negative values ρ Albedo n Surface normal 

30. Photometric stereo L M S Image n : Image 1 Light n : Light 1 SVD recovers L and S up to an ambiguity  nznz nznz nyny  n z 2 -1)  n x 2 -n y 2 ) nxnynxny nxnznxnz nynznynz (Basri, Jacobs &Kemelmacher)

31. Photometric stereo

32. Motion + lighting

33. (Basri & Frolova)

34. Small motion

35. Reconstruction

36. More reconstructions

37. Conclusion Understanding the effect of lighting on images is challenging, but can lead to better interpretation of images We surveyed several problems: – Photometric stereo – Shape from shading – Modeling multiple sources with spherical harmonics – Motion and lighting We only looked at Lambertian objects…