1. Analysis of Lighting Effects
Outline:
The problem
Lighting models
Shape from shading
Photometric stereo
Harmonic analysis of lighting

5. Applications Modeling the effect of lighting can be used for
Recognition – particularly face recognition
Shape reconstruction
Motion estimation
Re-rendering …

6. Lighting is Complex
Lighting can come from any direction and at any strength
Infinite degree of freedom

7. Issues in Lighting
Single light source (point, extended) vs. multiple light sources
Far light vs. near light
Matt surfaces vs. specular surfaces
Cast shadows
Inter-reflections

8. Lighting From a source – travels in straight lines
Energy decreases with r2 (r – distance from source)
When light rays reach an object
Part of the energy is absorbed
Part is reflected (possibly different amounts in different directions)
Part may continue traveling into the object, if object is transparent / translucent

9. Specular Reflectance
When a surface is smooth light reflects in the opposite direction of the surface normal

10. Specular Reflectance
When a surface is slightly rough the reflected light will fall off around the specular direction

11. Lambertian Reflectance
When the surface is very rough light may be reflected equally in all directions

12. Lambertian Reflectance
When the surface is very rough light may be reflected equally in all directions

13. Lambertian Reflectance

14. Lambert Law q or

15. BRDF
A general description of how opaque objects reflect light is given by the Bidirectional Reflectance Distribution Function (BRDF)
BRDF specifies for a unit of incoming light in a direction (θi,Φi) how much light will be reflected in a direction (θe,Φe) . BRDF is a function of 4 variables f(θi,Φi;θe,Φe).
(0,0) denotes the direction of the surface normal.
Most surfaces are isotropic, i.e., reflectance in any direction depends on the relative direction with respect to the incoming direction (leaving 3 parameters)

16. Why BRDF is Needed?
Light from front Light from back

17. Most Existing Algorithms
Assume a single, distant point source
All normals visible to the source (θ<90°)
Plus, maybe, ambient light (constant lighting from all directions)

18. Shape from Shading Input: a single image Output: 3D shape
Problem is ill-posed, many different shapes can give rise to same image
Common assumptions:
Lighting is known
Reflectance properties are completely known – For Lambertian surfaces albedo is known (usually uniform)

19. convex
concave

20. convex
concave

21. HVS Assumes Light from Above

22. HVS Assumes Light from Above

23. Lambertian Shape from Shading (SFS)
Image irradiance equation
Image intensity depends on surface orientation
It also depends on lighting and albedo, but those assumed to be known

24. Surface Normal A surface z(x,y) A point on the surface: (x,y,z(x,y))T
Tangent directions tx=(1,0,p)T, ty=(0,1,q)T with p=zx, q=zy

25. Lambertian SFS We obtain
Proportionality – because albedo is known up to scale
For each point one differential equation in two unknowns, p and q
But both come from an integrable surface z(x,y)
Thus, py= qx (zxy=zyx).
Therefore, one differential equation in one unknowns

26. Lambertian SFS

27. SFS with Fast Marching
Suppose lighting coincides with viewing direction l=(0,0,1)T, then
Therefore
For general l we can rotate the camera

28. Distance Transform is called Eikonal equation
Consider d(x) s.t. |dx|=1
Assume x0=0 d x x0

29. Distance Transform is called Eikonal equation
Consider d(x) s.t. |dx|=1
Assume x0=0 and x0=1 d x x0 x1

30. SFS with Fast Marching
- Some places are more difficult to walk than others
Solution to Eikonal equations –using a variation of Dijkstra’s algorithm
Initial condition: we need to know z at extrema
Starting from lowest points, we propagate a wave front, where we gradually compute new values of z from old ones

31. Results

32. Photometric Stereo
Fewer assumptions are needed if we have several images of the same object under different lightings
In this case we can solve for both lighting, albedo, and shape
This can be done by Factorization
Recall that
Ignore the case θ>90°

33. Photometric Stereo - Factorization
Goal: given M, find L and S
What should rank(M) be?

34. Photometric Stereo - Factorization
Use SVD to find a rank 3 approximation
Define So
Factorization is not unique, since
, A invertible
To reduce ambiguity we impose integrability

35. Reducing Ambiguity Assume We want to enforce integrability Notice that
Denote by the three rows of A, then
From which we obtain

36. Reducing Ambiguity Linear transformations of a surface
It can be shown that this is the only transformation that maintains integrability
Such transformations are called “generalized bas relief transformations” (GBR)
Thus, by imposing integrability the surface is reconstructed up to GBR

37. Relief Sculptures

38. Illumination Cone
Due to additivity, the set of images of an object under different lighting forms a convex cone in RN
This characterization is generic, holds also with specularities, shadows and inter-reflections
Unfortunately, representing the cone is complicated (infinite degree of freedom)
= 0.5*
+0.2*
+0.3*

39. Photobook/Eigenfaces (MIT Media Lab)

40. Recognition with PCA
Amano, Hiura, Yamaguti, and Inokuchi; Atick and Redlich; Bakry, Abo-Elsoud, and Kamel; Belhumeur, Hespanha, and Kriegman; Bhatnagar, Shaw, and Williams; Black and Jepson; Brennan and Principe; Campbell and Flynn; Casasent, Sipe and Talukder; Chan, Nasrabadi and Torrieri; Chung, Kee and Kim; Cootes, Taylor, Cooper and Graham; Covell; Cui and Weng; Daily and Cottrell; Demir, Akarun, and Alpaydin; Duta, Jain and Dubuisson-Jolly; Hallinan; Han and Tewfik; Jebara and Pentland; Kagesawa, Ueno, Kasushi, and Kashiwagi; King and Xu; Kalocsai, Zhao, and Elagin; Lee, Jung, Kwon and Hong; Liu and Wechsler; Menser and Muller; Moghaddam; Moon and Philips; Murase and Nayar; Nishino, Sato, and Ikeuchi; Novak, and Owirka; Nishino, Sato, and Ikeuchi; Ohta, Kohtaro and Ikeuchi; Ong and Gong; Penev and Atick; Penev and Sirivitch; Lorente and Torres; Pentland, Moghaddam, and Starner; Ramanathan, Sum, and Soon; Reiter and Matas; Romdhani, Gong and Psarrou; Shan, Gao, Chen, and Ma; Shen, Fu, Xu, Hsu, Chang, and Meng; Sirivitch and Kirby; Song, Chang, and Shaowei; Torres, Reutter, and Lorente; Turk and Pentland; Watta, Gandhi, and Lakshmanan; Weng and Chen; Yuela, Dai, and Feng; Yuille, Snow, Epstein, and Belhumeur; Zhao, Chellappa, and Krishnaswamy; Zhao and Yang…

41. Empirical Study (Yuille et al.) Ball Face Phone Parrot #1 48.2 53.7
67.9
42.8 #2
84.4
75.2
83.2
69.7 #3
94.4
90.2
88.2
76.3 #4
96.5
92.1
92.0
81.5 #5
97.9
93.5
94.1
84.7 #6
98.9
94.5
95.2
87.2 #7
99.1
95.3
96.3
88.5 #8
99.3
95.8
96.8
89.7 #9
99.5
97.2
90.7
#10
99.6
96.6
97.5
91.7
(Yuille et al.)

42. Intuition
lighting
reflectance

43. (Light → Reflectance) = Convolution+

44. (Light → Reflectance) = Convolution+

45. Spherical Harmonics
Positive values
Negative values 1 Z X Y XY XZ YZ

46. Harmonic Transform of Kernel

47. Cumulative Energy
(percents) N

48. Second Order Approximation

49. Yields 9D linear subspace.
Reflectance Near 9D
Yields 9D linear subspace.
4D approximation (first order) can also be used = point source + ambient

50. Harmonic Representations
ρ Albedo
n Surface normal
Positive values
Negative values r

51. SVD recovers L and S up to an ambiguity
Photometric Stereo S M L r
rnz
rny
r(3nz2-1)
r(nx2-ny2)
rnxny
rnxnz
rnynz
Image n :
Image 1
Light n :
Light 1 *
SVD recovers L and S up to an ambiguity

52. Photometric Stereo

53. Photometric Stereo

54. Summary Lighting effects are complex
Algorithms for SFS and photometric stereo for Lambertian object illuminated by a single light source
Harmonic analysis extends this to multiple light sources
Handling specularities, shadows, and inter-reflections is difficult