1. MAN-522 Computer Vision
Spring

2. Shape from shading (Photometric stereo)
Given a single image of an object with known surface reflectance taken under a known light source, can we recover the shape of the object?

3. Definition 1:
Irradiance is the radiant flux (power) received by a surface per unit area. The SI unit of irradiance is the watt per square metre (W/m2).
Irradiance is often called "intensity" in branches of physics other than radiometry, but in radiometry this usage leads to confusion with radiant intensity.
Definition 2:
In radiometry, radiance is the radiant flux emitted, reflected, transmitted or received by a surface, per unit solid angle per unit projected area,

4. Mechanisms of Reflection
source
incident
direction
surface
reflection
body
reflection
surface
Body Reflection:
Diffuse Reflection
Matte Appearance
Non-Homogeneous Medium
Clay, paper, etc
Surface Reflection:
Specular Reflection
Glossy Appearance
Highlights
Dominant for Metals
Image Intensity = Body Reflection + Surface Reflection

5. Diffuse Reflection and Lambertian BRDF
source intensity I
incident
direction
normal
viewing
direction
surface
element
Surface appears equally bright from ALL directions! (independent of )
albedo
Lambertian BRDF is simply a constant :
Surface Radiance :
source intensity
Commonly used in Vision and Graphics!

6. Diffuse and Specular Reflection
diffuse+specular

7. Image Intensity and 3D Geometry
Shading as a cue for shape reconstruction
What is the relation between intensity and shape?
Reflectance Map

8. Surface Normal
surface normal
Equation of plane or
Let
Surface normal

9. Gradient Space Normal vector Source vector
plane is called the Gradient Space (pq plane)
Every point on it corresponds to a particular surface orientation

10. Reflectance Map
Relates image irradiance I(x,y) to surface orientation (p,q) for given source direction and surface reflectance
Lambertian case:
: source brightness
: surface albedo (reflectance)
: constant (optical system)
Image irradiance:
Let
then

11. Reflectance Map Lambertian case Reflectance Map (Lambertian)
Iso-brightness contour
cone of constant

12. Reflectance Map Lambertian case Note: is maximum when iso-brightness
contour
Note: is maximum when

13. Reflectance Map Glossy surfaces (Torrance-Sparrow reflectance model)
diffuse term
specular term
Diffuse peak
Specular peak

14. Shape from a Single Image?
Given a single image of an object with known surface reflectance taken under a known light source, can we recover the shape of the object?
Given R(p,q) ( (pS,qS) and surface reflectance) can we determine (p,q) uniquely for each image point? NO

15. Solution Take more images Add more constraints Photometric stereo
Shape-from-shading

16. Photometric Stereo

17. Photometric Stereo We can write this in matrix form: Lambertian case:
Image irradiance:
We can write this in matrix form:

18. Solving the Equations
inverse

19. More than Three Light Sources
Get better results by using more lights
Least squares solution:
Moore-Penrose pseudo inverse
Solve for as before

20. Color Images The case of RGB images
get three sets of equations, one per color channel:
Simple solution: first solve for using one channel
Then substitute known into above equations to get
Or combine three channels and solve for

21. Computing light source directions
Trick: place a chrome sphere in the scene
the location of the highlight tells you the source direction

22. Specular Reflection - Recap
For a perfect mirror, light is reflected about N
We see a highlight when
Then is given as follows:

23. Computing the Light Source Direction
Chrome sphere that has a highlight at position h in the image N H h rN C c
sphere in 3D
image plane
Can compute N by studying this figure
Hints:
use this equation:
can measure c, h, and r in the image

24. Depth from Normals Get a similar equation for V2
Each normal gives us two linear constraints on z
compute z values by solving a matrix equation

25. Limitations Big problems Smaller problems
Doesn’t work for shiny things, semi-translucent things
Shadows, inter-reflections
Smaller problems
Camera and lights have to be distant
Calibration requirements
measure light source directions, intensities
camera response function

26. Trick for Handling Shadows
Weight each equation by the pixel brightness:
Gives weighted least-squares matrix equation:
Solve for as before

27. Original Images

28. Results - Shape Shallow reconstruction (effect of interreflections)
Accurate reconstruction
(after removing interreflections)

29. Results - Albedo
No Shading Information

30. Original Images

31. Results - Shape

32. Results - Albedo

33. Results Estimate light source directions Compute surface normals
Compute albedo values
Estimate depth from surface normals
Relight the object (with original texture and uniform albedo)

34. Shape from a Single Image?
Given a single image of an object with known surface reflectance taken under a known light source, can we recover the shape of the object?
Given R(p,q) ( (pS,qS) and surface reflectance) can we determine (p,q) uniquely for each image point? NO

35. Stereographic Projection
(p,q)-space (gradient space)
(f,g)-space
Problem
(p,q) can be infinite when
Redefine reflectance map as

36. Occluding Boundaries
and are known
The values on the occluding boundary can be used as
the boundary condition for shape-from-shading

37. Image Irradiance Constraint
Image irradiance should match the reflectance map
Minimize
(minimize errors in image irradiance in the image)

38. Smoothness Constraint
Used to constrain shape-from-shading
Relates orientations (f,g) of neighboring surface points
Minimize
: surface orientation under stereographic projection
(penalize rapid changes in surface orientation f and g over the image)

39. Shape-from-Shading
Find surface orientations (f,g) at all image points that minimize
weight
smoothness
constraint
image irradiance
error
Minimize

40. Numerical Shape-from-Shading
Smoothness error at image point (i,j)
Of course you can consider more neighbors (smoother results)
Image irradiance error at image point (i,j)
Find and that minimize
(Ikeuchi & Horn 89)

41. Numerical Shape-from-Shading
Find and that minimize
If and minimize , then
where and are 4-neighbors average around image point (k,l)
(Ikeuchi & Horn 89)

42. Numerical Shape-from-Shading
Update rule
Use known values on the occluding boundary to constrain the solution (boundary conditions)
Compare with for convergence test
As the solution converges, increase to remove the smoothness constraint
(Ikeuchi & Horn 89)

43. Calculus of Variations
Minimize
Euler equations for F
(read Horn A.6)
Euler equations for shape-from-shading
Solve this coupled pair of second-order partial differential equations
with the occluding boundary conditions!

44. Results

45. Results