1. Shape from Shading #1

2. Topics irradiance and radiance basic concepts of reflection
Reflectance map and Photometric stereo
irradiance and radiance
basic concepts of reflection
reflection map
photometric stereo

3. Brightness irradiance radiance amount of light falling on a surface
falling energy measured by a unit surface area [watt/m2]
radiance
amount of light radiated from a surface
emitting energy measured from a unit forshorted light source surface area to a unit solid angle [watt/ m2 ・Sr]
solid angle --- steradian R Θ A

4. Reflection geometry L N V i e g irradiance at a pixel depends on
illumination
materials
geometry
under the same illuminate condition, we observe irradiance difference on the same material surface
there is a relationship between pixel irradiance and geometry
Reflectance geometry L
i=incidence angle
e=emitting angle
g=phase angle N V i e g
L=illumination
N=normal
V=viewer

5. viewers is always on the Z axis
Gradient space
reflection functions are defined in the local coordinate system(e,i,g)
For our development, we will redefine the reflectance geometry in the gradient space
viewers is always on the Z axis q p

6. Surface and body reflection
Surface reflection and body reflection
body
air
incident light
surface
reflection
internal
pigment
surface reflection=gloss,highlights very directional(specular)
body reflection =object color all direction(diffuse)
plastic, paint have both
metal has only surface reflection

7. Model for body reflection
Diffuse---scatters in all directions
common approximation:
equal in all directions
“lambertian”Lambertian’s cosine law
“perfectly diffuse reflector”
reflectance=constant * geometric factor
f(i,e,g) = Kb * cos i
why cos i ?
angle of incidence affects “density” of illumination.(irradiance)
irradiance=light/area
light=1
area=1/cos i
irradiance = cos i

9. Calculating a reflection map (Lambertian)
for each(p,q), N=(p,q,1)
light source direction, S=
iso-brightness
contour p q
0.5
0.8
0.9

11. Reflectance map(continue)
Lambertian p q
Self-shadow line

12. Surface reflection metals have the only surface reflection
dielectrics(plastics,paint)have the surface reflection as well as the body reflection
simplest approximation: perfect mirror i i S N S’
reflection is specular direction, S’
S’ is coplanar with S,N
SN = i = NS’: opposite sides

13. Real surfaces are rough : light scatters
tells amount of light at each angle
Phong’s model
calculate angle between S’ and V ---α
f-surface(i,e,g)=Ks * cos α
typical : n = 10 to 500
heuristic model
n=1 3 n n
Cos α 5

15. Reflectance map q
bright
dark p q p

16. Better model of surface reflection
Phong’s model R=Ks cosnα not based on physics
just looks OK for graphics, not really accurate
off-specular effect
Phong’s model real surface
composite surface reflection
Phong’s model real surface
Torrance and Sparrow --- geometrical optics
Beckmann and Spizzichino --- physical optics

17. Torrance and Sparrow model
Geometrical optics
a collection of planar mirror-like facets
surface reflection caused only by these microfacets
their sizes are much larger than wave length
average normal direction α
facet normal
microfacet
facet slopes to be normally distributed
V-shaped valleys

18. surface reflectance = constant for material
effect of one ray
% not blocked by others (geometrical attenuation)
% of all facets involved

19. Effect of one ray incoming energy = A cos i outgoing energy
=(A cos i) / (A cos e)
=cos i / cos e e

20. Geometric attenuation
1) masking ) shadowing
g(i,e)

21. % of all facets involved
reflection distribution
facet normal distribution α N i α

22. Beckmann and Spizzichino model physical optics
surface is continuous h(x,y)
light is wave reflection off of surface
roughness is amplitude and spatial frequency of variations in h(x,y)
E(x,y,z) “field” of light energy
surface is assumed to be a perfect conductor(metal)
--- > Maxwell’s equation
exact solution is vicious integral
where

23. Our model (Nayer,Ikeuchi,Kanade89)
Torrance and Sparrow + Beckmann and Spizzichino
diffuse lobe --- cosine function
specular spike --- delta function
specular lobe --- gaussian function

25. Calculating reflectance map specular lobe + diffuse lobe
recall q
Lambertian (diffuse lobe) contours p
Specular peak

26. recover object shape (orientation) from image irradiance (brightness)
Shape-from-shading
recover object shape (orientation)
from image irradiance (brightness)
0.8
(p,q,1)
brightness surface orientation
E(x,y)=R(p,q) -- image irradiance equation
0.8 q p
gives one constraint on the gradient space at each pixel
--- > ill-posed problem (cannot solve !!!!!)

27. Photomotric stereo
one image irradiant equation gives only one constraint
--- > use multiple equations at each pixel.
take multiple images from the same points under different light source directions p q q p
recall different light source directions give different reflectance map
at each pixel, multiple irradiance values

28. Photometric　Stereo

29. Analytical solution
real world gives complicated light source direction
--- > look-up table method

30. Look-up table method

32. Summary Basic concepts of reflection Shape-from-shading problem
radiance and irradiance
reflection geometry
surface reflection and body reflection
Shape-from-shading problem
reflectance map
image irradiance equation
photometric stereo

33. Shape-from-shading #2

34. Shape-from-shading #2 get a depth map from a needle map
get a needle map from a single image

35. Depth from surface orientation
1 dimensional case
recall
2 dimensional case 1

36. Recovering depth map from a needle map (direct integration method)
Photometric stereo gives a needle map
assume a depth at the origin
get depth along the x of the needle map
get the depth map

37. Direct integration
rapid
accumulates errors

38. Relaxation method
observed orientation (p,q) should be same as those of the depth map (zx,zy)
reduce the total error within a boundary (the calculus of variations See Horn pp )
an iterative formula

39. Relaxation method (Example)
needle map
iterative method
brightness
depth map

40. Shape-from-shading with a single view
Photometric stereo uses multiple images.
Is there a way to recover shape from a single image?
Yes, there is a way.
1. characteristics strip expansion method:obtain surface orientation along characteristics strips of image irradiance equation (Horn 75)
2. relaxation method:obtain surface orientation using image irradiance equation and smoothness constraint (Ikeuchi and Horn 81)
3. global method: assume a surface is a part of sphere (Pentland 83)

41. Characteristic strip expansion method
the steep descent direction of the reflectance map
(gradient space)
the steep descent direction of the image brightness
(image brightness)

42. Characteristic Strip move towards the SDD of
the reflectance map on the image plane
the image brightness on the gradient space
SDD of RM
SDD of IB
SDD of RM
SDD of IB

43. Proof 1. Taylor expansion of p(x,y) and q(x,y)
2. derivative of the image irradiance equation

44. Move towards the SDD of the reflectance map on the image plane
then, what happen to (p(x,y),q(x,y)) ?
move towards the SDD of the image brightness on the gradient space
on the image on

45. x y p q from a known point, (you know (p,q) and E)
0.1
0.2
0.3
0.4 p q
0.1
0.2
0.3
0.4
0.5
from a known point, (you know (p,q) and E)
you can determine (p,q) along a characteristic strip

46. Horn 75 y x q p
character strip
reconstructed contour

47. Problem of characteristic stip method
1. Error accumulation.
2. The method starts from a singular point the start point is unreliable.
3. Determine surface orientation only along characteristic stripes.
4. Occluding contours are big evidences. We cannot use that information. (p,q) become infinite.
relaxation method with occluding contours
???

48. Occluding boundary
Surface orientations on occluding boundaries are known from the shape of silhouette.
These surface orientations cannot be represented by the gradient space. (p,q) becomes infinite.
We will use the stereographic plane, (f,g). On (f,g) plane, occluding boundaries lie on the unit circle.
boundary condition

49. Stereographic Projection
occluding boundaries lie on the unit circle p q
gradient
space
gaussian sphere

50. Relaxation method
1. Image irradiance equation on (f,g) space on the (f,g) space, we can also define a reflectance map.
2. Smoothness constraint. Neighboring points have roughly the same surface orientation.

51. Relaxation method 3. Set up a minimization problem.
4. Using the calculus of variations get iterative formula.
→min

52. Ikeuchi & Horn 81 brightness image depth map needle map
occluding boundary
n+h
solution
n+1+h

53. Summary 1. Gaussian sphere and reflectance map
2. Get a depth map from a needle map
1. direct integration
2. relaxation method
3. Get a needle map from a single image
1. characteristic strip expansion method
regularization
4. Read Horn pp , B&B pp

54. Color Theory for Computer Vision

55. Color Theory for Computer Vision
Color in several domains:
Physics
Human vision
Psychophysics
Perception
Computer Vision
Color problems in Computer Vision:
Color for segmentation
Color for reflection physics

56. Physics of Color Spectrum of electromagnetic radiation
wavelength =
characterized by spectral power distribution (SPD)
S( ), E( )
500nm
600nm
700nm
380nm
760nm
violet blue green yellow orange red
10-11
10-9
10-7
10-6
10-3
100
gamma
x-rays
infrared
micro
radio
ultra-
violet
visible
mixture (typical)
monochromatic

57. Physiology of human color vision
retina
The retina has four types of
receptor cells.
“red” cones
“green” cones
“blue” cones
rods (darkness)
color vision
380nm
760nm
red cone response =
green cone response =
blue cone response =

58. Color space If we approximate spectral power red =
green =
blue =
If we approximate spectral power
distribution by vector, it’s a
matrix multiplication.
red
green
blue =
spectral space : infinitely many dimensions
color space : 3 dimensions
green
blue
red
each point is a “color”
many SPDs have same “color”

59. Alternate color space
other isomorphic color spaces formed by linear transforms
green
blue
red A C B
red
green
blue =
define new axes A B C =
red
green
blue
linear transform gives new axes =
new response function =

60. Psychophysical color X-Y-Z:
international standard color space agreed upon by
Commision Internationale de I’Eclairage (CIE)
particular linear transform of human cone responses
Two spectral distributions that result in the same values in the space appear indistinguishable
all colors have positive x, y, z
Each point in X-Y-Z is a different color
Chromaticity
x = X / (X+Y+Z) ≒ R / (R+G+B)
y = Y / (X+Y+Z) ≒ G / (R+G+B)
z = Z / (X+Y+Z) ≒ B / (R+G+B)
since x+y+z = 1, z = 1-(x+y). --- redundant
usually plotted o x-y diagram
Each point is many XYZ colors

61. Chromaticity diagram
r = R / (R+G+B)
g = G / (R+G+B)
b = B / (R+G+B)

62. Color TV camera
electrons
CCD sensor
filter
lens
CCD
filter

63. Color TV YIQ RGB defined by
National Television Systems Committee (NTSC)
a linear transform of X-Y-Z
each value limited 0⇔255
commercial TV needs an intensity white-black component for black-and-white TV
also needs to span the color space
YIQ
Y → main axis: white-black
I → red-cyan
Q → magenta-green
magenta
green
black
red
white
cyan
blue Y I Q

64. Color perception How do people describe color ?
NOT “X-Y-Z” nor “R-G-B” !
People use cylindrical coordinates.
hue, saturation, brightness
One plane of constant brightness
blue
white
violet
red
yellow
green S H B H S
hue+saturation form polar coordinates
relationship to red-green-blue

65. Hue-Saturation-Brightness Space
blue
hue
white
black

66. Role of Color in Robot Vision
1. Feature space for 2D segmentation
more features → better discrimination
2. Color physics of reflection
What physical information can color provide?

67. Color reflection physics
surface reflection and body reflection
surface reflection has SPD of incident light
body reflection has SPD of body color
surface
reflection
body
reflection
incident light
internal
pigment
air
body

68. Separating reflection components by color
Pixel color vectors are
Make a histogram fit parallelogram
Project each pixel onto vectors
Determine everywhere
Klinker 88
body reflection
surface reflection

69. Color space analysis body color vector in RGB space
surface color vector in RGB space
Color vector at a pixel is a linear combination of
surface + body reflection color vector

70. Dichromatic Reflection Model
surface reflection has SPD of incident light
body reflection has SPD of body color
surface reflection
body reflection
SPD of incident light
SPD of body color

71. Color Space Surface comp: S Body comp: B Klinker/Shafer/Kanade 88
Linear combination aS + bB
Surface comp: S
Body comp: B
green
blue
red

72. Separation Results

73. Summary Color in several domains: Color problems in computer vision
physics
human vision
perception
TV camera
Color problems in computer vision
color for segmentation
color reflection physics

74. Reference
Horn, B.K.P “Obtaining shape from shading information,” in The Psychology of Computer Vision, P.H. Winston (ed.), McGraw-Hill, 1995
Ikeuchi, K. & B.K.P. Horn, “Numerical shape from shading and occluding boundaries,” Artificial Intelligence, Vol. 17, 1981.
Pentland, A.P., “Local shading analysis,” IEEE Trans. PAMI, Vol.6, 1984.
Klinker, G.J., S.A. Shafer & T. Kanade, “The measurement of highlight in color image,” Int. J. Computer Vision, Vol.2, 1988.
Nayar S.K., K. Ikeuchi, and T. Kanade "Surface reflection: physical and geometrical perspectives", IEEE Trans. PAMI, Vol. 13, pp , 1991.