1. Shape from Shading #1

2. Topics u irradiance and radiance u basic concepts of reflection u reflection map u photometric stereo Reflectance map and Photometric stereo

3. Brightness irradiance amount of light falling on a surface falling energy measured by a unit surface area [watt/m 2 ] amount of light radiated from a surface emitting energy measured from a unit forshorted light source surface area to a unit solid angle [watt/ m 2 ・ Sr] solid angle --- steradian radiance R Θ A

4. Reflection geometry irradiance at a pixel depends on illumination materials geometry under the same illuminate condition, we observe irradiance difference on the same material surface there is a relationship between pixel irradiance and geometry Reflectance geometry L=illumination N=normal V=viewer L i=incidence angle e=emitting angle g=phase angle N Vie g

5. Gradient space reflection functions are defined in the local coordinate system(e,i,g) For our development, we will redefine the reflectance geometry in the gradient space viewers is always on the Z axis q p

6. Surface and body reflection u Surface reflection and body reflection surface reflection=gloss,highlights very directional(specular) body reflection =object color all direction(diffuse) plastic, paint have both metal has only surface reflection body air incident light surface reflection body reflection internal pigment

7. Model for body reflection Diffuse---scatters in all directions common approximation: equal in all directions “lambertian”Lambertian’s cosine law “perfectly diffuse reflector” reflectance=constant * geometric factor f(i,e,g) = Kb * cos i why cos i ? angle of incidence affects “density” of illumination.(irradiance) irradiance=light/area light=1 area=1/cos i irradiance = cos i

9. Calculating a reflection map (Lambertian) u for each(p,q), N=(p,q,1) u light source direction, S= iso-brightness contour p q 0.5 0.8 0.9

11. Reflectance map(continue) u Lambertian Self-shadow line p q

12. Surface reflection metals have the only surface reflection dielectrics(plastics,paint)have the surface reflection as well as the body reflection simplest approximation: perfect mirror reflection is specular direction, S’ S’ is coplanar with S,N SN = i = NS’: opposite sides S N S’ ii

13. Phong’s model calculate angle between S’ and V ---α f-surface(i,e,g)=Ks * cos α typical : n = 10 to 500 heuristic model tells amount of light at each angle n=1 Cos α 3 5 Real surfaces are rough : light scatters n n

15. Reflectance map bright dark q q p p

16. Better model of surface reflection u Phong’s model R=Ks cos n α not based on physics just looks OK for graphics, not really accurate off-specular effect Torrance and Sparrow --- geometrical optics Beckmann and Spizzichino --- physical optics composite surface reflection Phong’s model real surface

17. Torrance and Sparrow model u Geometrical optics –a collection of planar mirror-like facets –surface reflection caused only by these microfacets –their sizes are much larger than wave length average normal direction microfacet facet slopes to be normally distributed V-shaped valleys facet normal α

18. u surface reflectance = constant for material *effect of one ray *% not blocked by others (geometrical attenuation) *% of all facets involved Surface reflectance

19. Effect of one ray incoming energy = A cos i outgoing energy =(A cos i) / (A cos e) =cos i / cos e i e

20. Geometric attenuation 1) masking 2) shadowing g(i,e)

21. % of all facets involved α i reflection distribution facet normal distribution α N

22. Beckmann and Spizzichino model physical optics surface is continuous h(x,y) light is wave reflection off of surface roughness is amplitude and spatial frequency of variations in h(x,y) E(x,y,z) “field” of light energy surface is assumed to be a perfect conductor(metal) --- > Maxwell’s equation exact solution is vicious integral where

23. Our model (Nayer,Ikeuchi,Kanade89) Torrance and Sparrow + Beckmann and Spizzichino diffuse lobe --- cosine function specular spike --- delta function specular lobe --- gaussian function

25. recall Calculating reflectance map specular lobe + diffuse lobe Lambertian (diffuse lobe) contours Specular peak p q

26. Shape-from-shading recover object shape (orientation) from image irradiance (brightness) brightness surface orientation E(x,y)=R(p,q) -- image irradiance equation gives one constraint on the gradient space at each pixel --- > ill-posed problem (cannot solve !!!!!) (p,q,1) 0.8 p q

27. Photomotric stereo one image irradiant equation gives only one constraint --- > use multiple equations at each pixel. take multiple images from the same points under different light source directions recall different light source directions give different reflectance map at each pixel, multiple irradiance values p q p q

28. Photometric stereo (theory)

29. Photometric Stereo

30. Analytical solution real world gives complicated light source direction --- > look-up table method

31. Look-up table method

33. Summary Basic concepts of reflection radiance and irradiance reflection geometry surface reflection and body reflection Shape-from-shading problem reflectance map image irradiance equation photometric stereo

34. Shape-from-shading #2

35. get a depth map from a needle map get a needle map from a single image

36. Shape-from-shading #2 Gaussian sphere and reflectance map get a depth map from a needle map get a needle map from a single image

37. Depth from surface orientation 1 dimensional case recall 2 dimensional case 1

38. Recovering depth map from a needle map (direct integration method) Photometric stereo gives a needle map assume a depth at the origin get depth along the x of the needle map get the depth map

39. Direct integration rapid accumulates errors

40. Relaxation method u observed orientation (p,q) should be same as those of the depth map (z x,z y ) u reduce the total error within a boundary (the calculus of variations See Horn pp.469-474) u an iterative formula

41. iterative method needle map brightness depth map Relaxation method (Example)

42. Shape-from-shading with a single view Photometric stereo uses multiple images. Is there a way to recover shape from a single image? Yes, there is a way. 1. characteristics strip expansion method: obtain surface orientation along characteristics strips of image irradiance equation (Horn 75) 2. relaxation method:obtain surface orientation using image irradiance equation and smoothness constraint (Ikeuchi and Horn 81) 3. global method: assume a surface is a part of sphere (Pentland 83)

43. Characteristic strip expansion method the steep descent direction of the reflectance map (gradient space) the steep descent direction of the image brightness (image brightness)

44. SDD of RM SDD of IB SDD of RM SDD of IB move towards the SDD of the reflectance map on the image plane move towards the SDD of the image brightness on the gradient space Characteristic Strip

45. Proof 1. Taylor expansion of p(x,y) and q(x,y) 2. derivative of the image irradiance equation

46. Move towards the SDD of the reflectance map on the image plane then, what happen to (p(x,y),q(x,y)) ? move towards the SDD of the image brightness on the gradient space on the image on

47. p q 0.10.20.30.4 0.5 x y 0.1 0.2 0.3 0.4 from a known point, (you know (p,q) and E) you can determine (p,q) along a characteristic strip

48. x y character strip reconstructed contour q p Horn 75

49. Problem of characteristic stip method 1. Error accumulation. 2. The method starts from a singular point the start point is unreliable. 3. Determine surface orientation only along characteristic stripes. 4. Occluding contours are big evidences. We cannot use that information. (p,q) become infinite. relaxation method with occluding contours ???

50. Occluding boundary Surface orientations on occluding boundaries are known from the shape of silhouette. These surface orientations cannot be represented by the gradient space. (p,q) becomes infinite. We will use the stereographic plane, (f,g). On (f,g) plane, occluding boundaries lie on the unit circle. boundary condition

51. Stereographic Projection occluding boundaries lie on the unit circle gradient space p q gaussian sphere

52. Relaxation method 1. Image irradiance equation on (f,g) space on the (f,g) space, we can also define a reflectance map. 2. Smoothness constraint. Neighboring points have roughly the same surface orientation.

53. Relaxation method 3. Set up a minimization problem. 4. Using the calculus of variations get iterative formula. →min

54. n+h solution n+1+h solution brightness image occluding boundary needle map depth map Ikeuchi & Horn 81

55. Summary 1. Gaussian sphere and reflectance map 2. Get a depth map from a needle map 1. direct integration 2. relaxation method 3. Get a needle map from a single image 1. characteristic strip expansion method 2. relaxation method regularization 4. Read Horn pp. 244-269, B&B pp. 93-101

56. Color Theory for Computer Vision

57. Color in several domains: –Physics –Human vision –Psychophysics –Perception –Computer Vision Color problems in Computer Vision: –Color for segmentation –Color for reflection physics

58. Physics of Color Spectrum of electromagnetic radiation wavelength = characterized by spectral power distribution (SPD) S( ), E( ) 500nm600nm700nm 380nm 760nm violet blue green yellow orange red 10 -11 10 -9 10 -7 10 -6 10 -3 10 0 gammax-raysinfraredmicroradio ultra- violet visible monochromatic mixture (typical)

59. Physiology of human color vision retina The retina has four types of receptor cells. “red” cones “green” cones “blue” cones rods (darkness) color vision 380nm760nm red cone response = green cone response = blue cone response =

60. Color space red = green = blue = If we approximate spectral power distribution by vector, it’s a matrix multiplication. red green blue = spectral space : infinitely many dimensions color space : 3 dimensions green blue red each point is a “color” many SPDs have same “color”

61. Alternate color space other isomorphic color spaces formed by linear transforms red green blue = define new axes ABCABC = red green blue = = linear transform gives new axes new response function green blue red A C B

62. Psychophysical color X-Y-Z: international standard color space agreed upon by Commision Internationale de I’Eclairage (CIE) –particular linear transform of human cone responses –Two spectral distributions that result in the same values in the space appear indistinguishable –all colors have positive x, y, z Each point in X-Y-Z is a different color Chromaticity x = X / (X+Y+Z) ≒ R / (R+G+B) y = Y / (X+Y+Z) ≒ G / (R+G+B) z = Z / (X+Y+Z) ≒ B / (R+G+B) since x+y+z = 1, z = 1-(x+y). --- redundant usually plotted o x-y diagram Each point is many XYZ colors

63. Chromaticity diagram r = R / (R+G+B) g = G / (R+G+B) b = B / (R+G+B)

64. Color TV camera CCD filter electrons CCD sensor filter lens

65. Color TV RGB defined by National Television Systems Committee (NTSC) –a linear transform of X-Y-Z –each value limited 0 ⇔ 255 –commercial TV needs an intensity white-black component for black-and-white TV –also needs to span the color space u YIQ Y → main axis: white-black I → red-cyan Q → magenta-green magenta green black red white cyan blue Y I Q

66. Color perception How do people describe color ? NOT “X-Y-Z” nor “R-G-B” ! People use cylindrical coordinates. hue, saturation, brightness BH S blue white violet red yellow green S H One plane of constant brightness hue+saturation form polar coordinates relationship to red-green-blue

67. Hue-Saturation-Brightness Space blue black white hue

68. Role of Color in Robot Vision 1. Feature space for 2D segmentation more features → better discrimination 2. Color physics of reflection What physical information can color provide?

69. Segmentation Consider the entire image as a region and compute histograms for each of the color components. Apply a peak-finding test to each histogram. If at least one component passes the test, pick the component with the most significant peak and determine two thresholds, one either side of the peak. Use these thresholds to divide the region into subregions. Repeat these steps until no new subregions are created.

70. Color reflection physics surface reflection and body reflection u surface reflection has SPD of incident light u body reflection has SPD of body color body air incident light surface reflection body reflection internal pigment

71. Separating reflection components by color Pixel color vectors are Make a histogram fit parallelogram Project each pixel onto vectors Determine everywhere Klinker 88 body reflection surface reflection

72. Color space analysis body color vector in RGB space surface color vector in RGB space Color vector at a pixel is a linear combination of surface + body reflection color vector

73. Dichromatic Reflection Model u surface reflection has SPD of incident light u body reflection has SPD of body color surface reflectionbody reflection SPD of body colorSPD of incident light

74. Klinker/Shafer/Kanade 88 u Linear combination aS + bB Color Space green blue red Surface comp: S Body comp: B

75. Separation Results

76. Summary Color in several domains: physics human vision perception TV camera Color problems in computer vision color for segmentation color reflection physics

77. Reference u Horn, B.K.P “Obtaining shape from shading information,” in The Psychology of Computer Vision, P.H. Winston (ed.), McGraw-Hill, 1995 u Ikeuchi, K. & B.K.P. Horn, “Numerical shape from shading and occluding boundaries,” Artificial Intelligence, Vol. 17, 1981. u Pentland, A.P., “Local shading analysis,” IEEE Trans. PAMI, Vol.6, 1984. u Klinker, G.J., S.A. Shafer & T. Kanade, “The measurement of highlight in color image,” Int. J. Computer Vision, Vol.2, 1988. u Nayar S.K., K. Ikeuchi, and T. Kanade "Surface reflection: physical and geometrical perspectives", IEEE Trans. PAMI, Vol. 13, pp.661- 634, 1991.