1. Shape from Shading #1

2. Topics u irradiance and radiance u basic concepts of reflection u reflection map u photometric stereo Reflectance map and Photometric stereo

3. Brightness irradiance amount of light falling on a surface falling energy measured by a unit surface area [watt/m 2 ] amount of light radiated from a surface emitting energy measured from a unit forshorted light source surface area to a unit solid angle [watt/ m 2 ・ Sr] solid angle --- steradian radiance R Θ A

4. Reflection geometry irradiance at a pixel depends on illumination materials geometry under the same illuminate condition, we observe irradiance difference on the same material surface there is a relationship between pixel irradiance and geometry Reflectance geometry L=illumination N=normal V=viewer L i=incidence angle e=emitting angle g=phase angle N Vie g

5. Gradient space reflection functions are defined in the local coordinate system(e,i,g) For our development, we will redefine the reflectance geometry in the gradient space viewers is always on the Z axis q p

6. Surface and body reflection u Surface reflection and body reflection surface reflection=gloss,highlights very directional(specular) body reflection =object color all direction(diffuse) plastic, paint have both metal has only surface reflection body air incident light surface reflection body reflection internal pigment

7. Model for body reflection Diffuse---scatters in all directions common approximation: equal in all directions “lambertian”Lambertian’s cosine law “perfectly diffuse reflector” reflectance=constant * geometric factor f(i,e,g) = Kb * cos i why cos i ? angle of incidence affects “density” of illumination.(irradiance) irradiance=light/area light=1 area=1/cos i irradiance = cos i

9. Calculating a reflection map (Lambertian) u for each(p,q), N=(p,q,1) u light source direction, S= iso-brightness contour p q 0.5 0.8 0.9

11. Reflectance map(continue) u Lambertian Self-shadow line p q

12. Surface reflection metals have the only surface reflection dielectrics(plastics,paint)have the surface reflection as well as the body reflection simplest approximation: perfect mirror reflection is specular direction, S’ S’ is coplanar with S,N SN = i = NS’: opposite sides S N S’ ii

13. Phong’s model calculate angle between S’ and V ---α f-surface(i,e,g)=Ks * cos α typical : n = 10 to 500 heuristic model tells amount of light at each angle n=1 Cos α 3 5 Real surfaces are rough : light scatters n n

15. Reflectance map bright dark q q p p

16. Better model of surface reflection u Phong’s model R=Ks cos n α not based on physics just looks OK for graphics, not really accurate off-specular effect Torrance and Sparrow --- geometrical optics Beckmann and Spizzichino --- physical optics composite surface reflection Phong’s model real surface

17. Torrance and Sparrow model u Geometrical optics –a collection of planar mirror-like facets –surface reflection caused only by these microfacets –their sizes are much larger than wave length average normal direction microfacet facet slopes to be normally distributed V-shaped valleys facet normal α

18. u surface reflectance = constant for material *effect of one ray *% not blocked by others (geometrical attenuation) *% of all facets involved Surface reflectance

19. Effect of one ray incoming energy = A cos i outgoing energy =(A cos i) / (A cos e) =cos i / cos e i e

20. Geometric attenuation 1) masking 2) shadowing g(i,e)

21. % of all facets involved α i reflection distribution facet normal distribution α N

22. Beckmann and Spizzichino model physical optics surface is continuous h(x,y) light is wave reflection off of surface roughness is amplitude and spatial frequency of variations in h(x,y) E(x,y,z) “field” of light energy surface is assumed to be a perfect conductor(metal) --- > Maxwell’s equation exact solution is vicious integral where

23. Our model (Nayer,Ikeuchi,Kanade89) Torrance and Sparrow + Beckmann and Spizzichino diffuse lobe --- cosine function specular spike --- delta function specular lobe --- gaussian function

25. recall Calculating reflectance map specular lobe + diffuse lobe Lambertian (diffuse lobe) contours Specular peak p q

26. Shape-from-shading recover object shape (orientation) from image irradiance (brightness) brightness surface orientation E(x,y)=R(p,q) -- image irradiance equation gives one constraint on the gradient space at each pixel --- > ill-posed problem (cannot solve !!!!!) (p,q,1) 0.8 p q

27. Photomotric stereo one image irradiant equation gives only one constraint --- > use multiple equations at each pixel. take multiple images from the same points under different light source directions recall different light source directions give different reflectance map at each pixel, multiple irradiance values p q p q

28. Photometric Stereo

29. Analytical solution real world gives complicated light source direction --- > look-up table method

30. Look-up table method

32. Summary Basic concepts of reflection radiance and irradiance reflection geometry surface reflection and body reflection Shape-from-shading problem reflectance map image irradiance equation photometric stereo

33. Shape-from-shading #2

34. get a depth map from a needle map get a needle map from a single image

35. Depth from surface orientation 1 dimensional case recall 2 dimensional case 1

36. Recovering depth map from a needle map (direct integration method) Photometric stereo gives a needle map assume a depth at the origin get depth along the x of the needle map get the depth map

37. Direct integration rapid accumulates errors

38. Relaxation method u observed orientation (p,q) should be same as those of the depth map (z x,z y ) u reduce the total error within a boundary (the calculus of variations See Horn pp.469-474) u an iterative formula

39. iterative method needle map brightness depth map Relaxation method (Example)

40. Jacobi’s method Iteratively computing the equation itself  convergence is very slow The number of iteration needed to converge is when the error decreased to 10 -p times, for the pixels whose size is J×J Eg. r=1200000 when J=400, p=15

41. Gauss-Seidel method Eg. r=600000 when J=400, p=15 Jacobi: Gauss-Seidel: scan-line ordercheckerboard pattern Iteration number:

42. SOR (successive overrelaxation) method ω: overrelaxation parameter Converge if 0<ω<2 0<ω<1 (underrelaxation): slower than Gauss-Seidel 1<ω<2 (overrelaxation): faster than Gauss-Seidel Optimal ω:Eg. ω=1.984 when J=400 Eg. r=2000 when J=400, p=15Iteration number: Chebyshev acceleration (faster convergence): change ω properly for each iteration same as Gauss-Seidel

43. ADI (alternating-direction implicit) method Eg. r=104 when J=400, p=15Iteration number:  : changed properly for each iteration if  =0  Gauss-Seidel  +2 … … H(1,y) H(J,y) … … * … … *  +2  +2 … … H(x,1) H(x,J) … … * … … *  +2 A: tridiagonal matrix (given) b: vector (given) x: solve Ax=b by using linear system solver (forward substitution)  O(N)

44. Natural boundary condition s… arc length of boundary Natural boundary condition of "gradient-to-height problem" is ·… dot product … the normal of boundary Algorithm: [Truth] [Wrong result] [Correct result] Same height boundary Natural boundary condition Fast convergenceSlow convergence Boundary: Height known  just use it Height unknown  natural boundary condition Calculate the height "H" at the boundary by orfor each iteration

45. Shape-from-shading with a single view Photometric stereo uses multiple images. Is there a way to recover shape from a single image? Yes, there is a way. 1. characteristics strip expansion method: obtain surface orientation along characteristics strips of image irradiance equation (Horn 75) 2. relaxation method:obtain surface orientation using image irradiance equation and smoothness constraint (Ikeuchi and Horn 81) 3. global method: assume a surface is a part of sphere (Pentland 83)

46. Characteristic strip expansion method the steep descent direction of the reflectance map (gradient space) the steep descent direction of the image brightness (image brightness)

47. SDD of RM SDD of IB SDD of RM SDD of IB move towards the SDD of the reflectance map on the image plane move towards the SDD of the image brightness on the gradient space Characteristic Strip

48. Proof 1. Taylor expansion of p(x,y) and q(x,y) 2. derivative of the image irradiance equation

49. Move towards the SDD of the reflectance map on the image plane then, what happen to (p(x,y),q(x,y)) ? move towards the SDD of the image brightness on the gradient space on the image on

50. p q 0.10.20.30.4 0.5 x y 0.1 0.2 0.3 0.4 from a known point, (you know (p,q) and E) you can determine (p,q) along a characteristic strip

51. x y character strip reconstructed contour q p Horn 75

52. Problem of characteristic stip method 1. Error accumulation. 2. The method starts from a singular point the start point is unreliable. 3. Determine surface orientation only along characteristic stripes. 4. Occluding contours are big evidences. We cannot use that information. (p,q) become infinite. relaxation method with occluding contours ???

53. Occluding boundary Surface orientations on occluding boundaries are known from the shape of silhouette. These surface orientations cannot be represented by the gradient space. (p,q) becomes infinite. We will use the stereographic plane, (f,g). On (f,g) plane, occluding boundaries lie on the unit circle. boundary condition

54. Stereographic Projection occluding boundaries lie on the unit circle gradient space p q gaussian sphere

55. Relaxation method 1. Image irradiance equation on (f,g) space on the (f,g) space, we can also define a reflectance map. 2. Smoothness constraint. Neighboring points have roughly the same surface orientation.

56. Relaxation method 3. Set up a minimization problem. 4. Using the calculus of variations get iterative formula. →min

57. n+h solution n+1+h solution brightness image occluding boundary needle map depth map Ikeuchi & Horn 81

58. Summary 1. Gaussian sphere and reflectance map 2. Get a depth map from a needle map 1. direct integration 2. relaxation method 3. Get a needle map from a single image 1. characteristic strip expansion method 2. relaxation method regularization 4. Read Horn pp. 244-269, B&B pp. 93-101

59. Shape-from-shading #3

60. –Shape from shading u More advanced researches –4-light photometric stereo –Extended light source –Photometric sampling –Shape from interreflection –Inverse polarization raytracing specular spike interreflection + transmission diffuse interreflection diffuse diffuse + specular specular spike diffuse + specular spike

61. 4-light photometric stereo [diffuse pixel] 4 correct answer small deviation [specular pixel] 1 correct answer large deviation

62. 4-light photometric stereo [Coleman&Jain] [diffuse pixel] Average of four [specular pixel] Use 3 dark pixels Deviation Smaller than thresholdLarger than threshold

63. 4-light photometric stereo [Barsky&Petrou] specular diffuse length threshold [specular] dark 3 [non-specular] bright 3 [specular pixel][diffuse pixel][shadow pixel] light specular normalsurface normal difference

64. 4-light photometric stereo [Solomon&Ikeuchi] 4 3 3 3 3 2 2 2 2 Gaussian sphere [4 light region] [Coleman&Jain] [3 light region] Use two lights p q Gradient space Two possible normals Choose negative shading of shadow light shadow specular [2 light region] Use two lightsTwo possible normals diffuse p q Gradient space Choose negative shading of shadow light shadow diffuse

65. Planar extended light source L l0l0 X Brightness: f Line lamp Specular object Lambertian plane (Extended light source) Camera r Irradiance: E(f,r,X,l 0,L)

66. Reflectance map 3 linear lamps3 reflectance mapsSolve by photometric stereo

67. Photometric sampler

68. Spherical extended light source L(  )=L(  ;  s,R,H,I,C) C: constant I: radiant intensity of light S  : termination angle

69. Photometric function for Lambertian surface Lambertian Lambertian of extended light Estimate reflectance A' and surface normal  n by fitting cosine function to images

70. Photometric function for specular surface Specular spike Specular spike of extended light Point sources are separated by source termination angle  Only two specular peaks appear

71. Photometric function for specular surface Point sources are separated by source termination angle  1-to-1 correspondence  n is calculated is calculated

72. Photometric function for hybrid surface 1.Remove 2 intensities 2.Calculate  nl by fitting Lambertian model from remaining intensities 3.Calculate  ns from 2 intensities Calculate (  nl,  ns ) for all possibilities Choose  where Finally,

73. Example

74. Interreflection

75. Form factor dx dx'r  '' Interreflection kernel: Visibility function: V=1 if visible, V=0 if invisible Radiance of xIrradiance of x Albedo Radiance of x'

76. Interreflection equation Facet radiance vector:Source contribution vector: Albedo matrix:Kernel matrix:

77. Inverse radiosity (Shading) Facet matrix: Albedo Surface normal Source direction vector: Pseudo facet matrix: (Pseudo shading) Estimate F p by conventional photometric stereo (L: input image, s: known light source) Estimate F iteratively from initial value F p andis straighforwardly calculated from Recovery algorithm:

78. Example

79. Application to book scanning Image scanner Scanning a book Observed image Estimated shape Restored image

80. Polarization  Light = wave  oscillates  Oscillates in certain direction  polarization u DOP = degree of polarization Unpolarized (DOP 0) Light Perfectly polarized (DOP 1) Polarizer Partially polarized (DOP 0~1) Incident Reflected Air Object Transmitted

81. Reflection and transmission Normal Unpolarized Air Object Partially polarized Light Depends upon

82. Polarization raytracing Calculate polarization (Mueller calculus) Calculate reflection & transmission (Ray-tracing) Light (4D vector) Reflection&transmission (4x4 matrix)

83. Minimization Solve by Shape-From-Shading technique Polarization raytracing equation InputCalculated by polarization raytracing

84. Example Result (10 iteration) Glass (refractive index 1.5)

85. Summary u 4-light photometric stereo –Photometric stereo from proper 3 lights u Extended light source –Photometric stereo from Lambertian planar light source u Photometric sampling –Shape and reflectance from spherical extended light source u Shape from interreflection –Photometric stereo for concave diffuse object u Inverse polarization raytracing –Shape of transparent object by using SFS technique

86. Reference u Horn, B.K.P. "Robot Vision," MIT Press, 1986 u Zhang, R., Tsai, P.S., Cryer, J.D., Shah, M., "Shape from Shading: A Survey," IEEE Trans. PAMI, Vol. 21, No. 8, pp. 690-706, 1999. u Horn, B.K.P “Obtaining shape from shading information,” in The Psychology of Computer Vision, P.H. Winston (ed.), McGraw-Hill, 1995 u Ikeuchi, K. & B.K.P. Horn, “Numerical shape from shading and occluding boundaries,” Artificial Intelligence, Vol. 17, 1981. u Pentland, A.P., “Local shading analysis,” IEEE Trans. PAMI, Vol.6, 1984. u Klinker, G.J., S.A. Shafer & T. Kanade, “The measurement of highlight in color image,” Int. J. Computer Vision, Vol.2, 1988. u Nayar S.K., K. Ikeuchi, and T. Kanade "Surface reflection: physical and geometrical perspectives", IEEE Trans. PAMI, Vol. 13, pp.661- 634, 1991.