1. A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi Stanford University Columbia University: Feb 11, 2002

2. Illumination Illusion People perceive materials more easily under natural illumination than simplified illumination. Images courtesy Ron Dror and Ted Adelson

3. Illumination Illusion People perceive materials more easily under natural illumination than simplified illumination. Images courtesy Ron Dror and Ted Adelson

4. Material Recognition Photographs of 4 spheres in 3 different lighting conditions courtesy Dror and Adelson

5. Estimating BRDF and Lighting Photographs Geometric model Inverse Rendering Algorithm Lighting BRDF

6. Estimating BRDF and Lighting Photographs Geometric model Forward Rendering Algorithm Rendering Lighting BRDF

7. Estimating BRDF and Lighting Photographs Geometric model Forward Rendering Algorithm Rendering Novel lighting BRDF

8. Inverse Problems: Difficulties Angular width of Light Source Surface roughness Ill-posed (ambiguous)

9. Real-Time Rendering Interactive rendering with natural lighting, physical BRDFs

10. Motivation Understand nature of reflection and illumination Applications in computer graphics Real-time forward rendering Inverse rendering

11. Contributions of Thesis 1.Formalize reflection as convolution 2.Signal-processing framework 3.Practical forward and inverse algorithms

12. Outline Motivation Reflection as Convolution Preliminaries, assumptions Reflection equation, Fourier analysis (2D) Spherical Harmonic Analysis (3D) Signal-Processing Framework Applications Summary and Implications

13. Assumptions Known geometry Laser range scanner Structured light

14. Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Complex geometry: use surface normal

15. Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Distant illumination Photograph of mirror sphere Illumination: Grace Cathedral courtesy Paul Debevec

16. Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Distant illumination Homogeneous isotropic materials Isotropic Anisotropic

17. Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Distant illumination Homogeneous isotropic materials Later, practical algorithms: relax some assumptions

18. Reflection Lighting L Reflected Light Field BRDF B 

19. Reflection as Convolution (2D) Lighting Reflected Light Field BRDF L B B L L

20. Reflection as Convolution (2D) Lighting Reflected Light Field BRDF L B L B

21. Reflection as Convolution (2D) L B L B

22. Convolution x Signal f(x) Filter g(x) Output h(u) u

23. Convolution x Signal f(x) Filter g(x) Output h(u) u u1u1

24. Convolution x Signal f(x) Filter g(x) Output h(u) u u2u2

25. Convolution x Signal f(x) Filter g(x) Output h(u) u u3u3

26. Convolution x Signal f(x) Filter g(x) Output h(u) u Fourier analysis

27. Reflection as Convolution (2D) Frequency: product Spatial: integral Fourier analysis R. Ramamoorthi and P. Hanrahan “Analysis of Planar Light Fields from Homogeneous Convex Curved Surfaces under Distant Illumination” SPIE Photonics West 2001: Human Vision and Electronic Imaging VI pp 195-208 L B L B

28. Related Work Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99 Reflection as frequency-space operator: D’Zmura 91 Lambertian reflection is convolution: Basri Jacobs 01 Our Contributions Explicitly derive frequency-space convolution formula Formal quantitative analysis in general 3D case

29. Spherical Harmonics -2012 0 1 2...2...

30. Spherical Harmonic Analysis 2D: 3D:

31. Outline Motivation Reflection as Convolution Signal-Processing Framework Insights, examples Well-posedness of inverse problems Applications Summary and Implications

32. Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution

33. Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Filter is Delta function : Output = Signal Mirror BRDF : Image = Lighting [Miller and Hoffman 84] Image courtesy Paul Debevec

34. Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Signal is Delta function : Output = Filter Point Light Source : Images = BRDF [Marschner et al. 00]

35. Amplitude Frequency Roughness Phong, Microfacet Models Mirror Illumination estimation ill-posed for rough surfaces Analytic formulae in R. Ramamoorthi and P. Hanrahan “A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 2001 pp 117-128

36. Lambertian Incident radiance (mirror sphere) Irradiance (Lambertian) N 0 01 2 R. Ramamoorthi and P. Hanrahan “On the Relationship between Radiance and Irradiance: Determining the Illumination from Images of a Convex Lambertian Object” Journal of the Optical Society of America A 18(10) Oct 2001 pp 2448-2459 R. Basri and D. Jacobs “Lambertian Reflectance and Linear Subspaces” ICCV 2001 pp 383-390

37. Inverse Lighting Well-posed unless denominator vanishes BRDF should contain high frequencies : Sharp highlights Diffuse reflectors low pass filters: Inverse lighting ill-posed Given: B,ρ find L

38. Inverse BRDF Well-posed unless L lm vanishes Lighting should have sharp features (point sources, edges) BRDF estimation ill-conditioned for soft lighting Directional Source Area source Same BRDF Given: B,L find ρ

39. Factoring the Light Field Light Field can be factored Up to global scale factor Assumes reciprocity of BRDF Can be ill-conditioned Analytic formula derived Given: B find L and ρ 4D2D3D More knowns (4D) than unknowns (2D/3D)

40. Outline Motivation Reflection as Convolution Signal-Processing Framework Applications Forward rendering (convolution) Inverse rendering (deconvolution) Summary and Implications

41. Computing Irradiance Classically, hemispherical integral for each pixel Lambertian surface is like low pass filter Frequency-space analysis Incident Radiance Irradiance

42. 9 Parameter Approximation Exact image Order 0 1 term (constant) RMS error = 25 % -2012 0 1 2

43. 9 Parameter Approximation Exact image Order 1 4 terms (linear) RMS Error = 8% -2012 0 1 2

44. 9 Parameter Approximation Exact image Order 2 9 terms (quadratic) RMS Error = 1% For any illumination, average error < 2% [Basri Jacobs 01] -2012 0 1 2

45. Comparison Incident illumination 300x300 Irradiance map Texture: 256x256 Hemispherical Integration 2Hrs Irradiance map Texture: 256x256 Spherical Harmonic Coefficients 1sec

46. Video R. Ramamoorthi and P. Hanrahan “An Efficient Representation for Irradiance Environment Maps” SIGGRAPH 2001 pp 497-500 R. Ramamoorthi and P. Hanrahan “Frequency Space Environment Map Rendering” submitted

47. Video

48. Inverse Rendering Miller and Hoffman 84 Marschner and Greenberg 97 Sato et al. 97 Dana et al. 99 Debevec et al. 00 Marschner et al. 00 Sato et al. 99 Lighting BRDF Known Unknown Known Textures are a third axis

49. Contributions Complex illumination Factorization of BRDF, lighting (find both) New representations and algorithms Formal study of inverse problems (well-posed?)

50. Complications Incomplete sparse data (few photographs) Concavities: Self Shadowing Spatially varying BRDFs

51. Complications Challenge: Incomplete sparse data (few photographs) Difficult to compute frequency spectra Solution: Use parametric BRDF model Dual angular and frequency space representation

52. Algorithm Validation Photograph KdKd 0.91 KsKs 0.09 μ 1.85  0.13 “True” values

53. Algorithm Validation Renderings Unknown lighting Photograph Known lighting KdKd 0.910.890.87 KsKs 0.090.110.13 μ 1.851.781.48  0.130.120.14 “True” values Image RMS error 5%

54. Inverse BRDF: Spheres Bronze Delrin PaintRough Steel Photographs Renderings (Recovered BRDF)

55. Complications Challenge: Complex geometry with concavities Self shadowing Solution: Use associativity of convolution Blur lighting, treat specular BRDF term as mirror Single ray for shadowing, easy in ray tracer

56. Complex Geometry 3 photographs of a sculpture Complex unknown illumination Geometry known Estimate microfacet BRDF and distant lighting

57. Comparison Photograph Rendering

58. New View, Lighting Photograph Rendering

59. Complications Challenge: Spatially varying BRDFs Solution: Use textures to modulate BRDF parameters

60. Textured Objects Photograph Rendering

61. Summary Reflection as convolution Frequency-space analysis gives many insights Practical forward and inverse algorithms Signal-Processing: A useful paradigm for forward and inverse rendering in graphics and vision

62. Implications and Future Work Duality between forward and inverse problems: Ill-posed inverse problem  fast forward algorithm Example: Inverse lighting from Lambertian surface ill-posed  computing irradiance is fast

63. Implications and Future Work Differential framework for reflection Analysis on object surface Complex illumination and BRDF

64. Implications and Future Work Analyzing intrinsic structure, complexity of light field: Sampling theory based on signal-processing How many images in image-based rendering? How many principal components in PCA?

65. Lighting Invariant Recognition Theory: Space of images infinite-dimensional for Lambertian [Belhumeur and Kriegman 98] Empirical: 5D subspace enough for diffuse objects [Hallinan 94, Epstein et al. 95, BK98, …] Images from Yale face database

66. Implications and Future Work Complex illumination in computer vision Generally assume simple lighting (point source) without considering visibility (attached shadows) Signal processing can be used to reduce effects of complex illumination (with shadows) to low-dimensional subspace Many applications: stereo, photometric stereo, shape from shading, lighting invariant recognition etc.

67. Acknowledgements Pat Hanrahan Marc Levoy Szymon Rusinkiewicz Steve Marschner Stanford graphics group Hodgson-Reed Stanford Graduate Fellowship NSF ITR grant #0085864: “Interacting with the Visual World”

68. Papers R. Ramamoorthi and P. Hanrahan “A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 2001 pp 117-128 R. Ramamoorthi and P. Hanrahan “An Efficient Representation for Irradiance Environment Maps” SIGGRAPH 2001 pp 497-500 R. Ramamoorthi and P. Hanrahan “Frequency Space Environment Map Rendering” submitted R. Ramamoorthi and P. Hanrahan “On the Relationship between Radiance and Irradiance: Determining the Illumination from images of a Convex Lambertian Object” Journal of the Optical Society of America A 18(10) 2001 pp 2448-2459 R. Ramamoorthi and P. Hanrahan “Analysis of Planar Light Fields from Homogeneous Convex Curved Surfaces under Distant Illumination” SPIE Photonics West 2001: Human Vision and Electronic Imaging VI pp 195-208 R. Ramamoorthi “Analytic PCA Construction for Theoretical Analysis of Lighting Variability, Including Attached Shadows, in a Single Image of a Convex Lambertian Object” CVPR 2001 workshop on Identifying Objects across Lighting Variations pp 48-55 ravir@graphics.stanford.edu http://graphics.stanford.edu/~ravir

69. The End

70. Photorealistic Rendering Materials/Lighting (Texture Reflectance[BRDF] Lighting) Realistic input models required Arnold Renderer: Marcos Fajardo Rendering Algorithm 80’ s,90’ s : Physically based Geometry 70’ s, 80’ s : Splines 90’ s : Range Data

71. Measuring Materials, Light Measure BRDF (reflectance): Point light source Measure Illumination: Mirror Sphere Illumination: Grace Cathedral courtesy Paul Debevec

72. Interactive Forward Rendering Classically, rendering with natural illumination is very expensive compared to using simplified illumination Directional Source Natural Illumination

73. Lighting Invariant Recognition Theory: Infinite number of light directions Space of images infinite-dimensional Empirical: 5D subspace enough for diffuse objects Images from Debevec et al. 00

74. Lighting Invariant Recognition Theory: Space of images infinite-dimensional for Lambertian [Belhumeur and Kriegman 98] Empirical: 5D subspace enough for diffuse objects [Hallinan 94, Epstein et al. 95, BK98, …]

75. Open Questions Relationship between spherical harmonics, PCA 9D approximation > 5D empirical subspace Key insight: Consider approximations over visible normals (upper hemisphere), not entire sphere Frontal % VAF 42%33%16% 4%2% SideAbove/Below Extreme sideCorner Ramamoorthi CVPR IOAVL 01

76. Light Field in 3D In flatland, 2D function In three dimensions, 4D function Plenoptic Light Field Surface Light Field

77. Dual Representation Dual Representation Diffuse BRDF: Filter width small in frequency domain Specular: Filter width small in spatial (angular) domain Practical Representation: Dual angular, frequency-space B B d diffuse B s specular Angular = + Frequency

78. Related Work Precomputed (prefiltered) Irradiance maps [Miller and Hoffman 84, Greene 86, Cabral et al 87] Empirical observation: Irradiance varies slowly with surface normal. Use low resolution irradiance maps Contributions Analytic Irradiance formula Fast computation Compact 9 parameter representation Procedural rendering with programmable shading hardware Our approach can be extended to general BRDFs

79. Comparison Photograph Rendering (unknown L) Rendering (known L)