1. A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi Ph.D. Orals: June 3, 2002

2. Illumination Illusion People perceive materials more easily under natural illumination than simplified illumination. Images courtesy Ron Dror and Ted Adelson

3. Illumination Illusion People perceive materials more easily under natural illumination than simplified illumination. Images courtesy Ron Dror and Ted Adelson

4. Material Recognition Photographs of 4 spheres in 3 different lighting conditions courtesy Dror and Adelson

5. Estimating BRDF and Lighting Photographs Geometric model Inverse Rendering Algorithm Lighting BRDF

6. Estimating BRDF and Lighting Photographs Geometric model Forward Rendering Algorithm Rendering Lighting BRDF

7. Estimating BRDF and Lighting Photographs Geometric model Forward Rendering Algorithm Rendering Novel lighting BRDF

8. Inverse Problems: Difficulties Angular width of Light Source Surface roughness Ill-posed (ambiguous)

9. Real-Time Rendering Interactive rendering with natural lighting, physical BRDFs

10. Contributions 1.Formalize reflection as convolution 2.Signal-processing framework 3.Practical forward and inverse algorithms

11. Outline Motivation Reflection as Convolution Preliminaries, assumptions Reflection equation, Fourier analysis (2D) Spherical Harmonic Analysis (3D) Signal-Processing Framework Applications Summary and Implications

12. Assumptions Known geometry Laser range scanner Structured light

13. Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Complex geometry: use surface normal

14. Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Distant illumination Photograph of mirror sphere Illumination: Grace Cathedral courtesy Paul Debevec

15. Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Distant illumination Homogeneous isotropic materials Isotropic Anisotropic

16. Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Distant illumination Homogeneous isotropic materials Later, practical algorithms: relax some assumptions

17. Reflection Lighting L Reflected Light Field BRDF B 

18. Reflection as Convolution (2D) Lighting Reflected Light Field BRDF L B B L L

19. Reflection as Convolution (2D) Lighting Reflected Light Field BRDF L B L B

20. Reflection as Convolution (2D) L B L B

21. Frequency: product Spatial: integral Fourier analysis R. Ramamoorthi and P. Hanrahan “Analysis of Planar Light Fields from Homogeneous Convex Curved Surfaces under Distant Illumination” SPIE Photonics West 2001: Human Vision and Electronic Imaging VI pp 195-208 L B L B

22. Related Work Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99 Reflection as frequency-space operator: D’Zmura 91 Lambertian reflection is convolution: Basri Jacobs 01 Our Contributions Explicitly derive frequency-space convolution formula Formal quantitative analysis in general 3D case

23. Spherical Harmonics -2012 0 1 2...2...

24. Spherical Harmonic Analysis 2D: 3D:

25. Outline Motivation Reflection as Convolution Signal-Processing Framework Insights, examples Well-posedness of inverse problems Applications Summary and Implications

26. Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution

27. Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Filter is Delta function : Output = Signal Mirror BRDF : Image = Lighting [Miller and Hoffman 84] Image courtesy Paul Debevec

28. Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Signal is Delta function : Output = Filter Point Light Source : Images = BRDF [Marschner et al. 00]

29. Amplitude Frequency Roughness Phong, Microfacet Models Mirror Illumination estimation ill-posed for rough surfaces Analytic formulae in R. Ramamoorthi and P. Hanrahan “A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 2001 pp 117-128

30. Lambertian Incident radiance (mirror sphere) Irradiance (Lambertian) N 0 01 2 R. Ramamoorthi and P. Hanrahan “On the Relationship between Radiance and Irradiance: Determining the Illumination from Images of a Convex Lambertian Object” Journal of the Optical Society of America A 18(10) Oct 2001 pp 2448-2459 R. Basri and D. Jacobs “Lambertian Reflectance and Linear Subspaces” ICCV 2001 pp 383-390

31. Inverse Lighting Well-posed unless denominator vanishes BRDF should contain high frequencies : Sharp highlights Diffuse reflectors low pass filters: Inverse lighting ill-posed Given: B,ρ find L

32. Inverse BRDF Well-posed unless L lm vanishes Lighting should have sharp features (point sources, edges) BRDF estimation ill-conditioned for soft lighting Directional Source Area source Same BRDF Given: B,L find ρ

33. Factoring the Light Field Light Field can be factored Up to global scale factor Assumes reciprocity of BRDF Can be ill-conditioned Analytic formula derived Given: B find L and ρ 4D2D3D More knowns (4D) than unknowns (2D/3D)

34. Outline Motivation Reflection as Convolution Signal-Processing Framework Applications Forward rendering (convolution) Inverse rendering (deconvolution) Summary and Implications

35. Computing Irradiance Classically, hemispherical integral for each pixel Lambertian surface is like low pass filter Frequency-space analysis Incident Radiance Irradiance

36. 9 Parameter Approximation Exact image Order 0 1 term (constant) RMS error = 25 % -2012 0 1 2

37. 9 Parameter Approximation Exact image Order 1 4 terms (linear) RMS Error = 8% -2012 0 1 2

38. 9 Parameter Approximation Exact image Order 2 9 terms (quadratic) RMS Error = 1% For any illumination, average error < 2% [Basri Jacobs 01] -2012 0 1 2

39. Comparison Incident illumination 300x300 Irradiance map Texture: 256x256 Hemispherical Integration 2Hrs Irradiance map Texture: 256x256 Spherical Harmonic Coefficients 1sec

40. Video R. Ramamoorthi and P. Hanrahan “An Efficient Representation for Irradiance Environment Maps” SIGGRAPH 2001 pp 497-500 R. Ramamoorthi and P. Hanrahan “Frequency Space Environment Map Rendering” SIGGRAPH 2002

41. Video

42. Inverse Rendering: Previous Work Miller and Hoffman 84 Marschner and Greenberg 97 Yu and Malik 98 Yu et al. 99 Marschner et al. 00 Boivin et al. 01 Sato et al. 99 Lighting BRDF Known Unknown Known Textures are a third axis

43. Contributions Complex illumination Factorization of BRDF, lighting (find both) New representations and algorithms Formal study of inverse problems (well-posed?)

44. Complications Incomplete sparse data (few photographs) Concavities: Self Shadowing Spatially varying BRDFs

45. Complications Challenge: Incomplete sparse data (few photographs) Difficult to compute frequency spectra Solution: Use parametric BRDF model Dual angular and frequency space representation

46. Algorithm Validation Photograph KdKd 0.91 KsKs 0.09 μ 1.85  0.13 “True” values

47. Algorithm Validation Renderings Unknown lighting Photograph Known lighting KdKd 0.910.890.87 KsKs 0.090.110.13 μ 1.851.781.48  0.130.120.14 “True” values Image RMS error 5%

48. Inverse BRDF: Spheres Bronze Delrin PaintRough Steel Photographs Renderings (Recovered BRDF)

49. Complications Challenge: Complex geometry with concavities Self shadowing Solution: Use associativity of convolution Blur lighting, treat specular BRDF term as mirror Single ray for shadowing, easy in ray tracer

50. Complex Geometry 3 photographs of a sculpture Complex unknown illumination Geometry known Estimate microfacet BRDF and distant lighting

51. Comparison Photograph Rendering

52. New View, Lighting Photograph Rendering

53. Complications Challenge: Spatially varying BRDFs Solution: Use textures to modulate BRDF parameters

54. Textured Objects Photograph Rendering

55. Summary Reflection as convolution Frequency-space analysis gives many insights Practical forward and inverse algorithms Signal-Processing: A useful paradigm for forward and inverse rendering in graphics and vision

56. Implications and Future Work Duality between forward and inverse problems Analyzing intrinsic structure of light field How many images in image-based rendering? How many principal components in PCA? Differential framework for reflection Complex illumination in computer vision

57. Papers R. Ramamoorthi and P. Hanrahan “A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 2001 pp 117-128 R. Ramamoorthi and P. Hanrahan “An Efficient Representation for Irradiance Environment Maps” SIGGRAPH 2001 pp 497-500 R. Ramamoorthi and P. Hanrahan “Frequency Space Environment Map Rendering” SIGGRAPH 2002 R. Ramamoorthi and P. Hanrahan “On the Relationship between Radiance and Irradiance: Determining the Illumination from images of a Convex Lambertian Object” Journal of the Optical Society of America A 18(10) 2001 pp 2448-2459 R. Ramamoorthi and P. Hanrahan “Analysis of Planar Light Fields from Homogeneous Convex Curved Surfaces under Distant Illumination” SPIE Photonics West 2001: Human Vision and Electronic Imaging VI 195-208 ravir@graphics.stanford.edu http://graphics.stanford.edu/~ravir

58. Acknowledgements Pat Hanrahan Committee: Marc, Jitendra, Ron, Bernd Szymon Rusinkiewicz and Steve Marschner Bill Mark and Kekoa Proudfoot GLAB gerth, ada, heather, seander, maneesh, henrik, jedavis, olaf, humper, dk, renng, tpurcell, psen, vaibhav, zsh, munzner, lucasp, liyiwei, … Hodgson-Reed Stanford Graduate Fellowship NSF ITR grant #0085864: “Interacting with the Visual World”

59. The End

60. Motivation Understand nature of reflection and illumination Applications in computer graphics Real-time forward rendering Inverse rendering

61. Photorealistic Rendering Materials/Lighting (Texture Reflectance[BRDF] Lighting) Realistic input models required Arnold Renderer: Marcos Fajardo Rendering Algorithm 80’ s,90’ s : Physically based Geometry 70’ s, 80’ s : Splines 90’ s : Range Data

62. Measuring Materials, Light Measure BRDF (reflectance): Point light source Measure Illumination: Mirror Sphere Illumination: Grace Cathedral courtesy Paul Debevec

63. Interactive Forward Rendering Classically, rendering with natural illumination is very expensive compared to using simplified illumination Directional Source Natural Illumination

64. Lighting Invariant Recognition Theory: Infinite number of light directions Space of images infinite-dimensional Empirical: 5D subspace enough for diffuse objects Images from Debevec et al. 00

65. Lighting Invariant Recognition Theory: Space of images infinite-dimensional for Lambertian [Belhumeur and Kriegman 98] Empirical: 5D subspace enough for diffuse objects [Hallinan 94, Epstein et al. 95]

66. Open Questions Relationship between spherical harmonics, PCA 9D approximation > 5D empirical subspace Key insight: Consider approximations over visible normals (upper hemisphere), not entire sphere Frontal % VAF 42%33%16% 4%2% SideAbove/Below Extreme sideCorner Ramamoorthi CVPR IOAVL 01

67. Light Field in 3D In flatland, 2D function In three dimensions, 4D function Plenoptic Light Field Surface Light Field

68. Dual Representation Dual Representation Diffuse BRDF: Filter width small in frequency domain Specular: Filter width small in spatial (angular) domain Practical Representation: Dual angular, frequency-space B B d diffuse B s specular Angular = + Frequency

69. Related Work Precomputed (prefiltered) Irradiance maps [Miller and Hoffman 84, Greene 86, Cabral et al 87] Empirical observation: Irradiance varies slowly with surface normal. Use low resolution irradiance maps Contributions Analytic Irradiance formula Fast computation Compact 9 parameter representation Procedural rendering with programmable shading hardware Our approach can be extended to general BRDFs

70. Comparison Photograph Rendering (unknown L) Rendering (known L)