1. A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu Pat Hanrahan hanrahan@graphics.stanford.edu Pat Hanrahan hanrahan@graphics.stanford.edu

2. OutlineOutline Motivation Forward Rendering Inverse Rendering Object Recognition Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Motivation Forward Rendering Inverse Rendering Object Recognition Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary

3. Interactive Rendering Directional Source Complex Illumination Ramamoorthi and Hanrahan, SIGGRAPH 2001b

4. Reflection Maps Blinn and Newell, 1976

5. Environment Maps Miller and Hoffman, 1984 Later, Greene 86, Cabral et al. 87

6. Reflectance Space Shading Cabral, Olano, Nemec 1999

7. Reflectance Maps Reflectance Maps (Index by N) Horn, 1977 Irradiance (N) and Phong (R) Reflection Maps Hoffman and Miller, 1984 Reflectance Maps (Index by N) Horn, 1977 Irradiance (N) and Phong (R) Reflection Maps Hoffman and Miller, 1984 Mirror Sphere Chrome SphereMatte (Lambertian) Sphere Irradiance Environment Map

8. Complex Illumination Complex Illumination Must (pre)compute hemispherical integral of lighting Efficient Prefiltering (> 1000x faster) Traditionally, requires irradiance map textures Real-Time Procedural Rendering (no textures) New representation for lighting design, IBR Must (pre)compute hemispherical integral of lighting Efficient Prefiltering (> 1000x faster) Traditionally, requires irradiance map textures Real-Time Procedural Rendering (no textures) New representation for lighting design, IBR Directional Source Complex Lighting Illumination Irradiance Environment Map

9. Photorealistic Rendering Geometry 70 s, 80 s : Splines 90 s : Range Data Materials/Lighting (Texture, reflectance [BRDF], Lighting) Realistic Input Models Required Arnold Renderer: Marcos Fajardo Rendering Algorithm 80 s,90 s : Physically-based

10. Inverse Rendering How to measure realistic material models, lighting? From real photographs by inverse rendering Can then change viewpoint, lighting, reflectance Rendered images very realistic: they use real data How to measure realistic material models, lighting? From real photographs by inverse rendering Can then change viewpoint, lighting, reflectance Rendered images very realistic: they use real data Illumination: Mirror Sphere Grace Cathedral courtesy Paul Debevec BRDF (reflectance): Images using point light source

11. FlowchartFlowchart Lighting BRDF New View New View, Light

12. ResultsResults Photograph Computer rendering New view, new lighting Ramamoorthi and Hanrahan, SIGGRAPH 2001a

13. Inverse Rendering: Goals Complex (possibly unknown) illumination Estimate both lighting and reflectance (factorization) Complex (possibly unknown) illumination Estimate both lighting and reflectance (factorization) Photographs of 4 spheres in 3 different lighting conditions courtesy Dror and Adelson

14. Factorization Ambiguities Width of Light Source Surface Roughness

15. Inverse Problems Sometimes ill-posed No solution or several solutions given data Often numerically ill-conditioned Answer not robust, sensitive to noise Need general framework to address these issues Mathematical theory for complex illumination Sometimes ill-posed No solution or several solutions given data Often numerically ill-conditioned Answer not robust, sensitive to noise Need general framework to address these issues Mathematical theory for complex illumination Directional Source Area source Same reflectance

16. Lighting effects in recognition Space of Images (Lighting) is Infinite Dimensional Prior empirical work: 5D subspace captures variability We explain empirical data, subspace methods Space of Images (Lighting) is Infinite Dimensional Prior empirical work: 5D subspace captures variability We explain empirical data, subspace methods Peter Belhumeur: Yale Face Database A

17. OutlineOutline Motivation Signal Processing Framework: Reflection as Convolution Reflection Equation (2D) Fourier Analysis (2D) Spherical Harmonic Analysis (3D) Examples Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Motivation Signal Processing Framework: Reflection as Convolution Reflection Equation (2D) Fourier Analysis (2D) Spherical Harmonic Analysis (3D) Examples Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary

18. Reflection as Convolution (2D) L B BRDF

19. Fourier Analysis (2D)

20. Spherical Harmonics (3D) -2012 0 1 2...2...

21. Spherical Harmonic Analysis 2D: 3D: Isotropic

22. InsightsInsights Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Inverse rendering is deconvolution Our contribution: Formal Frequency-space analysis Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Inverse rendering is deconvolution Our contribution: Formal Frequency-space analysis

23. Example: Mirror BRDF BRDF is delta function Harmonic Transform is constant (infinite width) Reflected light field corresponds directly to lighting Mirror Sphere (Gazing Ball) BRDF is delta function Harmonic Transform is constant (infinite width) Reflected light field corresponds directly to lighting Mirror Sphere (Gazing Ball)

24. Phong, Microfacet Models Rough surfaces blur highlight Analytic Formula Approximately Gaussian Rough surfaces blur highlight Analytic Formula Approximately Gaussian Mirror Matte Roughness

25. Example: Lambertian BRDF Ramamoorthi and Hanrahan, JOSA 2001

26. Second-Order Approximation Lambertian: 9 parameters only order 2 approx. suffices Quadratic polynomial Lambertian: 9 parameters only order 2 approx. suffices Quadratic polynomial L=0 L=1L=2Exact -2012 0 1 2 Similar to Basri & Jacobs 01

27. Dual Representation Dual Representation Practical Representation Diffuse localized in frequency space Specular localized in angular space Dual Angular, Frequency-Space representation Practical Representation Diffuse localized in frequency space Specular localized in angular space Dual Angular, Frequency-Space representation Frequency 9 param. B B d diffuse B s,slow slow specular (area sources) B s,fast fast specular (directional) = + + Angular Space Frequency 9 param.

28. OutlineOutline Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary

29. VideoVideo Ramamoorthi and Hanrahan, SIGGRAPH 2001b

30. OutlineOutline Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary

31. Lighting effects in recognition Space of Images (Lighting) is Infinite Dimensional Prior empirical work: 5D subspace captures variability We explain empirical data, subspace methods Space of Images (Lighting) is Infinite Dimensional Prior empirical work: 5D subspace captures variability We explain empirical data, subspace methods Peter Belhumeur: Yale Face Database A

32. Face Basis Functions 5 basis functions capture 95% of image variability Linear combinations of spherical harmonics Complex illumination not much harder than points 5 basis functions capture 95% of image variability Linear combinations of spherical harmonics Complex illumination not much harder than points Frontal Lighting Side Above/Below Extreme Side Corner

33. Inverse Lighting Well-posed unless  equals zero in denominator Cannot recover radiance from irradiance: contradicts theorem in Preisendorfer 76 Well-conditioned unless  small BRDF should contain high frequencies : Sharp highlights Diffuse reflectors are ill-conditioned : Low pass filters Well-posed unless  equals zero in denominator Cannot recover radiance from irradiance: contradicts theorem in Preisendorfer 76 Well-conditioned unless  small BRDF should contain high frequencies : Sharp highlights Diffuse reflectors are ill-conditioned : Low pass filters

34. Inverse Lambertian Sum l=2Sum l=4 True Lighting Mirror Teflon

35. OutlineOutline Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary

36. Inverse Rendering: Goals Formal study: Well-posedness, conditioning General Complex (Unknown) Illumination Formal study: Well-posedness, conditioning General Complex (Unknown) Illumination Bronze Delrin PaintRough Steel Photographs Renderings (Recovered BRDF) Quantitative Pixel error approximately 5%

37. Factoring the Light Field The light field may be factored to estimate both the BRDF and the lighting Knowns B (4D) Unknowns L (2D)  (½ 3D) -- Make use of reciprocity The light field may be factored to estimate both the BRDF and the lighting Knowns B (4D) Unknowns L (2D)  (½ 3D) -- Make use of reciprocity

38. Algorithms Validation Rendering Unknown Lighting Photograph Known Lighting Kd 0.910.890.87 Ks 0.090.110.13  1.851.781.48  0.130.12.14 Recovered Light Estimate  by ratio of intensity and total energy Marschner

39. Complex Geometry 3 photographs of a sculpture Complex unknown illumination Geometry KNOWN Estimate BRDF and Lighting

40. FlowchartFlowchart Lighting BRDF New View New View, Light

41. ComparisonComparison Rendered Known Lighting Photograph Rendered Unknown Lighting

42. New View, Lighting Photograph Computer rendering

43. Textured Objects Real Rendering Complex, Known Lighting

44. OutlineOutline Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Conclusions Pointers Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Conclusions Pointers

45. SummarySummary Reflection as Convolution Signal-Processing Framework Frequency-space analysis yields insights Lambertian: approximated with 9 parameters Phong/Microfacet: acts like Gaussian filter Inverse Rendering Formal Study: Well-posedness, conditioning Dual Representations Practical Algorithms: Complex Lighting, Factorization Efficient Forward Rendering (Environment Maps) Lighting Variability in Object Recognition Reflection as Convolution Signal-Processing Framework Frequency-space analysis yields insights Lambertian: approximated with 9 parameters Phong/Microfacet: acts like Gaussian filter Inverse Rendering Formal Study: Well-posedness, conditioning Dual Representations Practical Algorithms: Complex Lighting, Factorization Efficient Forward Rendering (Environment Maps) Lighting Variability in Object Recognition

46. PapersPapers http://graphics.stanford.edu/~ravir/research.html Theory Flatland or 2D using Fourier analysis [SPIE 01] Lambertian: radiance from irradiance [JOSA 01] General 3D, Isotropic BRDFs [SIGGRAPH 01a] Applications Inverse Rendering [SIGGRAPH 01a] Forward Rendering [SIGGRAPH 01b] Lighting variability [In preparation] http://graphics.stanford.edu/~ravir/research.html Theory Flatland or 2D using Fourier analysis [SPIE 01] Lambertian: radiance from irradiance [JOSA 01] General 3D, Isotropic BRDFs [SIGGRAPH 01a] Applications Inverse Rendering [SIGGRAPH 01a] Forward Rendering [SIGGRAPH 01b] Lighting variability [In preparation]

47. AcknowledgementsAcknowledgements Marc Levoy Szymon Rusinkiewicz Steve Marschner John Parissenti Jean Gleason Scanned cat sculpture is “Serenity” by Sue Dawes Hodgson-Reed Stanford Graduate Fellowship NSF ITR grant #0085864: “Interacting with the Visual World” Marc Levoy Szymon Rusinkiewicz Steve Marschner John Parissenti Jean Gleason Scanned cat sculpture is “Serenity” by Sue Dawes Hodgson-Reed Stanford Graduate Fellowship NSF ITR grant #0085864: “Interacting with the Visual World”

48. The End

49. Related Work Graphics: Prefiltering Environment Maps Qualitative observation of reflection as convolution Miller and Hoffman 84, Greene 86 Cabral, Max, Springmeyer 87 (use spherical harmonics) Cabral et al. 99 Vision, Perception D’Zmura 91: Reflection as frequency-space operator Basri and Jacobs 01: Lambertian reflection as convolution Recognition: Appearance models e.g. Belhumeur et al. Graphics: Prefiltering Environment Maps Qualitative observation of reflection as convolution Miller and Hoffman 84, Greene 86 Cabral, Max, Springmeyer 87 (use spherical harmonics) Cabral et al. 99 Vision, Perception D’Zmura 91: Reflection as frequency-space operator Basri and Jacobs 01: Lambertian reflection as convolution Recognition: Appearance models e.g. Belhumeur et al.

50. Related Work Graphics: Prefiltering Environment Maps Qualitative observation of reflection as convolution Miller and Hoffman 84, Greene 86 Cabral, Max, Springmeyer 87 (use spherical harmonics) Cabral et al. 99 Vision, Perception D’Zmura 91: Reflection as frequency-space operator Basri and Jacobs 01: Lambertian reflection as convolution Recognition: Appearance models e.g. Belhumeur et al. Our Contributions Explicitly derive frequency-space convolution formula Formal Quantitative Analysis in General 3D Case Graphics: Prefiltering Environment Maps Qualitative observation of reflection as convolution Miller and Hoffman 84, Greene 86 Cabral, Max, Springmeyer 87 (use spherical harmonics) Cabral et al. 99 Vision, Perception D’Zmura 91: Reflection as frequency-space operator Basri and Jacobs 01: Lambertian reflection as convolution Recognition: Appearance models e.g. Belhumeur et al. Our Contributions Explicitly derive frequency-space convolution formula Formal Quantitative Analysis in General 3D Case

51. Example: Directional Source Lighting is delta function Harmonic Transform is constant (infinite width) Reflected light field corresponds directly to BRDF Impulse response of BRDF filter Lighting is delta function Harmonic Transform is constant (infinite width) Reflected light field corresponds directly to BRDF Impulse response of BRDF filter

52. Practical Issues Incomplete sparse data: Few views Use practical Dual Representation Incomplete sparse data: Few views Use practical Dual Representation B B d diffuse B s,slow slow specular (area sources) B s,fast fast specular (directional) = + + Angular Space Frequency

53. Practical Issues Incomplete sparse data: Few views Use practical Dual Representation Concavities: Self Shadowing Incomplete sparse data: Few views Use practical Dual Representation Concavities: Self Shadowing B B d diffuse B s,slow slow specular (area sources) B s,fast fast specular (directional) = + + Reflected Ray Shadowed? Integrate Lighting Source Shadowed?

54. Practical Issues Incomplete sparse data: Few views Use practical Dual Representation Concavities: Self Shadowing Textures: Spatially Varying Reflectance Incomplete sparse data: Few views Use practical Dual Representation Concavities: Self Shadowing Textures: Spatially Varying Reflectance B B d diffuse B s,slow slow specular (area sources) B s,fast fast specular (directional) = + + Reflected Ray Shadowed? Integrate Lighting Source Shadowed? K d (x) K s (x)

55. Inverse BRDF Well-conditioned unless L small Lighting should have sharp features (point sources, edges) Ill-conditioned for soft lighting Well-conditioned unless L small Lighting should have sharp features (point sources, edges) Ill-conditioned for soft lighting Directional Source Area source Same BRDF

56. ComparisonComparison Rendering Photograph Kd 0.910.89 Ks 0.090.11  1.851.78  0.130.12 Marschner Our method Known Lighting Pixel Error Approximately 5%