A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi Ravi Ramamoorthi Pat Hanrahan Pat Hanrahan

OutlineOutline Motivation Forward Rendering Inverse Rendering Object Recognition Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Motivation Forward Rendering Inverse Rendering Object Recognition Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary

Interactive Rendering Directional Source Complex Illumination Ramamoorthi and Hanrahan, SIGGRAPH 2001b

Reflection Maps Blinn and Newell, 1976

Environment Maps Miller and Hoffman, 1984 Later, Greene 86, Cabral et al. 87

Reflectance Space Shading Cabral, Olano, Nemec 1999

Reflectance Maps Reflectance Maps (Index by N) Horn, 1977 Irradiance (N) and Phong (R) Reflection Maps Hoffman and Miller, 1984 Reflectance Maps (Index by N) Horn, 1977 Irradiance (N) and Phong (R) Reflection Maps Hoffman and Miller, 1984 Mirror Sphere Chrome SphereMatte (Lambertian) Sphere Irradiance Environment Map

Complex Illumination Complex Illumination Must (pre)compute hemispherical integral of lighting Efficient Prefiltering (> 1000x faster) Traditionally, requires irradiance map textures Real-Time Procedural Rendering (no textures) New representation for lighting design, IBR Must (pre)compute hemispherical integral of lighting Efficient Prefiltering (> 1000x faster) Traditionally, requires irradiance map textures Real-Time Procedural Rendering (no textures) New representation for lighting design, IBR Directional Source Complex Lighting Illumination Irradiance Environment Map

Photorealistic Rendering Geometry 70 s, 80 s : Splines 90 s : Range Data Materials/Lighting (Texture, reflectance [BRDF], Lighting) Realistic Input Models Required Arnold Renderer: Marcos Fajardo Rendering Algorithm 80 s,90 s : Physically-based

Inverse Rendering How to measure realistic material models, lighting? From real photographs by inverse rendering Can then change viewpoint, lighting, reflectance Rendered images very realistic: they use real data How to measure realistic material models, lighting? From real photographs by inverse rendering Can then change viewpoint, lighting, reflectance Rendered images very realistic: they use real data Illumination: Mirror Sphere Grace Cathedral courtesy Paul Debevec BRDF (reflectance): Images using point light source

FlowchartFlowchart Lighting BRDF New View New View, Light

ResultsResults Photograph Computer rendering New view, new lighting Ramamoorthi and Hanrahan, SIGGRAPH 2001a

Inverse Rendering: Goals Complex (possibly unknown) illumination Estimate both lighting and reflectance (factorization) Complex (possibly unknown) illumination Estimate both lighting and reflectance (factorization) Photographs of 4 spheres in 3 different lighting conditions courtesy Dror and Adelson

Factorization Ambiguities Width of Light Source Surface Roughness

Inverse Problems Sometimes ill-posed No solution or several solutions given data Often numerically ill-conditioned Answer not robust, sensitive to noise Need general framework to address these issues Mathematical theory for complex illumination Sometimes ill-posed No solution or several solutions given data Often numerically ill-conditioned Answer not robust, sensitive to noise Need general framework to address these issues Mathematical theory for complex illumination Directional Source Area source Same reflectance

Lighting effects in recognition Space of Images (Lighting) is Infinite Dimensional Prior empirical work: 5D subspace captures variability We explain empirical data, subspace methods Space of Images (Lighting) is Infinite Dimensional Prior empirical work: 5D subspace captures variability We explain empirical data, subspace methods Peter Belhumeur: Yale Face Database A

OutlineOutline Motivation Signal Processing Framework: Reflection as Convolution Reflection Equation (2D) Fourier Analysis (2D) Spherical Harmonic Analysis (3D) Examples Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Motivation Signal Processing Framework: Reflection as Convolution Reflection Equation (2D) Fourier Analysis (2D) Spherical Harmonic Analysis (3D) Examples Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary

Reflection as Convolution (2D) L B BRDF

Fourier Analysis (2D)

Spherical Harmonics (3D)

Spherical Harmonic Analysis 2D: 3D: Isotropic

InsightsInsights Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Inverse rendering is deconvolution Our contribution: Formal Frequency-space analysis Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Inverse rendering is deconvolution Our contribution: Formal Frequency-space analysis

Example: Mirror BRDF BRDF is delta function Harmonic Transform is constant (infinite width) Reflected light field corresponds directly to lighting Mirror Sphere (Gazing Ball) BRDF is delta function Harmonic Transform is constant (infinite width) Reflected light field corresponds directly to lighting Mirror Sphere (Gazing Ball)

Phong, Microfacet Models Rough surfaces blur highlight Analytic Formula Approximately Gaussian Rough surfaces blur highlight Analytic Formula Approximately Gaussian Mirror Matte Roughness

Example: Lambertian BRDF Ramamoorthi and Hanrahan, JOSA 2001

Second-Order Approximation Lambertian: 9 parameters only order 2 approx. suffices Quadratic polynomial Lambertian: 9 parameters only order 2 approx. suffices Quadratic polynomial L=0 L=1L=2Exact Similar to Basri & Jacobs 01

Dual Representation Dual Representation Practical Representation Diffuse localized in frequency space Specular localized in angular space Dual Angular, Frequency-Space representation Practical Representation Diffuse localized in frequency space Specular localized in angular space Dual Angular, Frequency-Space representation Frequency 9 param. B B d diffuse B s,slow slow specular (area sources) B s,fast fast specular (directional) = + + Angular Space Frequency 9 param.

OutlineOutline Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary

VideoVideo Ramamoorthi and Hanrahan, SIGGRAPH 2001b

OutlineOutline Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary

Lighting effects in recognition Space of Images (Lighting) is Infinite Dimensional Prior empirical work: 5D subspace captures variability We explain empirical data, subspace methods Space of Images (Lighting) is Infinite Dimensional Prior empirical work: 5D subspace captures variability We explain empirical data, subspace methods Peter Belhumeur: Yale Face Database A

Face Basis Functions 5 basis functions capture 95% of image variability Linear combinations of spherical harmonics Complex illumination not much harder than points 5 basis functions capture 95% of image variability Linear combinations of spherical harmonics Complex illumination not much harder than points Frontal Lighting Side Above/Below Extreme Side Corner

Inverse Lighting Well-posed unless  equals zero in denominator Cannot recover radiance from irradiance: contradicts theorem in Preisendorfer 76 Well-conditioned unless  small BRDF should contain high frequencies : Sharp highlights Diffuse reflectors are ill-conditioned : Low pass filters Well-posed unless  equals zero in denominator Cannot recover radiance from irradiance: contradicts theorem in Preisendorfer 76 Well-conditioned unless  small BRDF should contain high frequencies : Sharp highlights Diffuse reflectors are ill-conditioned : Low pass filters

Inverse Lambertian Sum l=2Sum l=4 True Lighting Mirror Teflon

OutlineOutline Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary

Inverse Rendering: Goals Formal study: Well-posedness, conditioning General Complex (Unknown) Illumination Formal study: Well-posedness, conditioning General Complex (Unknown) Illumination Bronze Delrin PaintRough Steel Photographs Renderings (Recovered BRDF) Quantitative Pixel error approximately 5%

Factoring the Light Field The light field may be factored to estimate both the BRDF and the lighting Knowns B (4D) Unknowns L (2D)  (½ 3D) -- Make use of reciprocity The light field may be factored to estimate both the BRDF and the lighting Knowns B (4D) Unknowns L (2D)  (½ 3D) -- Make use of reciprocity

Algorithms Validation Rendering Unknown Lighting Photograph Known Lighting Kd Ks   Recovered Light Estimate  by ratio of intensity and total energy Marschner

Complex Geometry 3 photographs of a sculpture Complex unknown illumination Geometry KNOWN Estimate BRDF and Lighting

FlowchartFlowchart Lighting BRDF New View New View, Light

ComparisonComparison Rendered Known Lighting Photograph Rendered Unknown Lighting

New View, Lighting Photograph Computer rendering

Textured Objects Real Rendering Complex, Known Lighting

OutlineOutline Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Conclusions Pointers Motivation Reflection as Convolution Efficient Rendering: Environment Maps Lighting Variability in Object Recognition Deconvolution, Inverse Rendering Summary Conclusions Pointers

SummarySummary Reflection as Convolution Signal-Processing Framework Frequency-space analysis yields insights Lambertian: approximated with 9 parameters Phong/Microfacet: acts like Gaussian filter Inverse Rendering Formal Study: Well-posedness, conditioning Dual Representations Practical Algorithms: Complex Lighting, Factorization Efficient Forward Rendering (Environment Maps) Lighting Variability in Object Recognition Reflection as Convolution Signal-Processing Framework Frequency-space analysis yields insights Lambertian: approximated with 9 parameters Phong/Microfacet: acts like Gaussian filter Inverse Rendering Formal Study: Well-posedness, conditioning Dual Representations Practical Algorithms: Complex Lighting, Factorization Efficient Forward Rendering (Environment Maps) Lighting Variability in Object Recognition

PapersPapers Theory Flatland or 2D using Fourier analysis [SPIE 01] Lambertian: radiance from irradiance [JOSA 01] General 3D, Isotropic BRDFs [SIGGRAPH 01a] Applications Inverse Rendering [SIGGRAPH 01a] Forward Rendering [SIGGRAPH 01b] Lighting variability [In preparation] Theory Flatland or 2D using Fourier analysis [SPIE 01] Lambertian: radiance from irradiance [JOSA 01] General 3D, Isotropic BRDFs [SIGGRAPH 01a] Applications Inverse Rendering [SIGGRAPH 01a] Forward Rendering [SIGGRAPH 01b] Lighting variability [In preparation]

AcknowledgementsAcknowledgements Marc Levoy Szymon Rusinkiewicz Steve Marschner John Parissenti Jean Gleason Scanned cat sculpture is “Serenity” by Sue Dawes Hodgson-Reed Stanford Graduate Fellowship NSF ITR grant # : “Interacting with the Visual World” Marc Levoy Szymon Rusinkiewicz Steve Marschner John Parissenti Jean Gleason Scanned cat sculpture is “Serenity” by Sue Dawes Hodgson-Reed Stanford Graduate Fellowship NSF ITR grant # : “Interacting with the Visual World”

The End

Related Work Graphics: Prefiltering Environment Maps Qualitative observation of reflection as convolution Miller and Hoffman 84, Greene 86 Cabral, Max, Springmeyer 87 (use spherical harmonics) Cabral et al. 99 Vision, Perception D’Zmura 91: Reflection as frequency-space operator Basri and Jacobs 01: Lambertian reflection as convolution Recognition: Appearance models e.g. Belhumeur et al. Graphics: Prefiltering Environment Maps Qualitative observation of reflection as convolution Miller and Hoffman 84, Greene 86 Cabral, Max, Springmeyer 87 (use spherical harmonics) Cabral et al. 99 Vision, Perception D’Zmura 91: Reflection as frequency-space operator Basri and Jacobs 01: Lambertian reflection as convolution Recognition: Appearance models e.g. Belhumeur et al.

Related Work Graphics: Prefiltering Environment Maps Qualitative observation of reflection as convolution Miller and Hoffman 84, Greene 86 Cabral, Max, Springmeyer 87 (use spherical harmonics) Cabral et al. 99 Vision, Perception D’Zmura 91: Reflection as frequency-space operator Basri and Jacobs 01: Lambertian reflection as convolution Recognition: Appearance models e.g. Belhumeur et al. Our Contributions Explicitly derive frequency-space convolution formula Formal Quantitative Analysis in General 3D Case Graphics: Prefiltering Environment Maps Qualitative observation of reflection as convolution Miller and Hoffman 84, Greene 86 Cabral, Max, Springmeyer 87 (use spherical harmonics) Cabral et al. 99 Vision, Perception D’Zmura 91: Reflection as frequency-space operator Basri and Jacobs 01: Lambertian reflection as convolution Recognition: Appearance models e.g. Belhumeur et al. Our Contributions Explicitly derive frequency-space convolution formula Formal Quantitative Analysis in General 3D Case

Example: Directional Source Lighting is delta function Harmonic Transform is constant (infinite width) Reflected light field corresponds directly to BRDF Impulse response of BRDF filter Lighting is delta function Harmonic Transform is constant (infinite width) Reflected light field corresponds directly to BRDF Impulse response of BRDF filter

Practical Issues Incomplete sparse data: Few views Use practical Dual Representation Incomplete sparse data: Few views Use practical Dual Representation B B d diffuse B s,slow slow specular (area sources) B s,fast fast specular (directional) = + + Angular Space Frequency

Practical Issues Incomplete sparse data: Few views Use practical Dual Representation Concavities: Self Shadowing Incomplete sparse data: Few views Use practical Dual Representation Concavities: Self Shadowing B B d diffuse B s,slow slow specular (area sources) B s,fast fast specular (directional) = + + Reflected Ray Shadowed? Integrate Lighting Source Shadowed?

Practical Issues Incomplete sparse data: Few views Use practical Dual Representation Concavities: Self Shadowing Textures: Spatially Varying Reflectance Incomplete sparse data: Few views Use practical Dual Representation Concavities: Self Shadowing Textures: Spatially Varying Reflectance B B d diffuse B s,slow slow specular (area sources) B s,fast fast specular (directional) = + + Reflected Ray Shadowed? Integrate Lighting Source Shadowed? K d (x) K s (x)

Inverse BRDF Well-conditioned unless L small Lighting should have sharp features (point sources, edges) Ill-conditioned for soft lighting Well-conditioned unless L small Lighting should have sharp features (point sources, edges) Ill-conditioned for soft lighting Directional Source Area source Same BRDF

ComparisonComparison Rendering Photograph Kd Ks   Marschner Our method Known Lighting Pixel Error Approximately 5%