A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi Stanford University Columbia University: Feb 11, 2002

Illumination Illusion People perceive materials more easily under natural illumination than simplified illumination. Images courtesy Ron Dror and Ted Adelson

Illumination Illusion People perceive materials more easily under natural illumination than simplified illumination. Images courtesy Ron Dror and Ted Adelson

Material Recognition Photographs of 4 spheres in 3 different lighting conditions courtesy Dror and Adelson

Estimating BRDF and Lighting Photographs Geometric model Inverse Rendering Algorithm Lighting BRDF

Estimating BRDF and Lighting Photographs Geometric model Forward Rendering Algorithm Rendering Lighting BRDF

Estimating BRDF and Lighting Photographs Geometric model Forward Rendering Algorithm Rendering Novel lighting BRDF

Inverse Problems: Difficulties Angular width of Light Source Surface roughness Ill-posed (ambiguous)

Real-Time Rendering Interactive rendering with natural lighting, physical BRDFs

Motivation Understand nature of reflection and illumination Applications in computer graphics Real-time forward rendering Inverse rendering

Contributions of Thesis 1.Formalize reflection as convolution 2.Signal-processing framework 3.Practical forward and inverse algorithms

Outline Motivation Reflection as Convolution Preliminaries, assumptions Reflection equation, Fourier analysis (2D) Spherical Harmonic Analysis (3D) Signal-Processing Framework Applications Summary and Implications

Assumptions Known geometry Laser range scanner Structured light

Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Complex geometry: use surface normal

Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Distant illumination Photograph of mirror sphere Illumination: Grace Cathedral courtesy Paul Debevec

Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Distant illumination Homogeneous isotropic materials Isotropic Anisotropic

Assumptions Known geometry Convex curved surfaces: no shadows, interreflection Distant illumination Homogeneous isotropic materials Later, practical algorithms: relax some assumptions

Reflection Lighting L Reflected Light Field BRDF B 

Reflection as Convolution (2D) Lighting Reflected Light Field BRDF L B B L L

Reflection as Convolution (2D) Lighting Reflected Light Field BRDF L B L B

Reflection as Convolution (2D) L B L B

Convolution x Signal f(x) Filter g(x) Output h(u) u

Convolution x Signal f(x) Filter g(x) Output h(u) u u1u1

Convolution x Signal f(x) Filter g(x) Output h(u) u u2u2

Convolution x Signal f(x) Filter g(x) Output h(u) u u3u3

Convolution x Signal f(x) Filter g(x) Output h(u) u Fourier analysis

Reflection as Convolution (2D) Frequency: product Spatial: integral Fourier analysis R. Ramamoorthi and P. Hanrahan “Analysis of Planar Light Fields from Homogeneous Convex Curved Surfaces under Distant Illumination” SPIE Photonics West 2001: Human Vision and Electronic Imaging VI pp L B L B

Related Work Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99 Reflection as frequency-space operator: D’Zmura 91 Lambertian reflection is convolution: Basri Jacobs 01 Our Contributions Explicitly derive frequency-space convolution formula Formal quantitative analysis in general 3D case

Spherical Harmonics

Spherical Harmonic Analysis 2D: 3D:

Outline Motivation Reflection as Convolution Signal-Processing Framework Insights, examples Well-posedness of inverse problems Applications Summary and Implications

Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution

Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Filter is Delta function : Output = Signal Mirror BRDF : Image = Lighting [Miller and Hoffman 84] Image courtesy Paul Debevec

Insights: Signal Processing Signal processing framework for reflection Light is the signal BRDF is the filter Reflection on a curved surface is convolution Signal is Delta function : Output = Filter Point Light Source : Images = BRDF [Marschner et al. 00]

Amplitude Frequency Roughness Phong, Microfacet Models Mirror Illumination estimation ill-posed for rough surfaces Analytic formulae in R. Ramamoorthi and P. Hanrahan “A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 2001 pp

Lambertian Incident radiance (mirror sphere) Irradiance (Lambertian) N R. Ramamoorthi and P. Hanrahan “On the Relationship between Radiance and Irradiance: Determining the Illumination from Images of a Convex Lambertian Object” Journal of the Optical Society of America A 18(10) Oct 2001 pp R. Basri and D. Jacobs “Lambertian Reflectance and Linear Subspaces” ICCV 2001 pp

Inverse Lighting Well-posed unless denominator vanishes BRDF should contain high frequencies : Sharp highlights Diffuse reflectors low pass filters: Inverse lighting ill-posed Given: B,ρ find L

Inverse BRDF Well-posed unless L lm vanishes Lighting should have sharp features (point sources, edges) BRDF estimation ill-conditioned for soft lighting Directional Source Area source Same BRDF Given: B,L find ρ

Factoring the Light Field Light Field can be factored Up to global scale factor Assumes reciprocity of BRDF Can be ill-conditioned Analytic formula derived Given: B find L and ρ 4D2D3D More knowns (4D) than unknowns (2D/3D)

Outline Motivation Reflection as Convolution Signal-Processing Framework Applications Forward rendering (convolution) Inverse rendering (deconvolution) Summary and Implications

Computing Irradiance Classically, hemispherical integral for each pixel Lambertian surface is like low pass filter Frequency-space analysis Incident Radiance Irradiance

9 Parameter Approximation Exact image Order 0 1 term (constant) RMS error = 25 %

9 Parameter Approximation Exact image Order 1 4 terms (linear) RMS Error = 8%

9 Parameter Approximation Exact image Order 2 9 terms (quadratic) RMS Error = 1% For any illumination, average error < 2% [Basri Jacobs 01]

Comparison Incident illumination 300x300 Irradiance map Texture: 256x256 Hemispherical Integration 2Hrs Irradiance map Texture: 256x256 Spherical Harmonic Coefficients 1sec

Video R. Ramamoorthi and P. Hanrahan “An Efficient Representation for Irradiance Environment Maps” SIGGRAPH 2001 pp R. Ramamoorthi and P. Hanrahan “Frequency Space Environment Map Rendering” submitted

Video

Inverse Rendering Miller and Hoffman 84 Marschner and Greenberg 97 Sato et al. 97 Dana et al. 99 Debevec et al. 00 Marschner et al. 00 Sato et al. 99 Lighting BRDF Known Unknown Known Textures are a third axis

Contributions Complex illumination Factorization of BRDF, lighting (find both) New representations and algorithms Formal study of inverse problems (well-posed?)

Complications Incomplete sparse data (few photographs) Concavities: Self Shadowing Spatially varying BRDFs

Complications Challenge: Incomplete sparse data (few photographs) Difficult to compute frequency spectra Solution: Use parametric BRDF model Dual angular and frequency space representation

Algorithm Validation Photograph KdKd 0.91 KsKs 0.09 μ 1.85  0.13 “True” values

Algorithm Validation Renderings Unknown lighting Photograph Known lighting KdKd KsKs μ  “True” values Image RMS error 5%

Inverse BRDF: Spheres Bronze Delrin PaintRough Steel Photographs Renderings (Recovered BRDF)

Complications Challenge: Complex geometry with concavities Self shadowing Solution: Use associativity of convolution Blur lighting, treat specular BRDF term as mirror Single ray for shadowing, easy in ray tracer

Complex Geometry 3 photographs of a sculpture Complex unknown illumination Geometry known Estimate microfacet BRDF and distant lighting

Comparison Photograph Rendering

New View, Lighting Photograph Rendering

Complications Challenge: Spatially varying BRDFs Solution: Use textures to modulate BRDF parameters

Textured Objects Photograph Rendering

Summary Reflection as convolution Frequency-space analysis gives many insights Practical forward and inverse algorithms Signal-Processing: A useful paradigm for forward and inverse rendering in graphics and vision

Implications and Future Work Duality between forward and inverse problems: Ill-posed inverse problem  fast forward algorithm Example: Inverse lighting from Lambertian surface ill-posed  computing irradiance is fast

Implications and Future Work Differential framework for reflection Analysis on object surface Complex illumination and BRDF

Implications and Future Work Analyzing intrinsic structure, complexity of light field: Sampling theory based on signal-processing How many images in image-based rendering? How many principal components in PCA?

Lighting Invariant Recognition Theory: Space of images infinite-dimensional for Lambertian [Belhumeur and Kriegman 98] Empirical: 5D subspace enough for diffuse objects [Hallinan 94, Epstein et al. 95, BK98, …] Images from Yale face database

Implications and Future Work Complex illumination in computer vision Generally assume simple lighting (point source) without considering visibility (attached shadows) Signal processing can be used to reduce effects of complex illumination (with shadows) to low-dimensional subspace Many applications: stereo, photometric stereo, shape from shading, lighting invariant recognition etc.

Acknowledgements Pat Hanrahan Marc Levoy Szymon Rusinkiewicz Steve Marschner Stanford graphics group Hodgson-Reed Stanford Graduate Fellowship NSF ITR grant # : “Interacting with the Visual World”

Papers R. Ramamoorthi and P. Hanrahan “A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 2001 pp R. Ramamoorthi and P. Hanrahan “An Efficient Representation for Irradiance Environment Maps” SIGGRAPH 2001 pp R. Ramamoorthi and P. Hanrahan “Frequency Space Environment Map Rendering” submitted R. Ramamoorthi and P. Hanrahan “On the Relationship between Radiance and Irradiance: Determining the Illumination from images of a Convex Lambertian Object” Journal of the Optical Society of America A 18(10) 2001 pp R. Ramamoorthi and P. Hanrahan “Analysis of Planar Light Fields from Homogeneous Convex Curved Surfaces under Distant Illumination” SPIE Photonics West 2001: Human Vision and Electronic Imaging VI pp R. Ramamoorthi “Analytic PCA Construction for Theoretical Analysis of Lighting Variability, Including Attached Shadows, in a Single Image of a Convex Lambertian Object” CVPR 2001 workshop on Identifying Objects across Lighting Variations pp

The End

Photorealistic Rendering Materials/Lighting (Texture Reflectance[BRDF] Lighting) Realistic input models required Arnold Renderer: Marcos Fajardo Rendering Algorithm 80’ s,90’ s : Physically based Geometry 70’ s, 80’ s : Splines 90’ s : Range Data

Measuring Materials, Light Measure BRDF (reflectance): Point light source Measure Illumination: Mirror Sphere Illumination: Grace Cathedral courtesy Paul Debevec

Interactive Forward Rendering Classically, rendering with natural illumination is very expensive compared to using simplified illumination Directional Source Natural Illumination

Lighting Invariant Recognition Theory: Infinite number of light directions Space of images infinite-dimensional Empirical: 5D subspace enough for diffuse objects Images from Debevec et al. 00

Lighting Invariant Recognition Theory: Space of images infinite-dimensional for Lambertian [Belhumeur and Kriegman 98] Empirical: 5D subspace enough for diffuse objects [Hallinan 94, Epstein et al. 95, BK98, …]

Open Questions Relationship between spherical harmonics, PCA 9D approximation > 5D empirical subspace Key insight: Consider approximations over visible normals (upper hemisphere), not entire sphere Frontal % VAF 42%33%16% 4%2% SideAbove/Below Extreme sideCorner Ramamoorthi CVPR IOAVL 01

Light Field in 3D In flatland, 2D function In three dimensions, 4D function Plenoptic Light Field Surface Light Field

Dual Representation Dual Representation Diffuse BRDF: Filter width small in frequency domain Specular: Filter width small in spatial (angular) domain Practical Representation: Dual angular, frequency-space B B d diffuse B s specular Angular = + Frequency

Related Work Precomputed (prefiltered) Irradiance maps [Miller and Hoffman 84, Greene 86, Cabral et al 87] Empirical observation: Irradiance varies slowly with surface normal. Use low resolution irradiance maps Contributions Analytic Irradiance formula Fast computation Compact 9 parameter representation Procedural rendering with programmable shading hardware Our approach can be extended to general BRDFs

Comparison Photograph Rendering (unknown L) Rendering (known L)