Computational Fundamentals of Reflection COMS , Lecture

Motivation Understand intrinsic computational structure of reflection and illumination Necessary for many applications in computer graphics (cannot solve by brute force!!) Real-time forward rendering IBR sampling rates, dimensionality explosion Inverse rendering and inverse problems in general Computer vision: complex lighting, materials

Real-Time Rendering Demo Motivation: Interactive rendering with complex natural illumination and realistic, measured BRDFs

Questions Images are view-dependent (4D quantity) Can we find low-dimensional structure to capture view-dependence?

Space of images as lighting varies Illuminate subject from many incident directions

Example Images Images from Debevec et al. 00

Principal Component Analysis Try to approximate with low dimensional subspace Linear combination of few principal components = … = … Principal component images

Lighting Variability Theory Infinite number of light directions, one coefficient/direction Space of images infinite dimensional [Belhumeur 98] Empirical [Hallinan 94, Epstein 95] Diffuse objects: 5D subspace suffices No satisfactory theoretical explanation of observations

Complex Light Transport Shadows high frequency Analysis possible? Low-dim. structure? Real-time complex lights? Agrawala, Ramamoorthi, Heirich, Moll SIGGRAPH 00

Challenges Illumination complexity Material (BRDF)/view complexity Transport complexity (shadows, interreflection) Fundamental questions Theoretical analysis of intrinsic complexity Sampling rates and resolutions Efficient practical algorithms

Outline Lighting variability in appearance [PAMI Oct, 2002] View variability real-time rendering [SIGGRAPH 02] Visibility/shadows [In Progress]

Lighting variability analysis 1.Frequency space analytic PCA construction 2.Mathematical derivation of principal components 3.Explain empirical results quantitatively Dimensionality of approximating subspace Forms of principal components Relative importance of principal components

Assumptions Single view of single object Lambertian Distant illumination Discount texture Discount concavities: interreflection, cast shadows Consider attached shadows (backfacing normals)

Definitions Irradiance (image) Radiance (light) Normal Lambertian = half-cosine

Previous Theoretical Work Discount attached shadows [Shashua 97, …] Resulting 3D subspace does not fully explain data Analytic PCA (without shadows) [Zhao & Yang 99] Irradiance (image) Radiance (light) Normal Lambertian = half-cosine

Spherical Harmonics

Spherical Harmonic Expansion Expand lighting (L), irradiance (E) in basis functions = …

Lambertian BRDF Expansion Lambertian coefficients

Analytic Irradiance Formula Lambertian surface acts like low-pass filter Basri & Jacobs 01 Ramamoorthi & Hanrahan 01

9 Parameter Approximation Exact image Order 2 9 terms RMS Error = 1% For any illumination, average error < 2% [Basri Jacobs 01]

Open Questions Relationship between spherical harmonics, PCA 9D approximation > 5D empirical subspace Key insight: Consider approximations over visible normals (upper hemisphere), not entire sphere

Intuition: Backwards Half-Cosine parameter irrelevant 8 or fewer params enough

Intuition: dimensionality reduction Start with 9D space, remove dimensions Mean (constant term) subtracted Backwards half-cosine x, xz very similar y, yz very similar Left with 5D subspace

Results: Image of a Sphere Principal components (eigenvectors) mix (linear combinations of) spherical harmonics Results agree with experiment [Epstein 95] We predict: 3 eigenvectors = 91% variance, 5 give 96% Empirical : 3 eigenvectors = 94% variance, 5 give 98% % VAF (eigenvalue) 43%24% 2%

Results: Human Face Numerically compute orthogonality matrix Specific distribution of surface normals important Symmetries in sphere broken (faces are elongated) Principal components somewhat different from sphere % VAF 42%33%16% 4%2%

Results: Human Face Prediction: Principal components have specific forms Empirical : [Hallinan 94] Frontal lighting, side, above/below, extreme side, corner Frontal % VAF 42%33%16% 4%2% SideAbove/Below Extreme sideCorner

Results: Human Face Prediction: Space is close to 5D 3 principal components = 91% variance, 5 components = 97% Empirical : [Epstein 95] 3 principal components = 90% variance, 5 components = 94% Frontal % VAF 42%33%16% 4%2% SideAbove/Below Extreme sideCorner

Results: Human Face Prediction: groups of principal components Group 1: first two (frontal and side) Group 2: next three [with above/below always 3 rd ] Empirical: [Hallinan 94] Two groups [first two (frontal,side) and next three] Within group, %VAF close, may exchange places Frontal % VAF 42%33%16% 4%2% SideAbove/Below Extreme sideCorner

Summary: Lighting Analysis Analytic PCA construction with attached shadows Spherical harmonic analysis: Orthogonality matrix Mathematically derive principal components Qualitative, quantitative agreement with experiment Extend 9D Lambertian model to single view case

Implications Lighting Analysis Attached shadows nearly free: 5D subspace enough Mathematical derivation of principal components Basis functions for subspace methods for recognition,… Graphics applications: Image-Based, inverse rendering Complex illumination in computer vision

Outline Lighting variability in appearance [PAMI Oct, 2002] View variability real-time rendering [SIGGRAPH 02] Visibility/shadows [In Progress]

Reflection Equation 2D Environment Map

Reflection Equation 2D Environment Map N L BRDF

Reflection Equation 4D Orientation Light Field 2D Environment Map Previous Work: Blinn & Newell 76, Miller & Hoffman 84, Greene 86, Kautz & McCool 99, Cabral et al. 99, … BRDF N L

Goals Efficiently precompute and represent OLF Real-time rendering with OLF

Questions Parameterization and structure of OLF Structure leads to representation Computation and rendering of OLF

OLF Parameterization N L N V

N V N V R Reparameterize by reflection vector

OLF Parameterization Captures structure of BRDF (and hence OLF) better Reflective BRDFs become low-dimensional N V N V R Reparameterize by reflection vector

OLF Structure 2D view array of reflection maps 2D image array of view maps

OLF Structure: Phong 2D view array of reflection maps 2D image array of view maps Environment Map Phong Reflection Map (blurred environment map) Same reflection map for all views

OLF Structure: Phong Same reflection map for all views View x View y

OLF Structure: Phong Same reflection map for all views View maps constant for each R View x View y

OLF Structure: Phong Same reflection map for all views View maps constant for each R View x View y Reflection x Reflection y

OLF Structure: Lafortune View y Single 2D reflection map no longer sufficient But variation with viewing direction is slow View x

OLF Structure: Lafortune View maps vary slowly Reflection x Reflection y View y View x

A Simple Factorization View x View y Reflection y Reflection x *

Spherical Harmonic Reflection Map View-dependent reflection (cube)map Encode view maps with low-order spherical harmonics

Prefiltering Directly compute SHRM from Lighting, BRDF Convolution easier to compute in frequency domain Input Lighting and BRDF Spherical Harmonic coeffs. Convolution SHRM

Prefiltering 3 to 4 orders of magnitude faster (< 1 s compared to minutes or hours) Detailed analysis, algorithms, experiments in paper Input Lighting and BRDF Spherical Harmonic coeffs. Convolution SHRM

Number of terms: CURET Analysis for all 61 samples [full bar chart in paper] For essentially all materials, 9-16 terms in SHRM suffice

Demo

Summary view variability Theoretical, empirical analysis of sampling rates and resolutions Frequency space analysis directly on lighting, BRDF Low order expansion suffices for essentially all BRDFs Spherical Harmonic Reflection Maps Hybrid angular-frequency space Compact, efficient, accurate Easy to analyze errors, determine number of terms Fast computation using convolution

Implications Frequency space methods for rendering Global illumination Fast computation of surface light fields Compression for optimal factored representations PCA on SHRMs Theoretical analysis of sampling rates, resolutions General framework for sampling in image-based rendering

Outline Lighting variability in appearance [PAMI Oct, 2002] View variability real-time rendering [SIGGRAPH 02] Visibility/shadows [In Progress]

Visibility complexity (high freq)

But Sparse (< 4%)

Questions on Visibility Theory Locally low-dimensional subspaces? Intrinsic complexity of binary function? Practical Real-time rendering with complex soft shadows, changing illumination for lighting design, simulation Efficient encoding/decoding (wavelets, PCA, dictionaries, hierarchical?) In progress….

Overall Summary Many applications in graphics cannot be solved by brute force Real-time rendering IBR sampling rates, dimensionality explosion Inverse rendering, inverse problems Computer vision: complex lighting, materials Need fundamental understanding of nature of reflection/lighting Illumination complexity Material (view) complexity Transport complexity

Overall Summary Theoretical analysis tools Signal processing, sampling theory Low-dimensional subspaces Information theory, information-based complexity? Practical algorithms Real-time rendering with complex lights,view, transport? Lighting, Material design? Exploit theoretical analysis (sampling rates, forward/inverse duality, angular/frequency/sparsity duality, subspace results, differential analysis, perception)