View-Dependent Precomputed Light Transport Using Nonlinear Gaussian Function Approximations Paul Green 1 Jan Kautz 1 Wojciech Matusik 2 Frédo Durand 1 MIT CSAIL 1 MERL 2 ACM Symposium in Interactive 3D Graphics 2006 View-Dependent Precomputed Light Transport Using Nonlinear Gaussian Function Approximations

Geometry & Viewpoint All-Frequency Lighting Rendered Frame Reflectance Interactive 6D Relighting

Goal: 6D Relighting  High-quality view-dependent effects Sharp highlights  Spatially varying BRDFs  Allow rendering w/ large environment maps (e.g. 6x256x256)  Without paying a prohibitive data storage price

Applications  Games  Architectural Visualization

Outline  Background / Previous Work PRT Nonlinear Approximation  Our Representation Rendering Fitting Results  Conclusion

Courtesy of Sloan et al Exit Radiance Distant Radiance Incident Radiance Shadowing Inter-reflection Transport function maps distant light to incident light Can Include BRDF if outgoing direction ω o is fixed Precomputed Light Transport

Courtesy of Sloan et al  Radiance L o at point p along direction is weighted sum of distant radiance L i Outgoing Radiance Transport Vector Distant Radiance (Environment Map) Light Transport

Example Transport Function Environment Map BRDF Weighted Incident Radiance Exit Radiance (outgoing color) It’s a Dot Product Between Lighting and Transport Vectors!!

Light Transport  Data Size Problem Many GB’s of data Rendering is slow  6x64x64 cubemap ~24,000 mults/vert  Can reduce size in different basis: Spherical Harmonics Wavelets Zonal Harmonics

PRT with Spherical Harmonics  Precomputed Radiance Transfer [Sloan et al 02,03] Low Order Spherical Harmonics Soft Shadows and Low Frequency Lighting Not Suitable For Highly Glossy Materials Not Practical For High Frequency Lighting Image from slides by Ng et al.2003

Nonlinear Wavelet PRT  Nonlinear Wavelet Lighting Approximation [Ng et al 03] Haar Wavelets Nonlinear Approximation All Frequency Lighting Fixed View For Arbitrary BRDFs Image from slides by Ng et al.2003

Separable PRT  Factor BRDF into product of view-only and light-only functions [Liu et al 04, Wang et al 04]  Nonlinear Wavelet Approximation [Ng et al 03]  Need factorization per BRDF  Very specular materials still require many coefficients Liu et al 04

Our Goal: 6D Relighting  High quality view-dependent effects  Representation of transport that enables Spatially varying BRDFs Arbitrary highlight scale compact storage High-res environment maps (e.g. 6x256x256)  Sparse view sampling  Requires high-quality interpolation Over view directions Over mesh triangles

Outline  Background / Previous Work PRT Nonlinear Approximation  Our Representation Rendering Fitting Results  Conclusion

Factoring Transport Represent with SH or Wavelets View-independent (diffuse) View-dependent Nonlinear Gaussian Function Approximation ? Per view

Our Nonlinear Representation - mean- std. deviation Sum of N isotropic Gaussians

Previous work: Nonlinear Wavelet  Nonlinear: Approximating basis set depends on input  Truncate small coefficients  Effect of coefficients is still linear Linear: First 8 Coefficients SSE = Nonlinear: 8 Largest Coefficients SSE = 25.1

Our solution: Even More Nonlinear  We don’t start from linear basis  No Truncation of coefficients  Nonlinear parameter estimation  Parameters have nonlinear effects SSE = 5.61 Nonlinear sum of 2 Gaussians Nonlinear: 8 Largest Coefficients SSE = 25.1

 Arbitrary freq bandwidth  Accurate approx w/small storage  Good interpolation  Good visual quality Advantages of sum of Gaussians original N = 1 N = 2 Haar 70 terms

Examples p

Rendering Approximated Transport Lighting ?

Gaussian Pyramid Larger σ Pre-convolve environment with Gaussians of varying sizes Only done Once Can start with large cubemap e.g. 6x256x256

Rendering Tri-linear lookup in Gaussian Pyramid Exit Radiance (outgoing color) Approximated Transport Lighting

Rendering Novel Views  Precomputed Transport Functions for sparse set of outgoing directions  Naïve solution: (Gouraud Shading) Interpolate Outgoing Radiance  Cross-fading artifacts p ?

Better: Interpolate Parameters  Interpolate Gaussian parameters Mean, Std. Dev, Weights  Analogous to Phong vs Gouraud shading p ? t=0t=1t=0.5

Per-Pixel Interpolation

Interpolation Drawbacks  Visibility may not interpolate correctly But is usually plausible  Correspondences Makes fitting more difficult View-independent View-dependent

Data Fitting  Precompute Raw Transport Data  Solve large scale nonlinear optimization problem For each vertex  For each view Fit Gaussians to transport data p

Nonlinear Optimization Objective has terms for  Fitting the Data  Regularization Angular smoothness and correspondences Spatial smoothness and correspondences original N = 2 N=1N=1 - p

Error Plots

Results  6x256x256 Environment Maps  Single Gaussian  2.8 GHz P4  1GB RAM  Nvidia 6800 Ultra  Software screen capture

Contributions  New parametric representation of PLT Compact Storage High Quality Interpolation of parameters Sparse View Sampling Efficient Rendering Spatially varying BRDFs original N = 2Haar

Questions?