Non-Linear Kernel-Based Precomputed Light Transport Paul Green MIT Jan Kautz MIT Wojciech Matusik MIT Frédo Durand MIT Henrik Wann Jensen UCSD

Geometry & Viewpoint All-Frequency Lighting Rendered Frame Reflectance Interactive High Quality 6D Relighting

Precomputed Light Transport Courtesy of Sloan et al Exit Radiance Distant Radiance Incident Radiance Shadowing Inter-reflection Transport function maps distant light to incident light Can Include BRDF if outgoing direction is fixed

Light Transport Linear Operator Radiance L o at point p along direction is weighted sum of distant radiance L i Outgoing Radiance Transport Vector Distant Radiance (Environment Map)

ExampleExample Transport Function (log scale) Environment Map BRDF Weighted Incident Radiance Exit Radiance (outgoing color) It’s a Dot Product Between Lighting and Transport Vectors!!

Raw Transport Matrices Enormous –50,000 Vertices –24,200 Element Environment Map –92 View Directions Direct Lighting-Transport product too costly –24,200 multiplies / vertex Direct Evaluation Infeasible 50 GBs raw data But the formula still works in any other basis

PLT is Representation Key issue of PLT is representation of Transport and Lighting efficiently. Efficiency of: –Storage –Lighting-Transport Product

Linear Approximation Precomputed Radiance Transfer [Sloan et al 02,03] –Low Order Spherical Harmonics –Soft Shadows –Low Frequency Lighting –Not Practical For High Frequency Lighting Courtesy of Ng et al.2003

Non-Linear Approximation Non-Linear Wavelet Lighting Approximation [Ng et al 03] –Haar Wavelets –Non-Linear Approximation –All Frequency Lighting –Arbitrary BRDF -> Fixed View Courtesy of Ng et al.2003

What is Non-Linear Approximation? Non-linear: Approximating basis set depends on input Linear: First 5 Coefficients Non-linear: 5 Largest Coefficients SSE = 3,037 SSE = 935

Overview of our method Precompute Transport at a sparse set of sample view directions –Backwards Photon Tracing Approximate Transport –Non-Linear Parametric Representation Render –Fast Lighting-Transport Product –View Point Interpolation

PrecomputationPrecomputation Azimuth Elevation Transport In spherical coordinates (Lat/Long) Log Scale

PrecomputationPrecomputation Specular Diffuse Shadowed Inter-reflections Refracted Backwards photon tracing from fixed view, fixed position p

PrecomputationPrecomputation Photon Cloud Latitude / Longitude Map Elevation Angle Azimuthal Angle

Density Estimation Photon Cloud After Density Estimation

Factoring Transport View Independent (diffuse) Consistent across all views View Dependent (specular) One per view Full Transport

RecapRecap p Non-linear Approximation

Overview of our method Precompute Transport at a sparse set of sample view directions –Backwards Photon Tracing Approximate Transport –Non-Linear Parametric Representation Render –Fast Integration Method –View Point Interpolation p

Transport approximated by sum of constant valued box functions Arbitrary weight, size and position Expressive as Haar Wavelets –Even More Flexible! Non-Linear Approximation w j – weight K j – size and position Original Non-linear Approximation Approximated

Non-Linear Approximation Box: Arbitrary location & size Our approach Wavelet: Rigid dyadic domain [Ng et al. 03]

Non-Linear Approximation SSE = 652 SSE = 935 Non-Linear Wavelet Approximation 5 Largest Coefficients [Ng et al.] Non-Linear Box Kernel Approximation 5 Boxes Our approach

Non-Linear Approximation How do you find boxes? It is hard –Decomposition is not unique (Non-Orthogonal) Our Approach –Greedy Strategy for View Dependent –K-d subdivision for View Independent [Matusik et al 04]

Non-Linear Approximation View Independent (diffuse) View Dependent (specular) Original Approximated Original Approximated

Overview of our method Pre-compute Transport at a sparse set of sample view directions –Backwards Photon Tracing Approximate Transport –Non-Linear Parametric Representation Render –Fast Lighting-Transport Product –View Point Interpolation p

Rendering Box Kernels Summed Area Table Lookup Exit Radiance (outgoing color) Approximated Transport Lighting

Only Pre-computed Transport Functions for sparse set of outgoing directions Interpolate Outgoing Radiance ? Rendering Novel Views p ?

Interpolate Parameters Interpolate Box parameters –Position –Size –Weight Drawbacks: Visibility –But is at least plausible Shadowing is View Independent –Does not need to be interpolated p ?

Interpolating Views Example Resulting Color Interpolate Parameters Our Approach Interpolate Radiance Values Standard Linear Fading

SummarySummary Compute Transport Matrix T at sparse set of sample view directions Factor T into view dependent and view independent parts Approximate T using Non-Linear Parametric Representation Render by interpolating parameters from closest precomputed Transport Vectors p p ?

VideoVideo

ContributionContribution Non-Linear Representation View Point Interpolation Technique All-Frequency Relighting From Non-Fixed Viewpoints with Arbitrary Reflectance and Transport Effects

Future Work Other Non-Linear Approximations Formal Box Decomposition Method Compression of Transport Vectors Hardware Acceleration

Thank You