A Theory of Locally Low Dimensional Light Transport Dhruv Mahajan (Columbia University) Ira Kemelmacher-Shlizerman (Weizmann Institute) Ravi Ramamoorthi (Columbia University) Peter Belhumeur (Columbia University)

Image Relighting Ng et al 2003

Relighting – Linear Combination = Images lit by directional light sources Lighting Intensities Nimeroff et al 94 Dorsey 95 Hallinan 94

Relighting – Matrix Vector Multiply = = Input Lighting (Unfolded Cubemap) Output Image Vector Transport Matrix T L B

Light transport matrix dimensions n 512 x 512 images n 6 x 32 x 32 = 6144 cubemap lighting Multiplication / Relighting cost n Approximately computations per frame n Multiplication intractable in real time Need to compress the light transport Light Transport – Computational Cost

Light Transport – SVD Transport Matrix.... U S L Lighting Vector Relit Image Eigenvalues Hallinan 94 V T V T Basis Images Projection Weights

Light Transport – SVD - Global Dimensionality Large.... V T Transport Matrix Eigenvalues Energy (in %) No. of Eigenvalues

Computation still intractable Global Dimensionality

Locally Low Dimensional Light Transport p pixels p rows SVD Locally Low Dimensional Transport Lighting Resolution Dimensionality of the patch Transport Matrix

Previous Work Blockwise PCA – Nayar et al. 04 n Image divided in to fixed size square patches n Each patch compressed using PCA Clustered PCA – Sloan et al. 03 n Object divided in to fixed number of clusters n Each cluster compressed using PCA

Previous Work Surface light fields  Nishino et al. 01  Chen et al. 02 General reflectance fields  Matusik et al. 02  Garg et al. 06 Compression  JPEG, MPEG No Theoretical Analysis Dimensionality vs Patch Size? Dimensionality vs Material Properties? Dimensionality vs Global Effects ?

Local Light Transport Dimensionality Analysis of local light transport dimensionality P Dimensionality Cost Patch Area 1

Local Light Transport Dimensionality Analysis of local light transport dimensionality Dimensionality Cost Patch Area 2 x 2

Local Light Transport Dimensionality Analysis of local light transport dimensionality Dimensionality Cost Patch Area

Local Light Transport Dimensionality Analysis of local light transport dimensionality Dimensionality Cost Patch Area

Local Light Transport Dimensionality Analysis of local light transport dimensionality Dimensionality Cost Patch Area

Local Light Transport Dimensionality Analysis of local light transport dimensionality Dimensionality Cost Patch Area

Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering Dimensionality

Overhead Cost Global Lighting Dimensionality cost = number of bases Overhead Cost = Projection Weights Cost Patch Area

Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering P

Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering

Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering

Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering

Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering

Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering

Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering Patch Size Optimal Rendering cost = Dimensionality + Overhead

Contributions Analysis of dimensionality of local light transport n Change of dimensionality with size n Diffuse and glossy reflections n Shadows n Analyzing rendering cost n Analytical formula for optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale images to very high resolutions n Develop adaptive clustering algorithm

Local Light Transport Dimensionality Analysis of local light transport dimensionality Dimensionality Cost Patch Area

Dimensionality vs. Patch Size Large Area : linear relationship slope = 1 slope - rate of change of dimensionality Independent of material properties log (Dimensionality) log (Patch Area) pixels dimensionality Diffuse/Specular BRDF Dimensionality Patch Area

Dimensionality vs. Patch Size Small Area : sub - linear relationship log (Dimensionality) log (Patch Area) pixels dimensionality slope < 1 Diffuse/Specular BRDF

Mathematical Tools for Analysis Convolution formula for glossy reflections and shadows n Ramamoorthi and Hanrahan 01 n Basri and Jacobs 01 n Ramamoorthi et al 04 Szego’s Eigenvalue Distribution Theorem n Eigenvalues of the light transport matrix of the patch Fourier Scale and Convolution Theorems n Dimensionality as a function of patch size

Bandwidth of BRDF Central Result Patch Dimensionality Patch Area Constant Bandwidth of BRDF Patch Dimensionality Patch Area Constant Lighting BRDF low pass filter Material property

Fourier Transform BRDF/ Material Properties Bandwidth of BRDF Central Result Patch Dimensionality Patch Area Constant 99% Energy low frequency high frequency Bandwidth

Central Result Large Area log (Dimensionality) log (Patch area) Diffuse/Specular BRDF Bandwidth of BRDF Patch Dimensionality Patch Area ( ( ) ) Bandwidth of BRDF Patch Area Constant

Large Area log (Dimensionality) log (Patch area) Diffuse/Specular BRDF Bandwidth of BRDF Patch Dimensionality Patch Area ( ( ) )

Large Area Bandwidth of BRDF ) ( log (Dimensionality) log (Patch area) Diffuse/Specular BRDF Patch Dimensionality Patch Area ( ( ) ) Bandwidth of BRDF Patch Dimensionality Patch Area ( ( ) )

Large Area Bandwidth of BRDF ) ( log (Dimensionality) log (Patch area) Diffuse/Specular BRDF linear relationship slope = 1 Patch Dimensionality Patch Area ( ( ) )

Small Area log (Dimensionality) log (Patch area) Diffuse/Specular BRDF slope < 1 sublinear relationship Bandwidth of BRDF ) ( Patch Dimensionality Patch Area ( ( ) )

Contributions Analysis of dimensionality of local light transport n Change of dimensionality with size n Glossy reflections n Shadows n Analyzing rendering cost n Analytical formula for optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale images to very high resolutions n Develop adaptive clustering algorithm

Visibility Function Blocker Visibility Function = 0 Visibility Function = 1 P Lighting Directions

Shadows Dimensionality changes slowly in presence of shadows Diffuse and Specular BRDF Shadows slope =.5 slope = 1 log (Dimensionality) log (Patch area) Light Transport = Visibility Function

Shadows – Step Blocker x y z Step Blocker Dimensionality √Patch Area Same Visibility Function Dimensionality changes only along one dimension Lighting Direction log (Dimensionality).5 log(Patch Area) Different Visibility Function Light Transport = Visibility Function

Shadows – Step Blocker x y z Step Blocker Dimensionality √Patch Area Same Visibility Function Dimensionality changes only along one dimension log (Dimensionality).5 log(Patch Area) Different Visibility Function Light Transport = Visibility Function x z

Contributions Analysis of dimensionality of local light transport n Change of dimensionality with size n Glossy reflections n Shadows n Analyzing rendering cost n Analytical formula for optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale images to very high resolutions n Develop adaptive clustering algorithm

Local Light Transport Dimensionality Analysis of dimensionality of local light transport Diffuse and Glossy reflections, dimensionality area Shadows, dimensionality √ area Bandwidth of BRDF Patch Dimensionality Patch Area Constant

Contributions Analysis of dimensionality of local light transport n Change of dimensionality with size n Glossy reflections n Shadows n Analyzing rendering cost n Analytical formula for optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale images to very high resolutions n Develop adaptive clustering algorithm

Overhead Cost Cost Patch Area Dimensionality

Overhead Cost Cost Patch Area P Overhead Dimensionality

Overhead Cost Cost Patch Area Dimensionality Overhead

Overhead Cost Cost Patch Area Dimensionality Overhead

Overhead Cost Cost Patch Area Dimensionality Overhead

Overhead Cost Cost Patch Area Dimensionality Overhead

Overhead Cost Cost Patch Area Dimensionality Overhead

Rendering Cost Cost Patch Area Rendering Cost Dimensionality Overhead

Rendering Cost vs. Patch Size Large Patch size : Increasing patch size increases total cost Rate of increase in dimensionality Rate of decrease in overhead > Cost Patch Area Rendering Cost Dimensionality Overhead Linear regime

Rendering Cost vs. Patch Size Dimensionality Overhead Large Patch size : Linear regime Increasing patch size increases total cost Rate of increase in dimensionality Rendering cost = Dimensionality + Overhead Rate of decrease in overhead > Rendering Cost Cost Patch Area

Rendering Cost vs. Patch Size Small Patch size : Increasing patch size decreases total cost Rate of increase in dimensionality Rate of decrease in overhead < Cost Patch Area Rendering Cost Dimensionality Sublinear regime Overhead

Rendering Cost vs. Patch Size Dimensionality Overhead Small Patch size : Sublinear regime Rendering Cost Cost Patch Area Rendering cost = Dimensionality + Overhead Increasing patch size decreases total cost Rate of increase in dimensionality Rate of decrease in overhead <

Rendering Cost vs. Patch Size Intermediate size : Rate of increase in dimensionality Rate of decrease in overhead = Total cost minimum Cost Patch Area Rendering Cost Dimensionality Overhead Minimum

Rendering Cost vs. Patch Size Dimensionality Overhead Rendering Cost Minimum Intermediate size : Cost Patch Area Rendering cost = Dimensionality + Overhead Rate of increase in dimensionality Rate of decrease in overhead = Total cost minimum

Optimal Patch Size - Global Dimensionality

Optimal Patch Size - Global Dimensionality Optimal Patch Size - Function of slope of dimensionality curve Dimensionality Curve - From our theoretical analysis - Empirically from the given dataset

Optimal Patch Size – CPCA Example Optimal Patch Size Total cost Face dataset across lighting average cluster size cost per pixel - Global Dimensionality - Function of slope of dimensionality curve

Glossy Reflections Optimal Patch Size - Global Dimensionality - Function of slope of dimensionality curve Number of pixels in the patch increases with glossiness Independent of material properties

Contributions Analysis of dimensionality of local light transport n Change of dimensionality with size n Glossy reflections n Shadows n Analyzing rendering cost n Analytical formula for optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale images to very high resolutions n Develop adaptive clustering algorithm

Setting Optimal Patch Size – CPCA

24000 vertices Estimated cost per pixel clusters large 6 X 32 X 32 Cube Map 45.0 Hz.

Contributions Analysis of dimensionality of local light transport n Change of dimensionality with size n Glossy reflections n Shadows n Analyzing rendering cost n Analytical formula for optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale images to very high resolutions n Develop adaptive clustering algorithm

Scaling of Cost With Resolution Subdivide More new resolution Independent of patch resolution Optimal patch size same for both resolutions - Global Dimensionality - Function of slope of dimensionality curve

Scaling of Cost With Resolution Sub-linear increase in cost with resolution Increase in resolution - Increase in cost new resolution

Sublinear increase in cost with resolution x 600 Scaling of Cost With Resolution

Summary Analysis of dimensionality of local light transport Diffuse and Glossy reflections, dimensionality area Shadows, dimensionality √ area Analysis of rendering cost Optimal patch size Scaling of cost with resolution Practical Applications Setting optimal parameters in existing methods Adaptive clustering algorithms

Summary Analysis of dimensionality of local light transport Diffuse and Glossy reflections, dimensionality area Shadows, dimensionality √ area Analysis of rendering cost Optimal patch size Scaling of cost with resolution Practical Applications Setting optimal parameters in existing methods Adaptive clustering algorithms Bandwidth of BRDF Patch Dimensionality Patch Area Constant

Summary Analysis of dimensionality of local light transport n Change of dimensionality with size n Glossy reflections, dimensionality area Shadows, dimensionality √ area n Analyzing rendering cost n Derive optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale to very high resolutions n Develop adaptive clustering algorithms

Future Work More solid theoretical foundation n High dimensional appearance compression n Representation ECCV 2006, PAMI 2007 Analysis of light transport in frequency domain TOG, Jan Analysis of light transport in gradient domain Siggraph 2007 Analysis of general local light transport for patches