1. A Theory of Locally Low Dimensional Light Transport Dhruv Mahajan (Columbia University) Ira Kemelmacher-Shlizerman (Weizmann Institute) Ravi Ramamoorthi (Columbia University) Peter Belhumeur (Columbia University)

2. Image Relighting Ng et al 2003

3. Relighting – Linear Combination = Images lit by directional light sources Lighting Intensities Nimeroff et al 94 Dorsey 95 Hallinan 94

4. Relighting – Matrix Vector Multiply = = Input Lighting (Unfolded Cubemap) Output Image Vector Transport Matrix T L B

5. Light transport matrix dimensions n 512 x 512 images n 6 x 32 x 32 = 6144 cubemap lighting Multiplication / Relighting cost n Approximately 10 10 computations per frame n Multiplication intractable in real time Need to compress the light transport Light Transport – Computational Cost

6. Light Transport – SVD Transport Matrix.... U S L Lighting Vector Relit Image Eigenvalues Hallinan 94 V T V T Basis Images Projection Weights

7. Light Transport – SVD - Global Dimensionality Large.... V T Transport Matrix Eigenvalues Energy (in %) No. of Eigenvalues

8. Computation still intractable Global Dimensionality

9. Locally Low Dimensional Light Transport p pixels p rows SVD Locally Low Dimensional Transport Lighting Resolution Dimensionality of the patch Transport Matrix

10. Previous Work Blockwise PCA – Nayar et al. 04 n Image divided in to fixed size square patches n Each patch compressed using PCA Clustered PCA – Sloan et al. 03 n Object divided in to fixed number of clusters n Each cluster compressed using PCA

11. Previous Work Surface light fields  Nishino et al. 01  Chen et al. 02 General reflectance fields  Matusik et al. 02  Garg et al. 06 Compression  JPEG, MPEG No Theoretical Analysis Dimensionality vs Patch Size? Dimensionality vs Material Properties? Dimensionality vs Global Effects ?

12. Local Light Transport Dimensionality Analysis of local light transport dimensionality P Dimensionality Cost Patch Area 1

13. Local Light Transport Dimensionality Analysis of local light transport dimensionality Dimensionality Cost Patch Area 2 x 2

14. Local Light Transport Dimensionality Analysis of local light transport dimensionality Dimensionality Cost Patch Area

15. Local Light Transport Dimensionality Analysis of local light transport dimensionality Dimensionality Cost Patch Area

16. Local Light Transport Dimensionality Analysis of local light transport dimensionality Dimensionality Cost Patch Area

17. Local Light Transport Dimensionality Analysis of local light transport dimensionality Dimensionality Cost Patch Area

18. Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering Dimensionality

19. Overhead Cost Global Lighting Dimensionality cost = number of bases Overhead Cost = Projection Weights Cost Patch Area

20. Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering P

21. Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering

22. Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering

23. Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering

24. Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering

25. Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering

26. Rendering Cost Theoretical analysis of rendering cost Cost Patch Area Overhead cost for rendering Patch Size Optimal Rendering cost = Dimensionality + Overhead

27. Contributions Analysis of dimensionality of local light transport n Change of dimensionality with size n Diffuse and glossy reflections n Shadows n Analyzing rendering cost n Analytical formula for optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale images to very high resolutions n Develop adaptive clustering algorithm

28. Local Light Transport Dimensionality Analysis of local light transport dimensionality Dimensionality Cost Patch Area

29. Dimensionality vs. Patch Size Large Area : linear relationship slope = 1 slope - rate of change of dimensionality Independent of material properties log (Dimensionality) log (Patch Area) pixels dimensionality Diffuse/Specular BRDF Dimensionality Patch Area

30. Dimensionality vs. Patch Size Small Area : sub - linear relationship log (Dimensionality) log (Patch Area) pixels dimensionality slope < 1 Diffuse/Specular BRDF

31. Mathematical Tools for Analysis Convolution formula for glossy reflections and shadows n Ramamoorthi and Hanrahan 01 n Basri and Jacobs 01 n Ramamoorthi et al 04 Szego’s Eigenvalue Distribution Theorem n Eigenvalues of the light transport matrix of the patch Fourier Scale and Convolution Theorems n Dimensionality as a function of patch size

32. Bandwidth of BRDF Central Result Patch Dimensionality Patch Area Constant Bandwidth of BRDF Patch Dimensionality Patch Area Constant Lighting BRDF low pass filter Material property

33. Fourier Transform BRDF/ Material Properties Bandwidth of BRDF Central Result Patch Dimensionality Patch Area Constant 99% Energy low frequency high frequency Bandwidth

34. Central Result Large Area log (Dimensionality) log (Patch area) Diffuse/Specular BRDF Bandwidth of BRDF Patch Dimensionality Patch Area ( ( ) ) Bandwidth of BRDF Patch Area Constant

35. Large Area log (Dimensionality) log (Patch area) Diffuse/Specular BRDF Bandwidth of BRDF Patch Dimensionality Patch Area ( ( ) )

36. Large Area Bandwidth of BRDF ) ( log (Dimensionality) log (Patch area) Diffuse/Specular BRDF Patch Dimensionality Patch Area ( ( ) ) Bandwidth of BRDF Patch Dimensionality Patch Area ( ( ) )

37. Large Area Bandwidth of BRDF ) ( log (Dimensionality) log (Patch area) Diffuse/Specular BRDF linear relationship slope = 1 Patch Dimensionality Patch Area ( ( ) )

38. Small Area log (Dimensionality) log (Patch area) Diffuse/Specular BRDF slope < 1 sublinear relationship Bandwidth of BRDF ) ( Patch Dimensionality Patch Area ( ( ) )

39. Contributions Analysis of dimensionality of local light transport n Change of dimensionality with size n Glossy reflections n Shadows n Analyzing rendering cost n Analytical formula for optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale images to very high resolutions n Develop adaptive clustering algorithm

40. Visibility Function Blocker Visibility Function = 0 Visibility Function = 1 P Lighting Directions

41. Shadows Dimensionality changes slowly in presence of shadows Diffuse and Specular BRDF Shadows slope =.5 slope = 1 log (Dimensionality) log (Patch area) Light Transport = Visibility Function

42. Shadows – Step Blocker x y z Step Blocker Dimensionality √Patch Area Same Visibility Function Dimensionality changes only along one dimension Lighting Direction log (Dimensionality).5 log(Patch Area) Different Visibility Function Light Transport = Visibility Function

43. Shadows – Step Blocker x y z Step Blocker Dimensionality √Patch Area Same Visibility Function Dimensionality changes only along one dimension log (Dimensionality).5 log(Patch Area) Different Visibility Function Light Transport = Visibility Function x z

44. Contributions Analysis of dimensionality of local light transport n Change of dimensionality with size n Glossy reflections n Shadows n Analyzing rendering cost n Analytical formula for optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale images to very high resolutions n Develop adaptive clustering algorithm

45. Local Light Transport Dimensionality Analysis of dimensionality of local light transport Diffuse and Glossy reflections, dimensionality area Shadows, dimensionality √ area Bandwidth of BRDF Patch Dimensionality Patch Area Constant

46. Contributions Analysis of dimensionality of local light transport n Change of dimensionality with size n Glossy reflections n Shadows n Analyzing rendering cost n Analytical formula for optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale images to very high resolutions n Develop adaptive clustering algorithm

47. Overhead Cost Cost Patch Area Dimensionality

48. Overhead Cost Cost Patch Area P Overhead Dimensionality

49. Overhead Cost Cost Patch Area Dimensionality Overhead

50. Overhead Cost Cost Patch Area Dimensionality Overhead

51. Overhead Cost Cost Patch Area Dimensionality Overhead

52. Overhead Cost Cost Patch Area Dimensionality Overhead

53. Overhead Cost Cost Patch Area Dimensionality Overhead

54. Rendering Cost Cost Patch Area Rendering Cost Dimensionality Overhead

55. Rendering Cost vs. Patch Size Large Patch size : Increasing patch size increases total cost Rate of increase in dimensionality Rate of decrease in overhead > Cost Patch Area Rendering Cost Dimensionality Overhead Linear regime

56. Rendering Cost vs. Patch Size Dimensionality Overhead Large Patch size : Linear regime Increasing patch size increases total cost Rate of increase in dimensionality Rendering cost = Dimensionality + Overhead Rate of decrease in overhead > Rendering Cost Cost Patch Area

57. Rendering Cost vs. Patch Size Small Patch size : Increasing patch size decreases total cost Rate of increase in dimensionality Rate of decrease in overhead < Cost Patch Area Rendering Cost Dimensionality Sublinear regime Overhead

58. Rendering Cost vs. Patch Size Dimensionality Overhead Small Patch size : Sublinear regime Rendering Cost Cost Patch Area Rendering cost = Dimensionality + Overhead Increasing patch size decreases total cost Rate of increase in dimensionality Rate of decrease in overhead <

59. Rendering Cost vs. Patch Size Intermediate size : Rate of increase in dimensionality Rate of decrease in overhead = Total cost minimum Cost Patch Area Rendering Cost Dimensionality Overhead Minimum

60. Rendering Cost vs. Patch Size Dimensionality Overhead Rendering Cost Minimum Intermediate size : Cost Patch Area Rendering cost = Dimensionality + Overhead Rate of increase in dimensionality Rate of decrease in overhead = Total cost minimum

61. Optimal Patch Size - Global Dimensionality

62. Optimal Patch Size - Global Dimensionality Optimal Patch Size - Function of slope of dimensionality curve Dimensionality Curve - From our theoretical analysis - Empirically from the given dataset

63. Optimal Patch Size – CPCA Example Optimal Patch Size Total cost Face dataset across lighting 110 220 330 440 550 average cluster size cost per pixel - Global Dimensionality - Function of slope of dimensionality curve

64. Glossy Reflections Optimal Patch Size - Global Dimensionality - Function of slope of dimensionality curve Number of pixels in the patch increases with glossiness Independent of material properties

65. Contributions Analysis of dimensionality of local light transport n Change of dimensionality with size n Glossy reflections n Shadows n Analyzing rendering cost n Analytical formula for optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale images to very high resolutions n Develop adaptive clustering algorithm

66. Setting Optimal Patch Size – CPCA

67. 24000 vertices Estimated 220 114.78 cost per pixel clusters 130-600 114.78-130 11 310.7 large 6 X 32 X 32 Cube Map 45.0 Hz.

68. Contributions Analysis of dimensionality of local light transport n Change of dimensionality with size n Glossy reflections n Shadows n Analyzing rendering cost n Analytical formula for optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale images to very high resolutions n Develop adaptive clustering algorithm

69. Scaling of Cost With Resolution Subdivide More new resolution Independent of patch resolution Optimal patch size same for both resolutions - Global Dimensionality - Function of slope of dimensionality curve

70. Scaling of Cost With Resolution Sub-linear increase in cost with resolution Increase in resolution - Increase in cost - 1.85 new resolution

71. Sublinear increase in cost with resolution 1024 800 x 600 Scaling of Cost With Resolution

73. Summary Analysis of dimensionality of local light transport Diffuse and Glossy reflections, dimensionality area Shadows, dimensionality √ area Analysis of rendering cost Optimal patch size Scaling of cost with resolution Practical Applications Setting optimal parameters in existing methods Adaptive clustering algorithms

74. Summary Analysis of dimensionality of local light transport Diffuse and Glossy reflections, dimensionality area Shadows, dimensionality √ area Analysis of rendering cost Optimal patch size Scaling of cost with resolution Practical Applications Setting optimal parameters in existing methods Adaptive clustering algorithms Bandwidth of BRDF Patch Dimensionality Patch Area Constant

75. Summary Analysis of dimensionality of local light transport n Change of dimensionality with size n Glossy reflections, dimensionality area Shadows, dimensionality √ area n Analyzing rendering cost n Derive optimal patch size n Practical Applications n Fine tuning parameters of existing methods n Scale to very high resolutions n Develop adaptive clustering algorithms

76. Future Work More solid theoretical foundation n High dimensional appearance compression n Representation ECCV 2006, PAMI 2007 Analysis of light transport in frequency domain TOG, Jan. 2007 Analysis of light transport in gradient domain Siggraph 2007 Analysis of general local light transport for patches