1. All-Frequency Shadows Using Non-linear Wavelet Lighting Approximation Ren Ng Stanford Ravi Ramamoorthi Columbia Pat Hanrahan Stanford

2. From Frank Gehry Architecture, Ragheb ed. 2001 Lighting Design

3. From Frank Gehry Architecture, Ragheb ed. 2001 Lighting Design

4. Existing Fast Shadow Techniques We know how to render very hard and very soft shadows Sloan, Kautz, Snyder 2002 Shadows from smooth lighting (precomputed radiance transfer) Sen, Cammarano, Hanrahan, 2003 Shadows from point-lights (shadow maps, volumes)

5. All-Frequency Lighting Teapot in Grace Cathedral

6. Soft Lighting Teapot in Grace Cathedral

7. All-Frequency Lighting Teapot in Grace Cathedral

8. Soft Lighting Teapot in Grace Cathedral

9. All-Frequency Lighting Teapot in Grace Cathedral

10. Overview Matrix formulation of relighting n Pre-computing the light-transport matrix n Compressing the matrix multiplication Results n Demonstration n Error analysis (Comparison with PRT) Future directions n Limitations n Areas for research

11. Overview Matrix formulation of relighting n Pre-computing the light-transport matrix n Compressing the matrix multiplication Results n Demonstration n Error analysis (Comparison with PRT) Future directions n Limitations n Areas for research

12. Relighting as Matrix-Vector Multiply

13. Input Lighting (Cubemap Vector) Output Image (Pixel Vector) Transport Matrix

14. Ray-Tracing Matrix Columns

16. Light-Transport Matrix Rows

19. Rasterizing Matrix Rows Pre-computing rows n Rasterize visibility hemicubes with graphics hardware n Read back pixels and weight by reflection function

20. Overview Matrix formulation of relighting n Pre-computing the light-transport matrix n Compressing the matrix multiplication Results n Demonstration n Error analysis (Comparison with PRT) Future directions n Limitations n Areas for research

21. Overview Matrix formulation of relighting n Pre-computing the light-transport matrix n Compressing the matrix multiplication Results n Demonstration n Error analysis (Comparison with PRT) Future directions n Limitations n Areas for research

22. Matrix Multiplication is Enormous Dimension n 512 x 512 pixel images n 6 x 64 x 64 cubemap environments Full matrix-vector multiplication is intractable n On the order of 10 10 operations per frame

23. Sparse Matrix-Vector Multiplication Choose data representations with mostly zeroes Vector: Use non-linear wavelet approximation on lighting Matrix: Wavelet-encode transport rows

24. Non-linear Wavelet Light Approximation Wavelet Transform

25. Non-linear Wavelet Light Approximation Non-linear Approximation Retain 0.1% – 1% terms

26. Why Non-linear Approximation? Linear n Use a fixed set of approximating functions n Precomputed radiance transfer uses 25 - 100 of the lowest frequency spherical harmonics Non-linear n Use a dynamic set of approximating functions (depends on each frame’s lighting) n In our case: choose 10’s - 100’s from a basis of 24,576 wavelets Idea: Compress lighting by considering input data

27. Why Wavelets? Wavelets provide dual space / frequency locality n Large wavelets capture low frequency, area lighting n Small wavelets capture high frequency, compact features In contrast n Spherical harmonics n Perform poorly on compact lights n Pixel basis n Perform poorly on large area lights

28. Choosing Non-linear Coefficients Three methods of prioritizing 1. Magnitude of wavelet coefficient n Optimal for approximating lighting 2. Transport-weighted n Biases lights that generate bright images 3. Area-weighted n Biases large lights

29. Sparse Matrix-Vector Multiplication Choose data representations with mostly zeroes Vector: Use non-linear wavelet approximation on lighting Matrix: Wavelet-encode transport rows Choose data representations with mostly zeroes Vector: Use non-linear wavelet approximation on lighting Matrix: Wavelet-encode transport rows

30. Matrix Row Wavelet Encoding

31. Extract Row

32. Matrix Row Wavelet Encoding Wavelet Transform

33. Matrix Row Wavelet Encoding Wavelet Transform

34. Matrix Row Wavelet Encoding Wavelet Transform

35. Matrix Row Wavelet Encoding Wavelet Transform

36. Matrix Row Wavelet Encoding Store Back in Matrix 0 ’ ’ 0 0

37. Matrix Row Wavelet Encoding 0 ’ ’ 0 0 Only 3% – 30% are non-zero

38. Total Compression Lighting vector compression n Highly lossy n Compress to 0.1% – 1% Matrix compression n Essentially lossless encoding n Represent with 3% – 30% non-zero terms Total compression in sparse matrix-vector multiply n 3 – 4 orders of magnitude less work than full multiplication

39. Overall Relighting Algorithm Pre-compute (per scene) n Compute matrix in pixel basis n Wavelet transform rows n Quantize, store Interactive Relighting (each frame) n Wavelet transform lighting n Prioritize and retain N wavelet coefficients n Perform sparse-matrix vector multiplication

40. Overview Matrix formulation of relighting n Pre-computing the light-transport matrix n Compressing the matrix multiplication Results n Demonstration n Error analysis (Comparison with PRT) Future directions n Limitations n Areas for research

41. Overview Matrix formulation of relighting n Pre-computing the light-transport matrix n Compressing the matrix multiplication Results n Demonstration n Error analysis (Comparison with PRT) Future directions n Limitations n Areas for research

42. Demo

43. Error Analysis Compare to linear spherical harmonic approximation n [Sloan, Kautz, Snyder, 2002] Measure approximation quality in two ways: n Error in lighting n Error in output image pixels Main point (for detailed natural lighting) n Non-linear wavelets converge exponentially faster than linear harmonics

44. Error in Lighting: St Peter’s Basilica Approximation Terms Relative L 2 Error (%) Sph. Harmonics Non-linear Wavelets

45. Error in Output Image: Plant Scene Approximation Terms Relative L 2 Error (%) Sph. Harmonics Non-linear Wavelets

46. Output Image Comparison Top: Linear Spherical Harmonic Approximation Bottom: Non-linear Wavelet Approximation 252002,00020,000

47. Summary A viable representation for all-frequency lighting n Sparse non-linear wavelet approximation n 100 – 1000 times information reduction Efficient relighting from detailed environment maps

48. Future Improvements Matrix compression n Transport matrices are very large (over 1 GB) n We only compress matrix rows n Can also compress columns (images) Non-linear coefficient selection n Area-weighted priority works best n Develop theory for how to approximate input vector for minimum error in output vector

49. Signal Processing Taxonomy of Lighting Effects n “All-frequency” vs “Exclusively low frequency”

50. Future All-Frequency Lighting Research n Realistic materials with changing view n Local lighting effects Gehry’s E.M.P. Museum

51. Acknowledgments

52. Relighting with Changing View Matrix formulation still works n Output vector contains exit radiance at scene vertices rather than image pixels Main difference: Must assume diffuse surfaces

53. Plant Reference

54. Teapot Reference

55. Lighting Error

56. Light-Transport Matrix Columns

57. Light-Transport Matrix

58. Matrix Row Wavelet Encoding

59. Extract Row

60. Matrix Row Wavelet Encoding Wavelet Transform

61. Matrix Row Wavelet Encoding Wavelet Transform

62. Matrix Row Wavelet Encoding Wavelet Transform

63. Matrix Row Wavelet Encoding Wavelet Transform

64. Matrix Row Wavelet Encoding Store back in matrix

65. From Frank Gehry Architecture, Ragheb ed. 2001 Focus on Shadows