1. Exploiting Temporal Coherence for Incremental All-Frequency Relighting Ryan OverbeckRavi Ramamoorthi Aner Ben-ArtziEitan Grinspun Columbia University Ng et al. 2003 Our Method 30 Wavelet lights per frame

2. Ng et al. 2003 Our Method 30 Wavelet lights per frame

3. CG Lighting Design Doom 3 2004 (www.doom3.com)www.doom3.com Unreal Championship 2 2004 (www.unrealchampionship2.com)www.unrealchampionship2.com The Lord of the Rings: The Two Towers 2002 (www.lordoftherings.net)www.lordoftherings.net Star Wars Episode I 1999 (thecia.com.au/reviews/s/star-wars-1.shtml)thecia.com.au/reviews/s/star-wars-1.shtml Video Games Movies

4. CG Lighting Design: Why is it hard? n Complex Lighting n Complex Materials n Takes Hours to Render n Need Interactivity n PRT n Sloan et al. 2002 n Ng et al. 2003 Hard Shadows Soft Shadows Specularities / Reflections Caustics

5. PRT Relighting: Matrix-Vector Multiply Slides from Ng et al. SIGGRAPH 2003

6. PRT Relighting: Matrix-Vector Multiply Input Lighting (Cubemap Vector) Output Image (Pixel Vector) Transport Matrix Slides from Ng et al. SIGGRAPH 2003

7. Light-Transport Matrix Columns Slides from Ng et al. SIGGRAPH 2003

8. Light-Transport Matrix Columns Slides from Ng et al. SIGGRAPH 2003

9. Light-Transport Matrix Rows Slides from Ng et al. SIGGRAPH 2003

10. Light-Transport Matrix Rows Slides from Ng et al. SIGGRAPH 2003

11. Light-Transport Matrix Rows Slides from Ng et al. SIGGRAPH 2003

12. Matrix Multiplication is Enormous Dimension n 512 x 512 pixel images ( ) n 6 x 64 x 64 cubemap ( ) Full matrix-vector multiplication is intractable n On the order of 10 10 operations per frame PRT exploits coherence to enable real-time rendering Slides from Ng et al. SIGGRAPH 2003

13. Signal / Spatial Coherence [Sloan et al. 2003] [Liu et al. 2004] PRT: Exploiting Coherence Image / Vertex Colors Transport Matrix Lighting Vector Angular Coherence [Ng et al. 2003] 30 – 100 Wavelet Lights Temporal Coherence [Our Contribution]

14. Previous Work: PRT n Dorsey, J., Arvo, J., and Greenberg, D. 1995. Interactive Design of Complex Time-Dependent Lighting. In IEEE Computer Graphics and Applications, 15(2): 26-36. n Ramamoorthi, R., and Hanrahan, P. 2001. An efficient representation for irradiance environment maps. In Proceedings of SIGGRAPH 2001, 497-500. n Sloan, P., Kautz, J., Snyder, J. Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low- Frequency Environments. In Proceedings of SIGGRAPH 2002. n Ng R., Ramamoorthi R., Hanrahan P. All-frequency shadows using non-linear wavelet lighting approximation. ACM TOG(SIGGRAPH 03) 22, 3 (2003), 376-381. n Sloan P., Hall J., Hart. J, Snyder J. Clustered principal components for precomputed radiance transfer. ACM TOG (SIGGRAPH 03) 22, 3 (2003), 382-391. n Wang R., Tran J., Luebke D. All-frequency relighting of non-diffuse objects using separable BRDF approximation. In EGSR (2004), pp.345-354.

15. Previous Work: PRT Dorsey, J., Arvo, J., and Greenberg, D. 1995. Interactive Design of Complex Time-Dependent Lighting. In IEEE Computer Graphics and Applications, 15(2): 26-36. Ng R., Ramamoorthi R., Hanrahan P. All-frequency shadows using non-linear wavelet lighting approximation. ACM TOG(SIGGRAPH 03) 22, 3 (2003), 376-381. Sloan, P., Kautz, J., Snyder, J. Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Environments. In Proceedings of SIGGRAPH 2002 Ng R., Ramamoorthi R., Hanrahan P. Triple product wavelet integrals for all-frequency relighting. ACM TOG (SIGGRAPH 04) 23, 3 (2004), 475-485. Sloan P., Hall J., Hart. J, Snyder J. Clustered principal components for precomputed radiance transfer. ACM TOG (SIGGRAPH 03) 22, 3 (2003), 382-391. Sloan P., Luna B., Snyder J. Local, deformable precomputed radiance transfer. ACM TOG (SIGGRAPH 05) 24, 4 (2005), 1216-1224. Wang R., Tran J., Luebke D. All-frequency relighting of non-diffuse objects using separable BRDF approximation. In EGSR (2004), pp.345-354. Wang R., Tran J., Luebke D. All-frequency interactive relighting of translucent objects with single and multiple scattering. ACM TOG (SIGGRAPH 05) 24, 3 (2005), 1202-1207. Zhou K., Hu Y., Lin S., Guo B., Shum H. Precomputed shadow fields for dynamic scenes. ACM TOG (SIGGRAPH 05) 25, 3 (2005). Ben-Artzi A., Overbeck R., Ramamoorthi R. Real-time BRDF editing in complex lighting. ACM TOG (SIGGRAPH 06) (2006). Wang R., Luebke D., Humphreys G., Ng R. Efficient wavelet rotation for environment map rendering. In EGSR (2006). Kontkanen J., Turquin E., Holzschuch N., Sillion F. Wavelet radiance transport for real-time indirect lighting. In EGSR (2006) Einarsson P., Chabert C., Jones A., Lamond B., Ma A., Hawkins T., Sylwan S., Debevec P. Relighting human locomotion with flowed reflectance fields. In EGSR (2006) This Session

16. Previous Work: PRT Dorsey, J., Sillion, F., and Greenberg, D. 1991. Design and simulation of opera lighting and projection effects. In Computer Graphics (Proceedings of SIGGRAPH 91), vol. 25, 41-50. Ng R., Ramamoorthi R., Hanrahan P. All-frequency shadows using non-linear wavelet lighting approximation. ACM TOG(SIGGRAPH 03) 22, 3 (2003), 376-381. Sloan, P., Kautz, J., Snyder, J. Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Environments. In Proceedings of SIGGRAPH 2002 Ng R., Ramamoorthi R., Hanrahan P. Triple product wavelet integrals for all-frequency relighting. ACM TOG (SIGGRAPH 04) 23, 3 (2004), 475-485. Sloan P., Hall J., Hart. J, Snyder J. Clustered principal components for precomputed radiance transfer. ACM TOG (SIGGRAPH 03) 22, 3 (2003), 382-391. Sloan P., Luna B., Snyder J. Local, deformable precomputed radiance transfer. ACM TOG (SIGGRAPH 05) 24, 4 (2005), 1216-1224. Wang R., Tran J., Luebke D. All-frequency relighting of non-diffuse objects using separable BRDF approximation. In EGSR (2004), pp.345-354. Wang R., Tran J., Luebke D. All-frequency interactive relighting of translucent objects with single and multiple scattering. ACM TOG (SIGGRAPH 05) 24, 3 (2005), 1202-1207. Zhou K., Hu Y., Lin S., Guo B., Shum H. Precomputed shadow fields for dynamic scenes. ACM TOG (SIGGRAPH 05) 25, 3 (2005). Ben-Artzi A., Overbeck R., Ramamoorthi R. Real-time BRDF editing in complex lighting. ACM TOG (SIGGRAPH 06) (2006). Wang R., Luebke D., Humphreys G., Ng R. Efficient wavelet rotation for environment map rendering. In EGSR (2006). Kontkanen J., Turquin E., Holzschuch N., Sillion F. Wavelet radiance transport for real-time indirect lighting. In EGSR (2006) Einarsson P., Chabert C., Jones A., Lamond B., Ma A., Hawkins T., Sylwan S., Debevec P. Relighting human locomotion with flowed reflectance fields. In EGSR (2006) This Session Temporal Coherence [Our Contribution]

17. Our Method

18. Outline n Motivation / Previous Work n Basic Approach to Incremental Relighting n Analysis of Temporal Coherence in Lighting n Per-Band Incremental n Results

19. Basic Incremental Algorithm

21. More Compressible

22. Basic Incremental Algorithm

23. Basic Incremental

24. Basic Incremental: Problems Reference Incremental Frame 0

25. Basic Incremental: Problems Reference Incremental Frame 30

26. Basic Incremental: Problems Reference Incremental Frame 75 Ghost Shadows

27. Basic Incremental: Problems Reference Incremental Frame 125

28. Basic Incremental: Problems Reference Incremental Frame 400

29. Basic Incremental: Problems

30. Outline n Motivation / Previous Work n Basic Approach for Incremental Relighting n Analysis of Temporal Coherence in Lighting n Per-Band Incremental n Results

31. Medium Frequency Low Frequency Frequency Analysis of Temporal Coherence High Frequency

32. Medium Frequency Low Frequency Frequency Analysis of Temporal Coherence High Frequency

33. Medium Frequency Low Frequency Frequency Analysis of Temporal Coherence High Frequency

34. Low Frequency Medium Frequency Frequency Analysis of Temporal Coherence High Frequency

35. Low Frequency Medium Frequency Frequency Analysis of Temporal Coherence High Frequency

36. Low Frequency Medium Frequency Frequency Analysis of Temporal Coherence High Frequency

37. Frequency Analysis of Temporal Coherence Medium Frequency Low Frequency High Frequency Temporal Wavelet Transform

38. Frequency Analysis of Temporal Coherence Medium Frequency Low Frequency High Frequency Temporal Wavelet Transform

39. Frequency Analysis of Temporal Coherence Angular Frequency Temporal Frequency 100 % 0 % ENERGY

40. Outline n Motivation / Previous Work n Basic Approach for Incremental Relighting n Analysis of Temporal Coherence in Lighting n Per-Band Incremental n Results

41. B = + + Non-Incremental B = T L 11 22 33 Incremental +  +  1 B 2 B 3 B Per-Band Incremental (PBI)

42. n Exhaustive n Try all combinations over all wavelet bands. n Very Slow. n Simple n Compare L1 Error in each band individually. Oracle (Incremental or Not)

43. Exhaustive n Try all combinations over all wavelet bands. n Very Slow. Simple n Compare L1 Error in each band individually. Oracle (Incremental or Not) L1 Distance Incremental Non-Incremental

44. Oracle (Incremental or Not) L1 Distance Incremental Non-Incremental n Exhaustive n Try all combinations over all wavelet bands. n Very Slow. n Simple n Compare L1 Error in each band individually. n Almost zero overhead. n Comparable results to Exhaustive.

45. PBI vs. Basic Incremental

46. Outline n Motivation / Previous Work n Basic Approach for Incremental Relighting n Analysis of Temporal Coherence in Lighting n Per-Band Incremental n Results

47. Results

50. Per-Band Incremental 30 Wavelets 16 14 13 12 15 55 7085 Frame Percentage L1 Error Simple Exhaustive

51. Results

52. Per-Band Incremental 30 Wavelets 16 14 13 12 15 55 7085 Frame Percentage L1 Error Simple Exhaustive

53. Results

58. Per-Band Incremental n 3x – 4x Speed / Quality Improvement n Progressively Convergent n Minimal Overhead

59. Minimal Overhead n ~ 100 Lines of Code (Pseudo-code in paper). n < 10 % Memory Overheads n Speed: n Average case (30 wavelets): 5 % Overhead

60. Per-Band Incremental

61. Applies to All (?) Wavelet PRT Frameworks n Old PRT n Standard All-Frequency PRT [Ng et al. 2003] n Current PRT n Clustered PCA [Liu et al. 2004] n Changing View with Separable BRDF Approximation [Wang et al. 2004] n Future PRT n Real-time BRDF Editing [Ben-Artzi et al. SIGGRAPH 2006]

62. PBI: CPCA and Complex BRDFs

63. Summary n Added temporal coherence to PRT. n Analysis of temporal coherence in lighting. n Per-Band Incremental algorithm. n Minimal Overhead. n Minimal Code. n Applies to all (?) current PRT frameworks. n See Ben-Artzi et al. Real-time BRDF Editing at SIGGRAPH 2006

64. Future n Limitations n Shadows get softer during rotation. n Good for Lighting Design. n Bad for Video Games.

65. Future n Temporal Coherence n PRT Animation. n Beyond PRT.

66. Acknowledgments n Ren Ng for wavelet relighting framework code and slides from SIGGRAPH 2003. n Microsoft / Sloan et al. for Spherical Harmonics relighting framework in DirectX SDK. n This research was funded in part by a Sloan Research Fellowship and NSF grants #0305322 and #0446916. n Londa Fiorella for getting me here. Thank you!