Exploiting Temporal Coherence for Incremental All-Frequency Relighting Ryan OverbeckRavi Ramamoorthi Aner Ben-ArtziEitan Grinspun Columbia University Ng et al Our Method 30 Wavelet lights per frame

Ng et al Our Method 30 Wavelet lights per frame

CG Lighting Design Doom ( Unreal Championship ( The Lord of the Rings: The Two Towers 2002 ( Star Wars Episode I 1999 (thecia.com.au/reviews/s/star-wars-1.shtml)thecia.com.au/reviews/s/star-wars-1.shtml Video Games Movies

CG Lighting Design: Why is it hard? n Complex Lighting n Complex Materials n Takes Hours to Render n Need Interactivity n PRT n Sloan et al n Ng et al Hard Shadows Soft Shadows Specularities / Reflections Caustics

PRT Relighting: Matrix-Vector Multiply Slides from Ng et al. SIGGRAPH 2003

PRT Relighting: Matrix-Vector Multiply Input Lighting (Cubemap Vector) Output Image (Pixel Vector) Transport Matrix Slides from Ng et al. SIGGRAPH 2003

Light-Transport Matrix Columns Slides from Ng et al. SIGGRAPH 2003

Light-Transport Matrix Columns Slides from Ng et al. SIGGRAPH 2003

Light-Transport Matrix Rows Slides from Ng et al. SIGGRAPH 2003

Light-Transport Matrix Rows Slides from Ng et al. SIGGRAPH 2003

Light-Transport Matrix Rows Slides from Ng et al. SIGGRAPH 2003

Matrix Multiplication is Enormous Dimension n 512 x 512 pixel images ( ) n 6 x 64 x 64 cubemap ( ) Full matrix-vector multiplication is intractable n On the order of operations per frame PRT exploits coherence to enable real-time rendering Slides from Ng et al. SIGGRAPH 2003

Signal / Spatial Coherence [Sloan et al. 2003] [Liu et al. 2004] PRT: Exploiting Coherence Image / Vertex Colors Transport Matrix Lighting Vector Angular Coherence [Ng et al. 2003] 30 – 100 Wavelet Lights Temporal Coherence [Our Contribution]

Previous Work: PRT n Dorsey, J., Arvo, J., and Greenberg, D Interactive Design of Complex Time-Dependent Lighting. In IEEE Computer Graphics and Applications, 15(2): n Ramamoorthi, R., and Hanrahan, P An efficient representation for irradiance environment maps. In Proceedings of SIGGRAPH 2001, n Sloan, P., Kautz, J., Snyder, J. Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low- Frequency Environments. In Proceedings of SIGGRAPH n Ng R., Ramamoorthi R., Hanrahan P. All-frequency shadows using non-linear wavelet lighting approximation. ACM TOG(SIGGRAPH 03) 22, 3 (2003), n Sloan P., Hall J., Hart. J, Snyder J. Clustered principal components for precomputed radiance transfer. ACM TOG (SIGGRAPH 03) 22, 3 (2003), n Wang R., Tran J., Luebke D. All-frequency relighting of non-diffuse objects using separable BRDF approximation. In EGSR (2004), pp

Previous Work: PRT Dorsey, J., Arvo, J., and Greenberg, D Interactive Design of Complex Time-Dependent Lighting. In IEEE Computer Graphics and Applications, 15(2): Ng R., Ramamoorthi R., Hanrahan P. All-frequency shadows using non-linear wavelet lighting approximation. ACM TOG(SIGGRAPH 03) 22, 3 (2003), Sloan, P., Kautz, J., Snyder, J. Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Environments. In Proceedings of SIGGRAPH 2002 Ng R., Ramamoorthi R., Hanrahan P. Triple product wavelet integrals for all-frequency relighting. ACM TOG (SIGGRAPH 04) 23, 3 (2004), Sloan P., Hall J., Hart. J, Snyder J. Clustered principal components for precomputed radiance transfer. ACM TOG (SIGGRAPH 03) 22, 3 (2003), Sloan P., Luna B., Snyder J. Local, deformable precomputed radiance transfer. ACM TOG (SIGGRAPH 05) 24, 4 (2005), Wang R., Tran J., Luebke D. All-frequency relighting of non-diffuse objects using separable BRDF approximation. In EGSR (2004), pp Wang R., Tran J., Luebke D. All-frequency interactive relighting of translucent objects with single and multiple scattering. ACM TOG (SIGGRAPH 05) 24, 3 (2005), Zhou K., Hu Y., Lin S., Guo B., Shum H. Precomputed shadow fields for dynamic scenes. ACM TOG (SIGGRAPH 05) 25, 3 (2005). Ben-Artzi A., Overbeck R., Ramamoorthi R. Real-time BRDF editing in complex lighting. ACM TOG (SIGGRAPH 06) (2006). Wang R., Luebke D., Humphreys G., Ng R. Efficient wavelet rotation for environment map rendering. In EGSR (2006). Kontkanen J., Turquin E., Holzschuch N., Sillion F. Wavelet radiance transport for real-time indirect lighting. In EGSR (2006) Einarsson P., Chabert C., Jones A., Lamond B., Ma A., Hawkins T., Sylwan S., Debevec P. Relighting human locomotion with flowed reflectance fields. In EGSR (2006) This Session

Previous Work: PRT Dorsey, J., Sillion, F., and Greenberg, D Design and simulation of opera lighting and projection effects. In Computer Graphics (Proceedings of SIGGRAPH 91), vol. 25, Ng R., Ramamoorthi R., Hanrahan P. All-frequency shadows using non-linear wavelet lighting approximation. ACM TOG(SIGGRAPH 03) 22, 3 (2003), Sloan, P., Kautz, J., Snyder, J. Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Environments. In Proceedings of SIGGRAPH 2002 Ng R., Ramamoorthi R., Hanrahan P. Triple product wavelet integrals for all-frequency relighting. ACM TOG (SIGGRAPH 04) 23, 3 (2004), Sloan P., Hall J., Hart. J, Snyder J. Clustered principal components for precomputed radiance transfer. ACM TOG (SIGGRAPH 03) 22, 3 (2003), Sloan P., Luna B., Snyder J. Local, deformable precomputed radiance transfer. ACM TOG (SIGGRAPH 05) 24, 4 (2005), Wang R., Tran J., Luebke D. All-frequency relighting of non-diffuse objects using separable BRDF approximation. In EGSR (2004), pp Wang R., Tran J., Luebke D. All-frequency interactive relighting of translucent objects with single and multiple scattering. ACM TOG (SIGGRAPH 05) 24, 3 (2005), Zhou K., Hu Y., Lin S., Guo B., Shum H. Precomputed shadow fields for dynamic scenes. ACM TOG (SIGGRAPH 05) 25, 3 (2005). Ben-Artzi A., Overbeck R., Ramamoorthi R. Real-time BRDF editing in complex lighting. ACM TOG (SIGGRAPH 06) (2006). Wang R., Luebke D., Humphreys G., Ng R. Efficient wavelet rotation for environment map rendering. In EGSR (2006). Kontkanen J., Turquin E., Holzschuch N., Sillion F. Wavelet radiance transport for real-time indirect lighting. In EGSR (2006) Einarsson P., Chabert C., Jones A., Lamond B., Ma A., Hawkins T., Sylwan S., Debevec P. Relighting human locomotion with flowed reflectance fields. In EGSR (2006) This Session Temporal Coherence [Our Contribution]

Our Method

Outline n Motivation / Previous Work n Basic Approach to Incremental Relighting n Analysis of Temporal Coherence in Lighting n Per-Band Incremental n Results

Basic Incremental Algorithm

More Compressible

Basic Incremental Algorithm

Basic Incremental

Basic Incremental: Problems Reference Incremental Frame 0

Basic Incremental: Problems Reference Incremental Frame 30

Basic Incremental: Problems Reference Incremental Frame 75 Ghost Shadows

Basic Incremental: Problems Reference Incremental Frame 125

Basic Incremental: Problems Reference Incremental Frame 400

Basic Incremental: Problems

Outline n Motivation / Previous Work n Basic Approach for Incremental Relighting n Analysis of Temporal Coherence in Lighting n Per-Band Incremental n Results

Medium Frequency Low Frequency Frequency Analysis of Temporal Coherence High Frequency

Medium Frequency Low Frequency Frequency Analysis of Temporal Coherence High Frequency

Medium Frequency Low Frequency Frequency Analysis of Temporal Coherence High Frequency

Low Frequency Medium Frequency Frequency Analysis of Temporal Coherence High Frequency

Low Frequency Medium Frequency Frequency Analysis of Temporal Coherence High Frequency

Low Frequency Medium Frequency Frequency Analysis of Temporal Coherence High Frequency

Frequency Analysis of Temporal Coherence Medium Frequency Low Frequency High Frequency Temporal Wavelet Transform

Frequency Analysis of Temporal Coherence Medium Frequency Low Frequency High Frequency Temporal Wavelet Transform

Frequency Analysis of Temporal Coherence Angular Frequency Temporal Frequency 100 % 0 % ENERGY

Outline n Motivation / Previous Work n Basic Approach for Incremental Relighting n Analysis of Temporal Coherence in Lighting n Per-Band Incremental n Results

B = + + Non-Incremental B = T L Incremental +  +  1 B 2 B 3 B Per-Band Incremental (PBI)

n Exhaustive n Try all combinations over all wavelet bands. n Very Slow. n Simple n Compare L1 Error in each band individually. Oracle (Incremental or Not)

Exhaustive n Try all combinations over all wavelet bands. n Very Slow. Simple n Compare L1 Error in each band individually. Oracle (Incremental or Not) L1 Distance Incremental Non-Incremental

Oracle (Incremental or Not) L1 Distance Incremental Non-Incremental n Exhaustive n Try all combinations over all wavelet bands. n Very Slow. n Simple n Compare L1 Error in each band individually. n Almost zero overhead. n Comparable results to Exhaustive.

PBI vs. Basic Incremental

Outline n Motivation / Previous Work n Basic Approach for Incremental Relighting n Analysis of Temporal Coherence in Lighting n Per-Band Incremental n Results

Results

Per-Band Incremental 30 Wavelets Frame Percentage L1 Error Simple Exhaustive

Results

Per-Band Incremental 30 Wavelets Frame Percentage L1 Error Simple Exhaustive

Results

Per-Band Incremental n 3x – 4x Speed / Quality Improvement n Progressively Convergent n Minimal Overhead

Minimal Overhead n ~ 100 Lines of Code (Pseudo-code in paper). n < 10 % Memory Overheads n Speed: n Average case (30 wavelets): 5 % Overhead

Per-Band Incremental

Applies to All (?) Wavelet PRT Frameworks n Old PRT n Standard All-Frequency PRT [Ng et al. 2003] n Current PRT n Clustered PCA [Liu et al. 2004] n Changing View with Separable BRDF Approximation [Wang et al. 2004] n Future PRT n Real-time BRDF Editing [Ben-Artzi et al. SIGGRAPH 2006]

PBI: CPCA and Complex BRDFs

Summary n Added temporal coherence to PRT. n Analysis of temporal coherence in lighting. n Per-Band Incremental algorithm. n Minimal Overhead. n Minimal Code. n Applies to all (?) current PRT frameworks. n See Ben-Artzi et al. Real-time BRDF Editing at SIGGRAPH 2006

Future n Limitations n Shadows get softer during rotation. n Good for Lighting Design. n Bad for Video Games.

Future n Temporal Coherence n PRT Animation. n Beyond PRT.

Acknowledgments n Ren Ng for wavelet relighting framework code and slides from SIGGRAPH n Microsoft / Sloan et al. for Spherical Harmonics relighting framework in DirectX SDK. n This research was funded in part by a Sloan Research Fellowship and NSF grants # and # n Londa Fiorella for getting me here. Thank you!