Real-Time Rendering and Interaction with Complex Lighting and Materials Ravi Ramamoorthi Rendering Laboratory Columbia University Intel: August 13, 2004

High quality real-time rendering  Photorealism, not just more polygons  Natural lighting, materials, shadows Interiors by architect Frank Gehry. Note rich lighting, ranging from localized sources to reflections off vast sheets of glass.

High quality real-time rendering  Photorealism, not just more polygons  Natural lighting, materials, shadows Real materials diverse and not easy to represent by simple parameteric models. Want to support measured reflectance. Glass Vase Glass Star (courtesy Intel) Peacock feather

High quality real-time rendering  Photorealism, not just more polygons  Natural lighting, materials, shadows Natural lighting creates a mix of soft diffuse and hard shadows. Agrawala et al. 00 small area light, sharp shadows soft and hard shadows Ng et al. 03

Demo

Industrial Relevance  Very difficult computational problem  Both preprocessing (hours to days) and real-time rendering  Related to computational problems in other areas  Driving application  Interactive 3D graphics major use of PCs  Realism increases use, applications  Leverages high-performance processors  Exploit parallelism (at pixel level), modern feature  SIMD instructions, MMX and SSE instructions

Applications  Entertainment: Lighting design  Architectural visualization  Material design: Automobile industry  Realistic Video games  Electronic commerce

Challenges  Illumination complexity  Material / view complexity  Transport complexity (shadows)  Putting it together Ramamoorthi and Hanrahan, SIGGRAPH 01, 02

Environment Maps Miller and Hoffman, 1984 Previous work: Blinn 76, Greene 86, Cabral 87, 99

Irradiance Environment Maps Incident Radiance (Illumination Environment Map) Irradiance Environment Map R N

Assumptions  Diffuse surfaces (relaxed later)  Distant illumination  No shadowing (relaxed later) Hence, Irradiance is a function of surface normal

Computing Irradiance  Classically, hemispherical integral for each pixel  Lambertian surface is like low pass filter  Frequency-space analysis Incident Radiance Irradiance

Analytic Irradiance Formula Lambertian surface acts like low-pass filter Basri & Jacobs 01 Ramamoorthi & Hanrahan 01

9 Parameter Approximation Exact image Order 2 9 terms RMS Error = 1% For any illumination, average error < 3% [Basri Jacobs 01]

Real-Time Rendering Simple procedural rendering method (no textures)  Requires only matrix-vector multiply and dot-product  In software or NVIDIA vertex programming hardware Widely used in Games (AMPED for Microsoft Xbox), Movies (Pixar, Framestore CFC, …)

Convolution: General Materials Ramamoorthi and Hanrahan 01 Frequency: product Spatial: integral Spherical harmonic analysis

Natural Lighting, Realistic Materials Ramamoorthi and Hanrahan 02

Challenges  Illumination complexity  Material (BRDF)/view complexity  Transport complexity (shadows): Relighting  Putting it together Ng, Ramamoorthi and Hanrahan SIGGRAPH 03

Existing Fast Shadow Techniques We know how to render very hard, very soft shadows Sloan, Kautz, Snyder 2002 Shadows from smooth lighting (precomputed radiance transfer) Sen, Cammarano, Hanrahan, 2003 Shadows from point-lights (shadow maps, volumes)

Assumptions  Static geometry  Precomputation  Real-Time Rendering (relight all-frequency effects)

Relighting as a Matrix-Vector Multiply

Input Lighting (Cubemap Vector) Output Image (Pixel Vector) Transport Matrix Relighting as a Matrix-Vector Multiply

Problem Definition Matrix is Enormous  512 x 512 pixel images  6 x 64 x 64 cubemap environments Full matrix-vector multiplication is intractable  On the order of operations per frame How to relight quickly?

Sparse Matrix-Vector Multiplication Choose data representations with mostly zeroes Vector: Use non-linear wavelet approximation on lighting Matrix: Wavelet-encode transport rows

Non-linear Wavelet Approximation Wavelets provide dual space / frequency locality  Large wavelets capture low frequency area lighting  Small wavelets capture high frequency compact features Non-linear Approximation  Use a dynamic set of approximating functions (depends on each frame’s lighting)  By contrast, linear approx. uses fixed set of basis functions (like 25 lowest frequency)  We choose 10’s - 100’s from a basis of 24,576 wavelets

Non-linear Wavelet Light Approximation Wavelet Transform

Non-linear Approximation Retain 0.1% – 1% terms Non-linear Wavelet Light Approximation

Performance: Timing Precomputation of Transport Matrix  Ray tracing: several days (depends on ray tracer)  Graphics hardware: 80,000 vertex buddha = 3 hours Rendering (2.8 GHz Pentium IV)  For 100 terms, 4-6 fps RGB, 9-15 fps monochrome  Relighting alone (core matrix-vector) somewhat faster  Use almost full CPU perf. (implementation details later)

Video

Broader Computational Relevance Sparse matrix-vector multiplication standard numerical problem  Challenging problem (giant matrices, vectors)  Even more demanding: we need real-time performance  Note: we sparsify vector, not matrix unlike most methods Most computation in tight inner loop  General kernel to focus optimization effort on

Broader Computational Relevance Sparse matrix-vector multiplication standard numerical problem  Challenging problem (giant matrices, vectors)  Even more demanding: we need real-time performance  Note: we sparsify vector, not matrix unlike most methods Most computation in tight inner loop  General kernel to focus optimization effort on Very parallelizable  Each pixel/vertex has same operations  Vectorization possible (even RGB parallel x3 faster)  Not yet exploited (but improved performance with SSE2)

Implementation Quantize matrix elements (6-8 bits)  Discard 0 coefficients, gives 3-20% matrix sparsity  Note that both matrix, vector now sparse Memory bandwidth  Don’t use bandwidth for lights not in current frame  Store matrix coeffs for each light contiguously in memory

Implementation Quantize matrix elements (6-8 bits)  Discard 0 coefficients, gives 3-20% matrix sparsity  Note that both matrix, vector now sparse Memory bandwidth  Don’t use bandwidth for lights not in current frame  Store matrix coeffs for each light contiguously in memory Cache coherence  Segment matrix into blocks 256 pixels  Store coefficients as 8 bit value and 8 bit block index

Sparse matrix-vector multiply is 3 – 4 orders of magnitude less work than full multiplication Total Compression Compress to 3% – 20% Compress to 0.1% – 1%

Challenges  Illumination complexity  Material (BRDF)/view complexity  Transport complexity (shadows)  Putting it together (relight and change view) Ng, Ramamoorthi and Hanrahan, SIGGRAPH 04

Changing Only The View

Problem Characterization 6D Precomputation Space  Distant Lighting(2D)  View(2D)  Rigid Geometry(2D) With ~ 100 samples per dimension ~ samples total!! : Intractable computation, rendering

Factorization Approach 6D Transport ~ samples ~ 10 8 samples 4D Visibility4D BRDF * =

Double Product Integral Relighting

Triple Product Integral Relighting Changing Surface Position Changing Camera Viewpoint

Triple Product Integral Relighting n 300,000 vertices n 6 x 64 x 64 cubemaps n 0.1% - 1% sparsity for visibility, BRDF n 3-5 seconds / frame

Relit Images

Broader Computational Relevance * = Same machinery applies to basic operation: multiplication Signal multiplication for audio, image compositing,…. Compressed signals/videos (e.g. wavelets JPEG 2000) Broadly relevant computational problem in computer graphics, multimedia, numerical analysis with little previous work

Challenges  Illumination complexity  Material (BRDF)/view complexity  Transport complexity (shadows)  Putting it together

Other approaches: local coherence Software: Image Relighting Framework [Enrique, Ramamoorthi]

Industrial Impact Environment maps: Lambertian 9 term formula [RH 01]  Standard in games, movies Spherical harmonic lighting [RH 01, SKS 02, 03, …]  Full session at SIGGRAPH 03, EGSR 04  Sessions at Game Developer Conference  Adopted in Microsoft DirectX 9  Used by Sony, Microsoft, …

Future Work  Exploit parallelism  Efficiency in precomputation  Dynamic scenes  Broader mathematical, computational problems

Acknowledgements  Collaborators and students: Ren Ng, Pat Hanrahan, Sameer Agarwal, Henrik Wann Jensen, Sebastian Enrique, Nolan Goodnight, Greg Humphreys  Funding: NSF and Intel

The End  Questions?  More demos/videos if time permits

Research Projects  Mathematical foundations of appearance  High quality real-time rendering  Volumetric and Multiple Scattering Effects  Acquisition, Rendering with real lighting, materials  Complex lighting, materials in computer vision

Error in Lighting: St Peter’s Basilica Approximation Terms Relative L 2 Error (%) Sph. Harmonics Non-linear Wavelets Ng, Ramamoorthi, Hanrahan 03

Output Image Comparison Top: Linear Spherical Harmonic Approximation Bottom: Non-linear Wavelet Approximation ,00020,000

Matrix Compression Rates Teapot (Grace)Plant (St Peter’s) Light Res. Fraction non-zero Size Fraction non-zero Size 6 x 4 x 419%8.5 MB17%8.4 MB 6 x 8 x 816%34 MB15%31 MB 6 x 16 x 1614%113 MB14%115 MB 6 x 32 x 3210%185 MB12%394 MB 6 x 64 x 643.1%314 MB7.4%1.0 GB

Matrix Compression Rates Teapot (Grace)Plant (St Peter’s) Light Res. Fraction non-zero Size Fraction non-zero Size 6 x 4 x 419%8.5 MB17%8.4 MB 6 x 8 x 816%34 MB15%31 MB 6 x 16 x 1614%113 MB14%115 MB 6 x 32 x 3210%185 MB12%394 MB 6 x 64 x 643.1%314 MB7.4%1.0 GB

Double Product Integral Relighting Changing Surface Position Changing Camera Viewpoint Lighting Transport

Triple Product Integrals

Tripling Coefficient Density BasisComplexity PixelsN Fourier1/16 (1+3N) 2 = O(N 2 ) Spherical Harmonics 9/20 N 5/2 + 1/4 N 3/2 + 3/10 N 1/2 = O(N 5/2 ) Haar Wavelets2 – N + 3N log 4 N = O(N log N) General BasisN3N3

Summary of Wavelet Results  O(N) for Haar through factorization and dynamic programming exploiting regularity of coefficients  Actually essentially O(n) where n is number of terms used (in a nonlinear approximation)  Wavelets can represent all frequencies unlike pixels or spherical harmonics  Triple products are also efficient

Recent relevant SIGGRAPH papers 1.R. Ng, R. Ramamoorthi and P. Hanrahan, “Triple Product Wavelet Integrals for All-Frequency Relighting” SIGGRAPH 04 (ACM TOG 23(3)) 2.J. Lawrence, R. Ramamoorthi and S. Rusinkiewicz, “Efficient BRDF Importance Sampling using a Factored Representation” SIGGRAPH 04 (ACM TOG 23(3)) 3.R. Ng, R. Ramamoorthi and P. Hanrahan, “All-Frequency Shadows Using Non-Linear Wavelet Lighting Approximation” SIGGRAPH 03 (ACM TOG 22(3), pages ). 4.S. Agarwal, R. Ramamoorthi, S. Belongie and H. Jensen, “Structured Importance Sampling of Environment Maps” SIGGRAPH 03 (ACM TOG 22(3), pages ). 5.R. Ramamoorthi and P. Hanrahan, “Frequency Space Environment Map Rendering” SIGGRAPH 02 (ACM TOG 21(3), pages ) 6.R. Ramamoorthi and P. Hanrahan, “A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 01, pages R. Ramamoorthi and P. Hanrahan, “An Efficient Representation for Irradiance Environment Maps” SIGGRAPH 01, pages M. Agrawala, R. Ramamoorthi, A. Heirich and L. Moll, “Efficient Image- Based Methods for Rendering Soft Shadows” SIGGRAPH 00, pages