1. Real-Time Rendering and Interaction with Complex Lighting and Materials Ravi Ramamoorthi Rendering Laboratory Columbia University Intel: August 13, 2004

2. High quality real-time rendering  Photorealism, not just more polygons  Natural lighting, materials, shadows Interiors by architect Frank Gehry. Note rich lighting, ranging from localized sources to reflections off vast sheets of glass.

3. High quality real-time rendering  Photorealism, not just more polygons  Natural lighting, materials, shadows Real materials diverse and not easy to represent by simple parameteric models. Want to support measured reflectance. Glass Vase Glass Star (courtesy Intel) Peacock feather

4. High quality real-time rendering  Photorealism, not just more polygons  Natural lighting, materials, shadows Natural lighting creates a mix of soft diffuse and hard shadows. Agrawala et al. 00 small area light, sharp shadows soft and hard shadows Ng et al. 03

5. Demo

6. Industrial Relevance  Very difficult computational problem  Both preprocessing (hours to days) and real-time rendering  Related to computational problems in other areas  Driving application  Interactive 3D graphics major use of PCs  Realism increases use, applications  Leverages high-performance processors  Exploit parallelism (at pixel level), modern feature  SIMD instructions, MMX and SSE instructions

7. Applications  Entertainment: Lighting design  Architectural visualization  Material design: Automobile industry  Realistic Video games  Electronic commerce

8. Challenges  Illumination complexity  Material / view complexity  Transport complexity (shadows)  Putting it together Ramamoorthi and Hanrahan, SIGGRAPH 01, 02

9. Environment Maps Miller and Hoffman, 1984 Previous work: Blinn 76, Greene 86, Cabral 87, 99

10. Irradiance Environment Maps Incident Radiance (Illumination Environment Map) Irradiance Environment Map R N

11. Assumptions  Diffuse surfaces (relaxed later)  Distant illumination  No shadowing (relaxed later) Hence, Irradiance is a function of surface normal

12. Computing Irradiance  Classically, hemispherical integral for each pixel  Lambertian surface is like low pass filter  Frequency-space analysis Incident Radiance Irradiance

13. Analytic Irradiance Formula Lambertian surface acts like low-pass filter 0 01 2 Basri & Jacobs 01 Ramamoorthi & Hanrahan 01

14. 9 Parameter Approximation -201 2 0 1 2 Exact image Order 2 9 terms RMS Error = 1% For any illumination, average error < 3% [Basri Jacobs 01]

15. Real-Time Rendering Simple procedural rendering method (no textures)  Requires only matrix-vector multiply and dot-product  In software or NVIDIA vertex programming hardware Widely used in Games (AMPED for Microsoft Xbox), Movies (Pixar, Framestore CFC, …)

16. Convolution: General Materials Ramamoorthi and Hanrahan 01 Frequency: product Spatial: integral Spherical harmonic analysis

17. Natural Lighting, Realistic Materials Ramamoorthi and Hanrahan 02

18. Challenges  Illumination complexity  Material (BRDF)/view complexity  Transport complexity (shadows): Relighting  Putting it together Ng, Ramamoorthi and Hanrahan SIGGRAPH 03

19. Existing Fast Shadow Techniques We know how to render very hard, very soft shadows Sloan, Kautz, Snyder 2002 Shadows from smooth lighting (precomputed radiance transfer) Sen, Cammarano, Hanrahan, 2003 Shadows from point-lights (shadow maps, volumes)

20. Assumptions  Static geometry  Precomputation  Real-Time Rendering (relight all-frequency effects)

21. Relighting as a Matrix-Vector Multiply

22. Input Lighting (Cubemap Vector) Output Image (Pixel Vector) Transport Matrix Relighting as a Matrix-Vector Multiply

23. Problem Definition Matrix is Enormous  512 x 512 pixel images  6 x 64 x 64 cubemap environments Full matrix-vector multiplication is intractable  On the order of 10 10 operations per frame How to relight quickly?

24. Sparse Matrix-Vector Multiplication Choose data representations with mostly zeroes Vector: Use non-linear wavelet approximation on lighting Matrix: Wavelet-encode transport rows

25. Non-linear Wavelet Approximation Wavelets provide dual space / frequency locality  Large wavelets capture low frequency area lighting  Small wavelets capture high frequency compact features Non-linear Approximation  Use a dynamic set of approximating functions (depends on each frame’s lighting)  By contrast, linear approx. uses fixed set of basis functions (like 25 lowest frequency)  We choose 10’s - 100’s from a basis of 24,576 wavelets

26. Non-linear Wavelet Light Approximation Wavelet Transform

27. Non-linear Approximation Retain 0.1% – 1% terms Non-linear Wavelet Light Approximation

28. Performance: Timing Precomputation of Transport Matrix  Ray tracing: several days (depends on ray tracer)  Graphics hardware: 80,000 vertex buddha = 3 hours Rendering (2.8 GHz Pentium IV)  For 100 terms, 4-6 fps RGB, 9-15 fps monochrome  Relighting alone (core matrix-vector) somewhat faster  Use almost full CPU perf. (implementation details later)

29. Video

30. Broader Computational Relevance Sparse matrix-vector multiplication standard numerical problem  Challenging problem (giant matrices, vectors)  Even more demanding: we need real-time performance  Note: we sparsify vector, not matrix unlike most methods Most computation in tight inner loop  General kernel to focus optimization effort on

31. Broader Computational Relevance Sparse matrix-vector multiplication standard numerical problem  Challenging problem (giant matrices, vectors)  Even more demanding: we need real-time performance  Note: we sparsify vector, not matrix unlike most methods Most computation in tight inner loop  General kernel to focus optimization effort on Very parallelizable  Each pixel/vertex has same operations  Vectorization possible (even RGB parallel x3 faster)  Not yet exploited (but improved performance with SSE2)

32. Implementation Quantize matrix elements (6-8 bits)  Discard 0 coefficients, gives 3-20% matrix sparsity  Note that both matrix, vector now sparse Memory bandwidth  Don’t use bandwidth for lights not in current frame  Store matrix coeffs for each light contiguously in memory

33. Implementation Quantize matrix elements (6-8 bits)  Discard 0 coefficients, gives 3-20% matrix sparsity  Note that both matrix, vector now sparse Memory bandwidth  Don’t use bandwidth for lights not in current frame  Store matrix coeffs for each light contiguously in memory Cache coherence  Segment matrix into blocks 256 pixels  Store coefficients as 8 bit value and 8 bit block index

34. Sparse matrix-vector multiply is 3 – 4 orders of magnitude less work than full multiplication Total Compression Compress to 3% – 20% Compress to 0.1% – 1%

35. Challenges  Illumination complexity  Material (BRDF)/view complexity  Transport complexity (shadows)  Putting it together (relight and change view) Ng, Ramamoorthi and Hanrahan, SIGGRAPH 04

36. Changing Only The View

37. Problem Characterization 6D Precomputation Space  Distant Lighting(2D)  View(2D)  Rigid Geometry(2D) With ~ 100 samples per dimension ~ 10 12 samples total!! : Intractable computation, rendering

38. Factorization Approach 6D Transport ~ 10 12 samples ~ 10 8 samples 4D Visibility4D BRDF * =

39. Double Product Integral Relighting

40. Triple Product Integral Relighting Changing Surface Position Changing Camera Viewpoint

41. Triple Product Integral Relighting n 300,000 vertices n 6 x 64 x 64 cubemaps n 0.1% - 1% sparsity for visibility, BRDF n 3-5 seconds / frame

42. Relit Images

43. Broader Computational Relevance * = Same machinery applies to basic operation: multiplication Signal multiplication for audio, image compositing,…. Compressed signals/videos (e.g. wavelets JPEG 2000) Broadly relevant computational problem in computer graphics, multimedia, numerical analysis with little previous work

44. Challenges  Illumination complexity  Material (BRDF)/view complexity  Transport complexity (shadows)  Putting it together

45. Other approaches: local coherence Software: Image Relighting Framework [Enrique, Ramamoorthi] http://www.cs.columbia.edu/cg/relighting

46. Industrial Impact Environment maps: Lambertian 9 term formula [RH 01]  Standard in games, movies Spherical harmonic lighting [RH 01, SKS 02, 03, …]  Full session at SIGGRAPH 03, EGSR 04  Sessions at Game Developer Conference  Adopted in Microsoft DirectX 9  Used by Sony, Microsoft, …

47. Future Work  Exploit parallelism  Efficiency in precomputation  Dynamic scenes  Broader mathematical, computational problems

48. Acknowledgements  Collaborators and students: Ren Ng, Pat Hanrahan, Sameer Agarwal, Henrik Wann Jensen, Sebastian Enrique, Nolan Goodnight, Greg Humphreys  Funding: NSF and Intel http://www1.cs.columbia.edu/~ravir http://www1.cs.columbia.edu/~ravir/projects/rendering.html

49. The End  Questions?  More demos/videos if time permits

50. Research Projects  Mathematical foundations of appearance  High quality real-time rendering  Volumetric and Multiple Scattering Effects  Acquisition, Rendering with real lighting, materials  Complex lighting, materials in computer vision

51. Error in Lighting: St Peter’s Basilica Approximation Terms Relative L 2 Error (%) Sph. Harmonics Non-linear Wavelets Ng, Ramamoorthi, Hanrahan 03

52. Output Image Comparison Top: Linear Spherical Harmonic Approximation Bottom: Non-linear Wavelet Approximation 252002,00020,000

53. Matrix Compression Rates Teapot (Grace)Plant (St Peter’s) Light Res. Fraction non-zero Size Fraction non-zero Size 6 x 4 x 419%8.5 MB17%8.4 MB 6 x 8 x 816%34 MB15%31 MB 6 x 16 x 1614%113 MB14%115 MB 6 x 32 x 3210%185 MB12%394 MB 6 x 64 x 643.1%314 MB7.4%1.0 GB

54. Matrix Compression Rates Teapot (Grace)Plant (St Peter’s) Light Res. Fraction non-zero Size Fraction non-zero Size 6 x 4 x 419%8.5 MB17%8.4 MB 6 x 8 x 816%34 MB15%31 MB 6 x 16 x 1614%113 MB14%115 MB 6 x 32 x 3210%185 MB12%394 MB 6 x 64 x 643.1%314 MB7.4%1.0 GB

55. Double Product Integral Relighting Changing Surface Position Changing Camera Viewpoint Lighting Transport

56. Triple Product Integrals

57. Tripling Coefficient Density BasisComplexity PixelsN Fourier1/16 (1+3N) 2 = O(N 2 ) Spherical Harmonics 9/20 N 5/2 + 1/4 N 3/2 + 3/10 N 1/2 = O(N 5/2 ) Haar Wavelets2 – N + 3N log 4 N = O(N log N) General BasisN3N3

58. Summary of Wavelet Results  O(N) for Haar through factorization and dynamic programming exploiting regularity of coefficients  Actually essentially O(n) where n is number of terms used (in a nonlinear approximation)  Wavelets can represent all frequencies unlike pixels or spherical harmonics  Triple products are also efficient

59. Recent relevant SIGGRAPH papers 1.R. Ng, R. Ramamoorthi and P. Hanrahan, “Triple Product Wavelet Integrals for All-Frequency Relighting” SIGGRAPH 04 (ACM TOG 23(3)) 2.J. Lawrence, R. Ramamoorthi and S. Rusinkiewicz, “Efficient BRDF Importance Sampling using a Factored Representation” SIGGRAPH 04 (ACM TOG 23(3)) 3.R. Ng, R. Ramamoorthi and P. Hanrahan, “All-Frequency Shadows Using Non-Linear Wavelet Lighting Approximation” SIGGRAPH 03 (ACM TOG 22(3), pages 376-381). 4.S. Agarwal, R. Ramamoorthi, S. Belongie and H. Jensen, “Structured Importance Sampling of Environment Maps” SIGGRAPH 03 (ACM TOG 22(3), pages 605-612). 5.R. Ramamoorthi and P. Hanrahan, “Frequency Space Environment Map Rendering” SIGGRAPH 02 (ACM TOG 21(3), pages 517-526) 6.R. Ramamoorthi and P. Hanrahan, “A Signal-Processing Framework for Inverse Rendering” SIGGRAPH 01, pages 117-128. 7.R. Ramamoorthi and P. Hanrahan, “An Efficient Representation for Irradiance Environment Maps” SIGGRAPH 01, pages 497-500. 8.M. Agrawala, R. Ramamoorthi, A. Heirich and L. Moll, “Efficient Image- Based Methods for Rendering Soft Shadows” SIGGRAPH 00, pages 375-384.