1. Simulation Photoréaliste de l'Éclairage en Synthèse d'Images Nicolas Holzschuch Habilitation à Diriger des Recherches Université Joseph Fourier Grenoble I

2. Overview Motivations Previous work Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

3. Overview Motivations Previous work Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

4. Photorealistic Image Synthesis Starting from the 3D model of a scene… © C. Soler, 2007. Model by L. Boissieux.

5. Photorealistic Image Synthesis …we get a picture of this virtual world © C. Soler, 2007. Model by L. Boissieux.

6. Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math)

7. Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) Model from DaimlerChrysler / RealReflect IST project / picture by C. Soler

8. Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) © P. Müller

9. Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) © Valve, 2006

10. Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) © Lucas Digital Ltd., 2002

11. Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) Why is it difficult?

12. Motivations Why are we doing this? Why is it difficult? –Interactions between light and matter Complex local effects (reflections) –Self-dependent –Long distance interactions –Differences of scale/frequency –Time/quality compromise –Blurry effects are costlier –Too many computations?

13. Overview Motivations Timeline Selected contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

14. Timeline

15. Timeline: positions

16. Timeline: research themes

17. Timeline: projects and contracts

18. Overview Motivations Timeline Selected contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

19. Overview Motivations Timeline Selected contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

20. Wavelet Radiosity: credits 1993-1996: iMAGIS –Under the supervision of C. Puech, F. Sillion, G.Drettakis 1997-2000: ISA/LORIA –Working with J.-C. Paul, L. Alonso, F. Cuny, X. Cavin, C. Winkler, H. Barthélémy, S. Petitjean… 2000-2004: iMAGIS/GRAVIR –Working with L. Alonso (at ISA), C. Damez, F. Sillion…

21. Wavelet radiosity Extension of Hierarchical Radiosity –[Gortler et al., 1993] Uses wavelets for basis functions –Haar wavelets (pw. constant) equivalent to HR M 2, M 3 wavelets : linear, quadratic functions Higher order wavelets should work better –Better representation for smooth functions Experimental study: no, they dont! –[Wilmott & Heckbert, 1997]

22. Wavelet radiosity Everyone stopped working on WR –…except us at ISA/LORIA Point is: it should really work better –So why not? Experimental studies Everyone had used a na ï ve approach: –Just changing the basis function –But higher order wavelets are costlier n 2 for each patch n 4 for each interaction

23. Wavelet radiosity Na ï ve implementation doesn t work Must change the approach Adapt the algorithm: –Meshing strategy –No link storage + shooting –Refinement oracle –Visibility tests

24. Why didnt it work?

25. Wavelets need quad/triangular patches

26. Why didnt it work? Patches are then refined The initial triangulation binds the algorithm And each patch is n 2 times costlier

27. Extended mesh Keep the original mesh Adapt the radiosity algorithm [Holzschuch, Cuny, Alonso, EGWR 2000]

28. Extended mesh Separation between function and geometry

29. Link storage and consequences Biggest issue is memory cost –n 4 for each interaction –Solution: dont store them! –[ Stamminger, Schirmacher, Slusallek & Seidel, EG 1998] But think further: –Gathering/shooting –Gathering works better with HR –…except if you dont store the interactions We should use shooting! –…then it makes more sense not to store links!

30. Higher-Order Wavelets Other changes (refinement oracle, visibility) This time it works: –Smaller memory costs –Faster computations [Cuny, Holzschuch, Alonso, EG 2000]

31. Memory costs Observed memory costs for the different wavelet bases (rendering time follows similar curve) Consistent with [Willmott & Heckbert 1997]!

32. Higher order wavelets II Efficient global illumination algorithm –Uses higher-order wavelets –Robust, works on real-world scenes –Memory efficient So, is everything perfect? –Actually, no. –Shadow boundaries are a big issue

33. Shadow boundaries Discontinuity: the oracle keeps refining Each patch is n 2 times more costly –We lose all the ground we gained elsewhere

34. Discontinuity meshing Well-known method: –Shadow boundary position is known –Cut the mesh according to these discontinuities –Refine the resulting mesh Impossible with wavelet radiosity: –Wavelets only defined over quads/triangles –…but we already have a workaround!

35. DM and Higher-Order Wavelets [Holzschuch and Alonso, EGSR 2004] Discontinuities are introduced only if necessary –Use regular subdivision as much as possible Modified refinement oracle: –If refinement could have been avoided using DM –Then Subdivide patch using discontinuity –Embed the new patches inside quads –Regular subdivision over the new quads

36. DM and Higher-Order Wavelets M2 wavelets, point light source, no DM [Holzschuch and Alonso, EGSR 2004]

37. DM and Higher-Order Wavelets M2 wavelets, point light source, with DM [Holzschuch and Alonso, EGSR 2004]

38. DM and Higher-Order Wavelets M3 wavelets, area light source, with DM [Holzschuch and Alonso, EGSR 2004]

39. Wavelet Radiosity: summary Compact representation of illumination Applications: –Cooperation with Nancy School of Architecture –We tested our algorithms on real-world scenes –Great for robustness Research: –Real scientific problems came up –We had to solve them, resulting in nice papers Transfer: –The software was transferred to a start-up –VSP-Technology, headed by F. Cuny

40. Wavelet radiosity: looking back Shadow boundaries is an interesting issue Without DM: –90 % of computation time for shadows With DM: –90 % of implementation time for shadows Only important for (direct lighting+directly visible) Indirect lighting can use low quality version Waste of time?

41. Overview Motivations Timeline Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

42. Frequency Analysis: credits Joint research work with MIT/CSAIL At MIT/CSAIL: F. Durand, E. Chan At ARTIS/GRAVIR: C. Soler, F. Sillion Financed by an INRIA Équipe Associée [Durand, Holzschuch, Soler, Chan, Sillion, SIGGRAPH 2005]

43. Illumination effects Blurry reflections: From [Ramamoorthi and Hanrahan 2001]

44. Illumination effects Shadow boundaries: © U. Assarsson 2005. Point light source Area light source

45. Illumination effects Indirect lighting is usually blurry: Complete lighting © C. Soler 2005.

46. Illumination effects Indirect lighting is usually blurry: Indirect lighting only Direct lighting only © C. Soler 2005.

47. Frequency aspects of light transport Blurriness = frequency content –Sharp variations: high frequency –Smooth variations: low frequency All effects are expressed as frequency content: –Diffuse shading: low frequency –Shadows: introduce high frequencies –Indirect lighting: tends to be low frequency

48. Problem statement How does light interaction in a scene explain the frequency content? Theoretical framework: –Understand the frequency spectrum of the radiance function –From equations of light transport

49. Unified framework: Spatial frequency (e.g. shadows, textures) Angular frequency (e.g. blurry highlight)

50. Disclaimer: not Fourier optics We do not consider wave optics, interference, diffraction Only geometrical optics From [Hecht]

51. Disclaimer: not Fourier optics We do not consider wave optics, interference, diffraction Only geometrical optics From [Hecht]

52. Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

53. Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

54. Previous work Vast body of literature: –Light field sampling –Perceptually-based rendering –Wavelets for Computer Graphics –Irradiance caching –Photon mapping –… We focus on frequency analysis in graphics: –Light field sampling –Reflection as a convolution

55. Light field sampling [Chai et al. 00, Isaksen et al. 00, Stewart et al. 03] –Light field spectrum as a function of object distance –No BRDF, occlusion ignored From [Chai et al. 2000]

56. Signal processing for reflection [Ramamoorthi & Hanrahan 01, Basri & Jacobs 03] Reflection on a curved surface is a convolution Direction only From [Ramamoorthi and Hanrahan 2001]

57. Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

58. Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc.

59. Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal

60. Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Transport

61. Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Signal 1 Transport

62. Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Transport Occlusion Signal 2

63. Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Transport Occlusion Signal 3 Transport

64. Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Transport Occlusion Signal 4 Transport BRDF

65. Local light field 4D light field, around a central ray Central ray

66. Local light field 4D light field, around a central ray We study its spectrum during transport

67. Local light field 4D light field, around a central ray We study its spectrum during transport

68. Local light field 4D light field, around a central ray We study its spectrum during transport

69. Local light field We give explanations in 2D –Local light field is therefore 2D See SIGGRAPH paper for extension to 3D

70. Local light field parameterization Space and angle space angle Central ray

71. Local light field representation Density plot: Area light source Space Angle

72. Local light field Fourier spectrum We are interested in the Fourier spectrum of the local light field Also represented as a density plot

73. Local light field Fourier spectrum Spatial frequency Angular frequency Spectrum of an area light source:

74. Basics of Fourier analysis Spectrum corresponds to blurriness: –Sharpest feature has size 1/F max Convolution theorem: –Multiplication convolution Classical spectra: –Box sinc –Dirac constant

75. Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

76. Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

77. Example scene Blockers Light source Receiver

78. Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

79. Transport in free space Shear Space Angle

80. Transport in free space Shear Space Angle Shear Spatial frequency Angular freq. Spatial frequency Angular freq.

82. Transport Shear This is consistent with light field spectra [Chai et al. 00, Isaksen et al. 00] From [Chai et al. 2000]

83. Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

84. Occlusion: multiplication Occlusion is a multiplication in ray space –Convolution in Fourier space Creates new spatial frequency content –Related to the spectrum of the blockers

86. Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

88. Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

89. BRDF integration Outgoing light: –Integration of incoming light times BRDF Outgoing light Incoming light BRDF

90. BRDF integration Ray-space: convolution –Outgoing light: convolution of incoming light and BRDF –For rotationally-invariant BRDFs Fourier domain: multiplication –Outgoing spectrum: multiplication of incoming spectrum and BRDF spectrum

91. BRDF in Fourier: multiplication = BRDF is bandwidth-limiting in angle

92. Example: diffuse BRDF Convolve by constant: –multiply by Dirac –Only spatial frequencies remain =

93. Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

94. Curved receiver Reduce to the case of a planar surface: –Unroll the curved receiver Equivalent to changing angular content: –Linear effect on angular dimension –No effect on spatial dimension Shear in the angular dimension

96. Transforms: summary Radiance/FourierEffect TransportShear OcclusionMultiplication/ConvolutionAdds spatial frequencies BRDFConvolution/MultiplicationRemoves angular frequencies CurvatureShear

97. More effects in paper Cosine term (multiplication/convolution) Fresnel term (multiplication/convolution) Texture mapping (multiplication/convolution) Central incidence angle (scaling) Separable BRDF Spatially varying BRDF (semi-convolution) …and extension to 3D

98. Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

99. Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

100. Case study: diffuse soft shadows Decreasing blockers size: –First high-frequencies increase –Then only low frequency –Non-monotonic behavior

101. Diffuse soft shadows (2) Occlusion : convolution in Fourier –creates high frequencies –Blockers scaled down spectrum scaled up Fourier space v (angle) x (space) Fourier space v (angle) x (space) blocker spectrum

102. Diffuse soft shadows (3) Transport after occlusion: –Spatial frequencies moved to angular dimension Diffuse reflector: –Angular frequencies are cancelled

103. Diffuse soft shadows (3) Transport after occlusion: –Spatial frequencies moved to angular dimension Diffuse reflector: –Angular frequencies are cancelled

104. Frequency Analysis: summary Framework for frequency analysis of light transport Gives high frequencies locations Occlusion creates high frequencies Transport + reflection removes HFs High frequencies mostly if: –Small transport (quasi-contact) –Specular reflection (or highly glossy) –Point light source

105. Overview Motivations Timeline Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

106. GPU-based algorithms: credits Specular reflections: D. Roger Real-time soft shadows: L. Atty, M. Lapierre, C. Hansen, J.-M. Hasenfratz, F. Sillion Contact shadows: M. Malmer, F. Malmer, U. Assarsson Global Illumination: J. Kontkannen, E. Turquin, F. Sillion

107. GPU-based algorithms GPUs are highly powerful machines –Widely available –Used in the industry Video games, cinema But they also have their limitations –Memory accesses, per-pixel computations… They work better on local effects –Direct lighting, shadow boundaries… 2003: AS CNRS Real-Time Rendering

108. Looking back… Radiosity: –Lots of work for direct lighting+directly visible High frequency effects in illumination: –Small transport –Shadow boundaries –Specular reflections –…mostly local or semi-local effects Involve a small subset of the scene for each pixel

109. Think globally Our hard points: –Direct lighting+directly visible –Local or semi-local effects … all correspond to the forte of GPUs Main idea: –use GPUs for HF effects+directly visible –Something else (CPU) for low frequency effects

110. GPU-based algorithms Using GPUs for local/semi-local effects: –Specular reflections –Shadows (soft and hard) –Contact shadows Global illumination in static scenes: –GPU for direct lighting, CPU for indirect

111. Specular reflections Vertex based reflections For each vertex in the scene –Compute its reflected position –Find light paths of extremal length –Using gradient descent [Roger and Holzschuch, EG 2006]

112. Specular reflections Our methodEnv. mapping Reference [Roger and Holzschuch, EG 2006]

113. Real-time soft shadows Shadows: important issue for realism Real-time hard shadows: easy with GPU Soft shadows: still an open problem EG 2003: State-of-the-Art Report –Hasenfratz, Lapierre, Holzschuch, Sillion

114. Soft Shadow Maps Discretize occluders using shadow map Compute soft shadow for each surfel –Easy because surfel axis-aligned with LS Add the contributions of each surfel [Atty, Holzschuch, Lapierre, Hasenfratz, Hansen, Sillion, CGF 2006]

115. Contact shadows Shadows cast on nearby objects: [Malmer, Malmer, Assarsson, Holzschuch, JGT 2007]

116. Contact shadows Equivalent to ambient occlusion –Important for realism –Low-cost effect, but with high impact Low frequency effect Several techniques for storing AO field –Moving with the object –Casting shadows on close objects –Indexed by direction: k*n 2 storage –Post-processing to extract AO values

117. Contact shadows Our solution:

118. Contact shadows Store raw AO values, un-processed 3D grid: n 3 storage No post-processing required Why is it interesting? –Low frequency effect: small values of n –Raw values stored: no constant –Similar memory costs, faster rendering

119. Global Illumination in static scenes Direct lighting using GPU Indirect lighting: –Wavelet representation –precomputed Global Transport Operator [Kontkanen, Turquin, Holzschuch, Sillion, EGSR 2006]

120. Global Illumination in static scenes Wavelet basis for radiance Local Transport Operator –expressing a single light bounce –precomputed Global Transport Operator –Built from LTO, using Neumann series For each frame: –Compute direct lighting, projected on WT basis –Apply GTO to direct lighting –Result is indirect lighting

121. Global Illumination in static scenes DirectIndirect Global

122. GPU-based algorithms: summary GPU techniques for local/semi-local effects: –Specular reflections –Soft shadows –Contact shadows Interactive global illumination –GPU for direct lighting, CPU for indirect –Limited to static scenes

123. Overview Motivations Previous work Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

124. Photorealistic Image Synthesis Contributions: –Wavelet radiosity –Frequency analysis –GPU-based algorithms Several applications and contracts: –VSP-Tech, SIMULGEN, Eden Games…

125. Future work: Real Time Global Illumination Frequency analysis in real-time –Used to guide computations GPU for local and semi-local HF effects –Direct lighting, shadows –Specular/glossy reflections –Occlusion and reflection fields CPU for other effects –Indirect lighting –Low Frequency effects Lots of applications and contacts: –Video games (GENAC II) –Movies

126. Future work: Image-Based Rendering We want more realistic pictures Requires more realistic models –Hard to create from scratch –Image-Based Modeling for acquisition Opens several new directions of research: –Easy model acquisition –Real-time rendering of the model –Re-lighting of the model Need for cooperation: –2 ANR projects submitted this year Lots of potential applications

127. Thank you!

128. Acknowledgements Everyone I worked with In addition to those already mentioned: –iMAGIS, 1993-1996: A. Lamouret, F. Faure, JC Lombardo, JD Gascuel, N. Tsingos, M. Desbrun, S. Rivière, R. Orti, JC Vedel, A. Verroust, MP Cani, A. Opalach,… –UCT, 1996-1997: E. Blake, S. Nirenstein, D. Cook, A. Secchia,… –ISA, 1997-2000: S. Lazard, S. Merzouk, K. Tombre, H. Everett, MO Berger, A. Tabbone… –iMAGIS, 2000-2003: J. Thollot, X. Decoret, J. Turbet, F. Neyret, L. Reveret, G. Debunne, S. Grabli… –ARTIS, 2004-2007: A. Martinet, L. Boissieux, V. Ostromoukhov, E. Eisemann, P. Barla, L. Baboud, H. Bezerra, A. Bousseau, A. Orzan, T. Stein, K. Smith, F. Moulin, D. Vanderhaeghe…