Simulation Photoréaliste de l'Éclairage en Synthèse d'Images Nicolas Holzschuch Habilitation à Diriger des Recherches Université Joseph Fourier Grenoble I

Overview Motivations Previous work Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

Overview Motivations Previous work Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

Photorealistic Image Synthesis Starting from the 3D model of a scene… © C. Soler, Model by L. Boissieux.

Photorealistic Image Synthesis …we get a picture of this virtual world © C. Soler, Model by L. Boissieux.

Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math)

Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) Model from DaimlerChrysler / RealReflect IST project / picture by C. Soler

Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) © P. Müller

Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) © Valve, 2006

Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) © Lucas Digital Ltd., 2002

Motivations Why are we doing this? –Many applications: Virtual prototyping Cultural heritage Video games Special effects –Good looking pictures –Multi-disciplinar activity (CS, physics, math) Why is it difficult?

Motivations Why are we doing this? Why is it difficult? –Interactions between light and matter Complex local effects (reflections) –Self-dependent –Long distance interactions –Differences of scale/frequency –Time/quality compromise –Blurry effects are costlier –Too many computations?

Overview Motivations Timeline Selected contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

Timeline

Timeline: positions

Timeline: research themes

Timeline: projects and contracts

Overview Motivations Timeline Selected contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

Overview Motivations Timeline Selected contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

Wavelet Radiosity: credits : iMAGIS –Under the supervision of C. Puech, F. Sillion, G.Drettakis : ISA/LORIA –Working with J.-C. Paul, L. Alonso, F. Cuny, X. Cavin, C. Winkler, H. Barthélémy, S. Petitjean… : iMAGIS/GRAVIR –Working with L. Alonso (at ISA), C. Damez, F. Sillion…

Wavelet radiosity Extension of Hierarchical Radiosity –[Gortler et al., 1993] Uses wavelets for basis functions –Haar wavelets (pw. constant) equivalent to HR M 2, M 3 wavelets : linear, quadratic functions Higher order wavelets should work better –Better representation for smooth functions Experimental study: no, they dont! –[Wilmott & Heckbert, 1997]

Wavelet radiosity Everyone stopped working on WR –…except us at ISA/LORIA Point is: it should really work better –So why not? Experimental studies Everyone had used a na ï ve approach: –Just changing the basis function –But higher order wavelets are costlier n 2 for each patch n 4 for each interaction

Wavelet radiosity Na ï ve implementation doesn t work Must change the approach Adapt the algorithm: –Meshing strategy –No link storage + shooting –Refinement oracle –Visibility tests

Why didnt it work?

Wavelets need quad/triangular patches

Why didnt it work? Patches are then refined The initial triangulation binds the algorithm And each patch is n 2 times costlier

Extended mesh Keep the original mesh Adapt the radiosity algorithm [Holzschuch, Cuny, Alonso, EGWR 2000]

Extended mesh Separation between function and geometry

Link storage and consequences Biggest issue is memory cost –n 4 for each interaction –Solution: dont store them! –[ Stamminger, Schirmacher, Slusallek & Seidel, EG 1998] But think further: –Gathering/shooting –Gathering works better with HR –…except if you dont store the interactions We should use shooting! –…then it makes more sense not to store links!

Higher-Order Wavelets Other changes (refinement oracle, visibility) This time it works: –Smaller memory costs –Faster computations [Cuny, Holzschuch, Alonso, EG 2000]

Memory costs Observed memory costs for the different wavelet bases (rendering time follows similar curve) Consistent with [Willmott & Heckbert 1997]!

Higher order wavelets II Efficient global illumination algorithm –Uses higher-order wavelets –Robust, works on real-world scenes –Memory efficient So, is everything perfect? –Actually, no. –Shadow boundaries are a big issue

Shadow boundaries Discontinuity: the oracle keeps refining Each patch is n 2 times more costly –We lose all the ground we gained elsewhere

Discontinuity meshing Well-known method: –Shadow boundary position is known –Cut the mesh according to these discontinuities –Refine the resulting mesh Impossible with wavelet radiosity: –Wavelets only defined over quads/triangles –…but we already have a workaround!

DM and Higher-Order Wavelets [Holzschuch and Alonso, EGSR 2004] Discontinuities are introduced only if necessary –Use regular subdivision as much as possible Modified refinement oracle: –If refinement could have been avoided using DM –Then Subdivide patch using discontinuity –Embed the new patches inside quads –Regular subdivision over the new quads

DM and Higher-Order Wavelets M2 wavelets, point light source, no DM [Holzschuch and Alonso, EGSR 2004]

DM and Higher-Order Wavelets M2 wavelets, point light source, with DM [Holzschuch and Alonso, EGSR 2004]

DM and Higher-Order Wavelets M3 wavelets, area light source, with DM [Holzschuch and Alonso, EGSR 2004]

Wavelet Radiosity: summary Compact representation of illumination Applications: –Cooperation with Nancy School of Architecture –We tested our algorithms on real-world scenes –Great for robustness Research: –Real scientific problems came up –We had to solve them, resulting in nice papers Transfer: –The software was transferred to a start-up –VSP-Technology, headed by F. Cuny

Wavelet radiosity: looking back Shadow boundaries is an interesting issue Without DM: –90 % of computation time for shadows With DM: –90 % of implementation time for shadows Only important for (direct lighting+directly visible) Indirect lighting can use low quality version Waste of time?

Overview Motivations Timeline Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

Frequency Analysis: credits Joint research work with MIT/CSAIL At MIT/CSAIL: F. Durand, E. Chan At ARTIS/GRAVIR: C. Soler, F. Sillion Financed by an INRIA Équipe Associée [Durand, Holzschuch, Soler, Chan, Sillion, SIGGRAPH 2005]

Illumination effects Blurry reflections: From [Ramamoorthi and Hanrahan 2001]

Illumination effects Shadow boundaries: © U. Assarsson Point light source Area light source

Illumination effects Indirect lighting is usually blurry: Complete lighting © C. Soler 2005.

Illumination effects Indirect lighting is usually blurry: Indirect lighting only Direct lighting only © C. Soler 2005.

Frequency aspects of light transport Blurriness = frequency content –Sharp variations: high frequency –Smooth variations: low frequency All effects are expressed as frequency content: –Diffuse shading: low frequency –Shadows: introduce high frequencies –Indirect lighting: tends to be low frequency

Problem statement How does light interaction in a scene explain the frequency content? Theoretical framework: –Understand the frequency spectrum of the radiance function –From equations of light transport

Unified framework: Spatial frequency (e.g. shadows, textures) Angular frequency (e.g. blurry highlight)

Disclaimer: not Fourier optics We do not consider wave optics, interference, diffraction Only geometrical optics From [Hecht]

Disclaimer: not Fourier optics We do not consider wave optics, interference, diffraction Only geometrical optics From [Hecht]

Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

Previous work Vast body of literature: –Light field sampling –Perceptually-based rendering –Wavelets for Computer Graphics –Irradiance caching –Photon mapping –… We focus on frequency analysis in graphics: –Light field sampling –Reflection as a convolution

Light field sampling [Chai et al. 00, Isaksen et al. 00, Stewart et al. 03] –Light field spectrum as a function of object distance –No BRDF, occlusion ignored From [Chai et al. 2000]

Signal processing for reflection [Ramamoorthi & Hanrahan 01, Basri & Jacobs 03] Reflection on a curved surface is a convolution Direction only From [Ramamoorthi and Hanrahan 2001]

Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc.

Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal

Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Transport

Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Signal 1 Transport

Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Transport Occlusion Signal 2

Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Transport Occlusion Signal 3 Transport

Our approach Light sources are input signal Interactions are filters/transforms –Transport –Visibility –BRDF –Etc. Light source signal Transport Occlusion Signal 4 Transport BRDF

Local light field 4D light field, around a central ray Central ray

Local light field 4D light field, around a central ray We study its spectrum during transport

Local light field 4D light field, around a central ray We study its spectrum during transport

Local light field 4D light field, around a central ray We study its spectrum during transport

Local light field We give explanations in 2D –Local light field is therefore 2D See SIGGRAPH paper for extension to 3D

Local light field parameterization Space and angle space angle Central ray

Local light field representation Density plot: Area light source Space Angle

Local light field Fourier spectrum We are interested in the Fourier spectrum of the local light field Also represented as a density plot

Local light field Fourier spectrum Spatial frequency Angular frequency Spectrum of an area light source:

Basics of Fourier analysis Spectrum corresponds to blurriness: –Sharpest feature has size 1/F max Convolution theorem: –Multiplication convolution Classical spectra: –Box sinc –Dirac constant

Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

Example scene Blockers Light source Receiver

Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

Transport in free space Shear Space Angle

Transport in free space Shear Space Angle Shear Spatial frequency Angular freq. Spatial frequency Angular freq.

Transport Shear This is consistent with light field spectra [Chai et al. 00, Isaksen et al. 00] From [Chai et al. 2000]

Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

Occlusion: multiplication Occlusion is a multiplication in ray space –Convolution in Fourier space Creates new spatial frequency content –Related to the spectrum of the blockers

Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

BRDF integration Outgoing light: –Integration of incoming light times BRDF Outgoing light Incoming light BRDF

BRDF integration Ray-space: convolution –Outgoing light: convolution of incoming light and BRDF –For rotationally-invariant BRDFs Fourier domain: multiplication –Outgoing spectrum: multiplication of incoming spectrum and BRDF spectrum

BRDF in Fourier: multiplication = BRDF is bandwidth-limiting in angle

Example: diffuse BRDF Convolve by constant: –multiply by Dirac –Only spatial frequencies remain =

Overview Previous work Our approach: –Local light field –Transformations on local light field Transport Occlusion BRDF Curvature Case studies Conclusions and future directions

Curved receiver Reduce to the case of a planar surface: –Unroll the curved receiver Equivalent to changing angular content: –Linear effect on angular dimension –No effect on spatial dimension Shear in the angular dimension

Transforms: summary Radiance/FourierEffect TransportShear OcclusionMultiplication/ConvolutionAdds spatial frequencies BRDFConvolution/MultiplicationRemoves angular frequencies CurvatureShear

More effects in paper Cosine term (multiplication/convolution) Fresnel term (multiplication/convolution) Texture mapping (multiplication/convolution) Central incidence angle (scaling) Separable BRDF Spatially varying BRDF (semi-convolution) …and extension to 3D

Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

Overview Previous work Our approach: –Local light field –Transformations on local light field Case studies: –Diffuse soft shadows –Adaptive shading sampling Conclusions and future directions

Case study: diffuse soft shadows Decreasing blockers size: –First high-frequencies increase –Then only low frequency –Non-monotonic behavior

Diffuse soft shadows (2) Occlusion : convolution in Fourier –creates high frequencies –Blockers scaled down spectrum scaled up Fourier space v (angle) x (space) Fourier space v (angle) x (space) blocker spectrum

Diffuse soft shadows (3) Transport after occlusion: –Spatial frequencies moved to angular dimension Diffuse reflector: –Angular frequencies are cancelled

Diffuse soft shadows (3) Transport after occlusion: –Spatial frequencies moved to angular dimension Diffuse reflector: –Angular frequencies are cancelled

Frequency Analysis: summary Framework for frequency analysis of light transport Gives high frequencies locations Occlusion creates high frequencies Transport + reflection removes HFs High frequencies mostly if: –Small transport (quasi-contact) –Specular reflection (or highly glossy) –Point light source

Overview Motivations Timeline Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

GPU-based algorithms: credits Specular reflections: D. Roger Real-time soft shadows: L. Atty, M. Lapierre, C. Hansen, J.-M. Hasenfratz, F. Sillion Contact shadows: M. Malmer, F. Malmer, U. Assarsson Global Illumination: J. Kontkannen, E. Turquin, F. Sillion

GPU-based algorithms GPUs are highly powerful machines –Widely available –Used in the industry Video games, cinema But they also have their limitations –Memory accesses, per-pixel computations… They work better on local effects –Direct lighting, shadow boundaries… 2003: AS CNRS Real-Time Rendering

Looking back… Radiosity: –Lots of work for direct lighting+directly visible High frequency effects in illumination: –Small transport –Shadow boundaries –Specular reflections –…mostly local or semi-local effects Involve a small subset of the scene for each pixel

Think globally Our hard points: –Direct lighting+directly visible –Local or semi-local effects … all correspond to the forte of GPUs Main idea: –use GPUs for HF effects+directly visible –Something else (CPU) for low frequency effects

GPU-based algorithms Using GPUs for local/semi-local effects: –Specular reflections –Shadows (soft and hard) –Contact shadows Global illumination in static scenes: –GPU for direct lighting, CPU for indirect

Specular reflections Vertex based reflections For each vertex in the scene –Compute its reflected position –Find light paths of extremal length –Using gradient descent [Roger and Holzschuch, EG 2006]

Specular reflections Our methodEnv. mapping Reference [Roger and Holzschuch, EG 2006]

Real-time soft shadows Shadows: important issue for realism Real-time hard shadows: easy with GPU Soft shadows: still an open problem EG 2003: State-of-the-Art Report –Hasenfratz, Lapierre, Holzschuch, Sillion

Soft Shadow Maps Discretize occluders using shadow map Compute soft shadow for each surfel –Easy because surfel axis-aligned with LS Add the contributions of each surfel [Atty, Holzschuch, Lapierre, Hasenfratz, Hansen, Sillion, CGF 2006]

Contact shadows Shadows cast on nearby objects: [Malmer, Malmer, Assarsson, Holzschuch, JGT 2007]

Contact shadows Equivalent to ambient occlusion –Important for realism –Low-cost effect, but with high impact Low frequency effect Several techniques for storing AO field –Moving with the object –Casting shadows on close objects –Indexed by direction: k*n 2 storage –Post-processing to extract AO values

Contact shadows Our solution:

Contact shadows Store raw AO values, un-processed 3D grid: n 3 storage No post-processing required Why is it interesting? –Low frequency effect: small values of n –Raw values stored: no constant –Similar memory costs, faster rendering

Global Illumination in static scenes Direct lighting using GPU Indirect lighting: –Wavelet representation –precomputed Global Transport Operator [Kontkanen, Turquin, Holzschuch, Sillion, EGSR 2006]

Global Illumination in static scenes Wavelet basis for radiance Local Transport Operator –expressing a single light bounce –precomputed Global Transport Operator –Built from LTO, using Neumann series For each frame: –Compute direct lighting, projected on WT basis –Apply GTO to direct lighting –Result is indirect lighting

Global Illumination in static scenes DirectIndirect Global

GPU-based algorithms: summary GPU techniques for local/semi-local effects: –Specular reflections –Soft shadows –Contact shadows Interactive global illumination –GPU for direct lighting, CPU for indirect –Limited to static scenes

Overview Motivations Previous work Contributions : –Wavelet Radiosity –Frequency Analysis –GPU-based algorithms Conclusion & future work

Photorealistic Image Synthesis Contributions: –Wavelet radiosity –Frequency analysis –GPU-based algorithms Several applications and contracts: –VSP-Tech, SIMULGEN, Eden Games…

Future work: Real Time Global Illumination Frequency analysis in real-time –Used to guide computations GPU for local and semi-local HF effects –Direct lighting, shadows –Specular/glossy reflections –Occlusion and reflection fields CPU for other effects –Indirect lighting –Low Frequency effects Lots of applications and contacts: –Video games (GENAC II) –Movies

Future work: Image-Based Rendering We want more realistic pictures Requires more realistic models –Hard to create from scratch –Image-Based Modeling for acquisition Opens several new directions of research: –Easy model acquisition –Real-time rendering of the model –Re-lighting of the model Need for cooperation: –2 ANR projects submitted this year Lots of potential applications

Thank you!

Acknowledgements Everyone I worked with In addition to those already mentioned: –iMAGIS, : A. Lamouret, F. Faure, JC Lombardo, JD Gascuel, N. Tsingos, M. Desbrun, S. Rivière, R. Orti, JC Vedel, A. Verroust, MP Cani, A. Opalach,… –UCT, : E. Blake, S. Nirenstein, D. Cook, A. Secchia,… –ISA, : S. Lazard, S. Merzouk, K. Tombre, H. Everett, MO Berger, A. Tabbone… –iMAGIS, : J. Thollot, X. Decoret, J. Turbet, F. Neyret, L. Reveret, G. Debunne, S. Grabli… –ARTIS, : A. Martinet, L. Boissieux, V. Ostromoukhov, E. Eisemann, P. Barla, L. Baboud, H. Bezerra, A. Bousseau, A. Orzan, T. Stein, K. Smith, F. Moulin, D. Vanderhaeghe…