Real-Time Relighting Digital Image Synthesis Yung-Yu Chuang 1/10/2008 with slides by Ravi Ramamoorthi, Robin Green and Milos Hasan

Realistic rendering We have talked about photorealistic rendering for complex materials, complex geometry and complex lighting. They are realistic but slow.

Real-time rendering Its goal is to achieve interactive rendering with reasonable quality. It ’ s important in many applications such as games, visualization, computer-aided design, …

Real-Time relighting Lighting is the process of adjusting lights. It is an important but time-consuming step in animation production pipeline. Relighting algorithms for two kinds of lights –Distant environment lights –Near-field lights for production

Relighting algorithms for distant environment lights

Natural illumination People perceive materials more easily under natural illumination than simplified illumination. Images courtesy Ron Dror and Ted Adelson

Natural illumination Rendering with natural illumination is more expensive compared to using simplified illumination directional source natural illumination

Reflection maps Blinn and Newell, 1976

Environment maps Miller and Hoffman, 1984

HDR lighting

Examples of complex environment light

Direct lighting with complex illumination p q

Function approximation G(x): the function to approximate B 1 (x), B 2 (x), … B n (x): basis functions We want Storing a finite number of coefficients c i gives an approximation of G(x)

Function approximation How to find coefficients c i ? –Minimize an error measure What error measure? –L 2 error Coefficients

Basis Functions are pieces of signal that can be used to produce approximations to a function Function approximation

We can then use these coefficients to reconstruct an approximation to the original signal Function approximation

We can then use these coefficients to reconstruct an approximation to the original signal Function approximation

Orthogonal basis functions Orthogonal Basis Functions –These are families of functions with special properties –Intuitively, it’s like functions don’t overlap each other’s footprint A bit like the way a Fourier transform breaks a functions into component sine waves

Integral of product

Basis functions Transform data to a space in which we can capture the essence of the data better Spherical harmonics, similar to Fourier transform in spherical domain, is used in PRT.

Real spherical harmonics A system of signed, orthogonal functions over the sphere Represented in spherical coordinates by the function where l is the band and m is the index within the band

Real spherical harmonics

Reading SH diagrams – + Not this direction This direction

Reading SH diagrams – + Not this direction This direction

The SH functions

Spherical harmonics

m l

SH projection First we define a strict order for SH functions Project a spherical function into a vector of SH coefficients

SH reconstruction To reconstruct the approximation to a function We truncate the infinite series of SH functions to give a low frequency approximation

Examples of reconstruction

An example Take a function comprised of two area light sources –SH project them into 4 bands = 16 coefficients

Low frequency light source We reconstruct the signal –Using only these coefficients to find a low frequency approximation to the original light source

Harr wavelets Scaling functions ( V j ) Wavelet functions ( W j ) The set of scaling functions and wavelet functions forms an orthogonal basis

Harr wavelets

Example for wavelet transform Delta functions, f=(9,7,3,5) in V 2

Wavelet transform V 1, W 1

Example for wavelet transform V 0, W 0, W 1

Example for wavelet transform

Quadratic B–spline scaling and wavelets

2D Harr wavelets

Example for 2D Harr wavelets

Applications 19% 5% L 2 1% 15% L 2 3% 10% L 2

Relighting algorithms for animation production

Relighting for production Lighting is a time-consuming process. Artists adjust lighting parameters and wait for a couple of hours or days to get feedback. Local shading with complex scene and many lights Interactive relighting –Interative visual eedback –Fixed scene and camera –Lower quality –Scalable with sene complexity and number of lights

Deep framebuffer Gershbein and Hanrahan, SIGGRAPH 2000

Deep framebuffer

LPICS Pixar, SIGGRPH A practical realization for the deep framebuffer approach on GPUs LPICS 0.1s Final renderer 2,000s video

Lightspeed ILM, SIGGRAPH 2007 An even more practical system with automatic shader conversion. (2.7s v.s. 57m)

Direct-to-indirect transfer Hasan et. al. SIGGRAPH 2006 Deep framebuffer approaches only support local shading, but not indirect lighting direct lightingWith indirect lighting

Concept Distribute gather samples on scene surfaces

Concept Direct illumination on both gather samples and view samples

Concept Inter-reflections between gather samples

Concept Final gather on view samples

Inter-reflections between gather samples gather sample

Inter-reflections between gather samples Assume all gather samples are diffuse

Inter-reflections between gather samples

Final gathering

Direct on gather Indirect on view Final Transfer matrix Direct on view Concept

Scene: Still Life Precomputation: 1.6 hours 11.4 – 18.7 fps Polygon: 107k

Scene: Temple Precomputation: 2.5 hours8.5 – 25.8 fpsPolygon: 2M

Scene: Hair Ball Precomputation: 2.9 hours9.7 – 24.7 fpsPolygon: 320k

Scene: Sponza Atrium Precomputation: 1.5 hours13.7 – 24.9 fpsPolygon: 66k

Comparison DTI: 8-25 fps (2.5 hr precomputation) Monte Carlo path tracer: 32 hours