1. Local, Deformable Precomputed Radiance Transfer
Peter-Pike Sloan, Ben Luna
Microsoft Corporation
John Snyder
Microsoft Research

2. “Local” Global Illumination
Renders GI effects on
local details
Rotates transfer model
This paper is focused on “local” global illumination. Here we see a deformed object with square bumps.
[click]
The rendering technique in this paper allows global illumination effects on local details.
Like the shadows on the bumps shown here.
To do this, at every point on the object we store a spherical function representing transfer -- how the bumps shadow and cast light onto themselves.
So at every point on the object, we must rotate transfer from the original texture frame where we precomputed it…
…into the frame of the current, deformed object. Doing this efficiently is our main challenge.
Our technique neglects effects from “coarse scale” features, like the shadow the object should be casting on itself in the highlighted region.
Neglects gross shadowing

3. “Local” Global Illumination
To understand the nature of this local approximation, let’s look at this head model example.
[click]
If we deform this model, and compute a global illumination solution off-line we get an accurate image like this.
However if we just rotate the transfer information from the rest pose, we get this, very plausible, image. Some regions (the brow, under the nose) are casting different shadows, but much of the shading comes from nearby features which are handled accurately.
Original
Ray Traced
Rotated

4. Bat Demo

5. Precomputed Radiance Transfer (PRT)
Vector
illuminate
response
The idea behind precomputed radiance transfer or PRT is to precompute the response of an object, like this diffuse Max Planck bust, to lighting.
The images on the left show the lighting basis functions, and those on the right show the object’s corresponding response.
[click]
At each point on the object, we concatenate these response coefficients to build a transfer vector.
By the linearity of light transport, an object’s response to a linear combination of the lighting basis functions on the left is the same linear combination of the transfer responses on the right.
So exit radiance at a point is obtained by dotting with a lighting vector, which expresses the current lighting environment in this lighting basis, with the transfer vector.

6. Related Work: Area Lighting
[Ramamoorthi2001]
[Sloan2003]
[Muller2004]
[Kautz2004]
[Ng2003]
[Sloan2002]
Our technique deals with a general, spherical model of illumination. There is a lot of related work and I will briefly touch on it.
A family of papers use pre-filtered environment maps to render objects in complex lighting conditions. However these techniques do not model the transport in the scene in any way. Ramamoorthi’s siggraph 2001 paper is an example of this for diffuse surfaces.
[click]
PRT, as just described, enables objects to be illuminated in low frequency lighting environments, and includes complex light transport effects like soft shadows, inter-reflections, caustics and subsurface scattering.
Ng et al used basis functions (haar wavelets) that can model illumination at all frequencies, not just low frequencies, for diffuse objects.
This work was extended to handle glossy surfaces last year independently by two sets of authors. The glossy surface rendering method of these papers is the most practical to date, and is also used in this paper.
Kautz and colleagues had a paper last year that computed direct lighting for fully dynamic objects using hemispherical rasterization of shadowing geometry. They handle more general deformations, but only scenes of limited complexity.
Doug James had a related paper in siggraph 2003, where he precomputed both the deformation and lighting response of an object to a limited set of impulse forces. Our goal is to handle arbitrary deformations which aren’t precomputed or known in advance.
In siggraph 2003, we combined PRT at coarse scales, with ideas from bidirectional texture functions to model fine scale global illumination, but just for rigid objects.
Finally, you already saw a paper by Zhou et al that handles gross shadowing effects from dynamic, articulated models.
[James2003]
[Zhou2005]
[Liu2004;Wang2004]

7. Other Related Work Directional Lighting Ambient Occlusion
[Malzbender2001],[Ashikhmin2002]
[Heidrich2000]
[Max1988],[Dana1999]
Ambient Occlusion
[Miller1994],[Phar2004]
[Kontkanen2005],[Bunnel2005]
Environmental Lighting
[McCallister2002]
Other related work uses directional, rather than low-frequency or area, lighting models. For example polynomial texture maps and steerable illumination textures.
[click]
Heidrich in siggraph 2000 precomputed visibility allowing scene relighting, taking into account inter-reflections from the object onto itself. Curvature change could even be handled which we neglect.
Horizon mapping and bidirectional texture functions model “local” global illumination effects as well, but are focused on directional lighting models.
Accessibility shading and ambient occlusion model a scalar representing how much lighting can reach a point on the surface, but don’t take the light direction into account. So they can’t really cast recognizable shadows.
Two concurrent papers model ambient occlusion for dynamic objects.
McCallister 2002 approximated a multi-lobed Lafortune BRDF that varies spatially over a surface. Our transfer models are more general than spatially-varying BRDFs and include effects like shadows cast from one local feature onto another.

8. Spherical Harmonics (SH)
Spherical Analog to the Fourier basis
Used extensively in graphics
[Kajiya84;Cabral87;Sillion91;Westin92;Stam95]
Polynomials in R3 restricted to sphere
Our work focuses on low frequency lighting, which we represent using the spherical harmonic basis.
They are the spherical analog to the fourier basis on the unit circle, and have been used extensively in graphics.
They can be represented as polynomials in R3 restricted to the unit sphere.
A function f can be projected into this basis, generating a vector of coefficients , and reconstructed by scaling the basis functions, y_lm by the corresponding coefficients.
This can be represented in vector form as a dot product.
projection
reconstruction

9. Spherical Harmonics (SH)
Spherical Analog to the Fourier basis
Used extensively in graphics
[Kajiya84;Cabral87;Sillion91;Westin92;Stam95]
Polynomials in R3 restricted to sphere
projection
reconstruction

10. Low Frequency Lighting
order 1
order 2
order 4
We are focused on distant, environmental lighting.
We’re projecting a lighting environment into progressively higher frequencies. Most of the results in this talk use order 6 lighting which is clearly very blurry.
The main effect of restricting the lighting frequency is to limit the sharpness of shadows we can obtain on shaded surfaces.
But there are many techniques for obtaining sharp shadows from point or directional light sources in real-time graphics. But soft shadows are more difficult.
order 8
order 16
order 32
original

11. SH Rotational Invariance
rotate SH SH
One advantage of using the SH lighting basis is its rotational invariance property.
If you take a lighting environment and project it into low-frequency SH, you get a low frequency approximation. Rotating, which can be done in closed form with SH, we get this result.
This is the same result you get if you rotate the original, high frequency lighting environment and then project it using SH; in other words, the diagram commutes.
This means that lighting is stable under rotations, so you won’t get the temporal aliasing artifacts that you can get with other basis functions.
rotate

12. Spherical Harmonics (SH)
nth order, n2 coefficients
Evaluation O(n2)
This slide visualizes the SH basis functions by scaling by the absolute value of the function and coloring based on its sign – red is positive and blue is negative.
This first basis function is just a constant, this is the average or “DC” term.
The next band are the linear functions – y followed by z followed by x.
The next band are the quadratic functions – each successive band is a higher degree polynomial and can represent more oscillations.
Each band has 2 more basis functions then the one before it.
Nth order spherical harmonics have a total of n squared coefficients. And evaluation is order n squared.

13. Zonal Harmonics (ZH) Polynomials in Z Circular Symmetry
We’re going to make use of a subset of the spherical harmonics called “zonal harmonics”, shown here in the center column.
Zonal harmonics have only 1 basis function per band, and represent polynomials in Z.
Any function that has circular symmetry around the Z axis when projected into SH only has non-zero coefficients corresponding to the zonal harmonics.

14. SH Rotation Structure O(n3) Too Slow!
The structure of a spherical harmonic rotation matrix is fairly straightforward. Each band is rotationally independent, so the matrix is block diagonal.
Here we label the SH vector using the polynomial forms of its basis functions.
The DC term is unaffected by rotation, there is a 3x3 matrix for the linear basis functions, and a 5x5 matrix for the quadratic functions.
Because of its block diagonal structure, SH rotation is order n^3, where n is the SH order.
For orders beyond about 3, it is too slow to be densely evaluated, even on graphics hardware.

15. ZH Rotation Structure O(n2)
Zonal harmonic rotation is simpler than SH rotation, which is why we will make use of it to represent transfer. There is only a single non-zero coefficient per band, so only a single column of the rotation matrix is applied.
Rotating ZH is thus order n^2, and we will later show how it can be accelerated even further using light specialization.

16. What’s that column? z z’ Rotate delta function  so that z → z’ :
Evaluate delta function at z = (0,0,1)
Rotating scales column C by dl
Equals y(z’) due to rotation invariance z’
What is the form of this ZH rotation column?
To derive it, consider rotating a delta function, circularly symmetric around Z, so that the Z axis rotates to a new axis Z’.
The delta functions are here visualized bandlimited to order 6. This is the “peakiest” signal we can get woth order 6 SH.
Integrating against the delta function simply evaluates the ZH basis functions at z, generating the constants d_l.
From the previous slide, rotating this delta function scales the central column C of each band’s rotation matrix by d_l.
But due to the rotational invariance of SH, this scaled column must equal projecting a delta function around the rotated axis Z’, which evaluates the SH basis functions at z’.
[click]
This tells us what the central column has to be.

17. What’s that column? z z’ Rotate delta function  so that z → z’ :
Evaluate delta function at z = (0,0,1)
Rotating scales column C by dl
Equals y(z’) due to rotation invariance z’

18. Efficient ZH Rotation z g(s)
Now we can apply the rotation we just derived to an arbitrary circularly symmetric function about z, rather than a delta function.
We begin with a spherical function g(s) circularly symmetric about z.
[click]
We can find it’s zonal harmonic coefficients g_l using this integral, producing the numerical vector shown for this example.
Now we rotate this function so that z goes to z’, resulting in function g’(s).
To find the SH coefficients of this rotated function,
we evaluate the SH basis functions at z’ and scale the result by the diagonal matrix G*.
[cilck]
Each component of the diagonal g*_l is just a scaled version of the original ZH coefficient g_l.

19. Efficient ZH Rotation z
g(s)

20. Efficient ZH Rotation z z’
g’(s)
g(s)

21. Efficient ZH Rotation z z’
g’(s)
g(s)

22. Efficient ZH Rotation z z’
g’(s)
g(s)

23. Transfer Approx. Using ZH
Approximate transfer vector t by sum of N “lobes” +
We now apply zonal harmonics to approximate a transfer vector t so that it can be rotated quickly.
Our transfer approximation, t tilde, is just a sum of N rotated ZH lobes, indexed by i.
Each lobe i is circularly symmetric about its own axis s*_i; its coefficients G*_i determine its shape around that axis.
A 4 lobe example illustrates how the approximation works.
[click]
Now to rotate this representation given a 3x3 rotation matrix R, all we have to do is rotate the lobe axes s*_i before evaluating the basis functions y.
The lobe coefficients don’t change under rotations.
As long as N is small, this is much more efficient than SH rotation, about an order of magnitude more efficient for order 6 lighting and N=4.
To find the lobe axes and coefficients, we minimize squared error integrated over the sphere, which reduces
to the squared length of the vector difference.
This is a non-linear least squares problem.

24. Transfer Approx. Using ZH
Approximate transfer vector t by sum of N “lobes”

25. Transfer Approx. Using ZH
Approximate transfer vector t by sum of N “lobes”
Minimize squared error over the sphere

26. Single Lobe Solution For known direction s*, closed form solution
“Optimal linear” direction is often good
Reproduces linear, formed by gradient of linear terms
Well behaved under interpolation
Cosine weighted direction of maximal visibility in AO
To fit a single lobe, the coefficients can be computed in closed form if the lobe direction is s* is known.
But in general, we don’t know what the best direction should be.
The optimal linear direction is often a good choice.
It is the unique direction that exactly reproduces the linear part of the signal, and is given by the gradient of the linear terms.
It is well behaved under interpolation,
and this direction for transfer vectors that represent direct shadowing only can be viewed as the cosine-weighted direction of maximal visibility in ambient occlusion.

27. Multiple Lobes
Here we see a complex lobe that clearly can’t be approximated by a single circularly symmetric function.
This is one of the lobes used to approximate this shape. The top row shows the individual lobes, and the bottom row shows the running sum. 4 lobes does a good job approximating this particular function.

28. Random vs. PRT Signals
To understand the number of lobes required to fit a spherical function, we generated 16,000 uniform random 36D points, then ran our fitting code over the samples with different number of lobes. We did not encode DC/linear in these examples.
Squared error is plotted on a log scale in this figure, and we are just plotting the worst and average error over the random samples.
[click]
These random signals are more difficult to fit compared to actual PRT signals such as this example.
4 lobes provides good enough error for rendering, and we use 4 lobes for many of our other examples.

29. Energy Distribution of Transfer Signals
The spectral distribution of squared energy in our datasets decreases rapidly at higher frequencies.
This has two benefits for us: it allows us to get away with truncating the transfer vector at order 6, and it also makes the transfer signals easier to fit with zonal harmonics.

30. Energy Distribution and Subsurface Scatter
Inter-reflections and subsurface scattering tend to shift more energy into the lower frequency bands.
Here we see graphs of a diffuse object (in blue) and two different subsurface scattering simulations, the red one being more translucent.
As the translucency increases, more of the energy gets pushed into the lower frequencies.

31. Rendering Rotate lobe axis, reconstruct transfer and dot with lighting
Care must be taken when interpolating
Non-linear parameters
Lobe correspondence with multiple-lobes
Rendering LDPRT is straight forward.
Rotating the lobe axis is done using a conventional 3x3 rotation matrix.
We then reconstruct the transfer vector by summing over lobes i, bands l, and basis functions in a band m.
Dotting the reconstructed transfer vector with the light vector produces the shaded result.
Care must be taken when interpolating lobe parameters.
The shaded result does not depend linearly on the lobe axis, and only lobes that correspond should be interpolated in a multi-lobe model.

32. Light Specialized Rendering
A simple optimization can significantly improve the rendering performance.

33. Light Specialized Rendering
If you distribute the dot product into the summation,

34. Light Specialized Rendering
You can then pull the zonal harmonic coefficients out of the inner summation.
The boxed expression simply computes a dot product in each band l between the SH basis functions evaluated at s*_i with the lighting environment coefficients.

35. Light Specialized Rendering
The previous boxed expression can now be precomputed and stored in a texture map specialized to the lighting.
We begin with the original lighting environment, for example this grove environment.
[click]
Then we project to its low frequencies using spherical harmonics, visualizing just the red channel.
The light specialized texture map then dots the SH basis vector with the lighting coefficients separately in each band.
We obtain a set of texture maps over spherical directions: one for each band and each color channel of the environment.
Light specialization reduces shader complexity by a factor of the SH order, 6 for most of our examples.

36. Light Specialized Rendering
Quadratic
Cubic
O(N n2) → O(N n)
Quartic
Quintic

37. Generating LDPRT Models
PRT simulation over mesh
texture: specify patch (a)
per-vertex: specify mesh (b)
Parameterized models
ad-hoc using intuitive parameters (c)
fit to simulation data (d)
(a) LDPRT texture
To generate LDPRT data, we use a few different methods.
One way is to simulate PRT over specified geometry like a small patch which can then be mapped onto deformable objects.
We can also simulate PRT over an entire mesh and then deform that mesh, as in the Mayan head example.
We have also built parameterized models.
Ad-hoc models directly construct ZH coefficients from intuitive parameters, such as the bat model you’ve seen already.
We also can fit ZH models to data simulated over the parameterized model, as in this wrinkle model example.
(c) thin-membrane model
(d) wrinkle model
(b) LDPRT mesh

38. LDPRT Texture Pipeline
Start with “tileable” heightmap
Simulate 3x3 grid
Extract and fit LDPRT
Store in texture maps
We briefly summarize the pipeline to generate LDPRT textures.
We start with a tileable heightmap which we place it on a 3x3 grid.
[click]
Offline, we then ray trace this grid to get the PRT solution. Note how neighboring bumps cast shadows onto each other.
[ciick]
Now we extract the transfer from the center tile and solve for the LDPRT approximation.
This result is stored into a texture map and can be applied to deformable scene objects.

39. Thin Membrane Model Single degree of freedom (DOF)
“optical thickness”: light bleed in negative normal direction
The bat example has a single DOF, “optical thickness” which models how much light bleeds through opposite the normal.
This could have been based on subsurface scattering simulations, but instead is based on a simple ad-hoc model described in the paper.
The artist just has to paint this optical thickness over an object, such as this bat.

40. Wrinkle Model Two DOF Phase, position along canonical wrinkle
The wrinkle model is an example where we parameterized the geometry but then fit an LDPRT model to an actual PRT simulation.
Our model has 2 DOF, there is the phase – which is just the position along a canonical wrinkle

41. Wrinkle Model Two DOF Phase, position along canonical wrinkle
Amplitude, max magnitude of wrinkle
And amplitude – which is the maximum height of the wrinkle.

42. Wrinkle Model Fit Compute several simulations Fit 32x32 textures
64 discrete amplitudes
255 unique points in phase
Fit 32x32 textures
One optimization for all DOF simultaneously
Optimized for bi-linear reconstruction
3 lobes
For this data we computed a dense set of simulations over amplitude and phase.
We then fit low resolution textures to this data.
All of the DOF were fit together, optimizing for bi-linear reconstruction over the dense dataset.
It took 5-10 minutes to solve the optimization problem.

43. Glossy LDPRT Use separable BRDF
Encode each “row” of transfer matrix using multiple lobes (3 lobes, 4th order lighting)
See paper for details
Glossy materials can also be rendered using LDPRT.
We use the separable BRDF approximation, and independently encode each row of the resulting transfer matrix using multiple lobes.
We used 3 lobes and 4th order lighting to generate this figure; the paper has more of the details.

44. Demo

45. Conclusions/Future Work
“local” global illumination effects
soft shadows, inter-reflections, translucency
easy-to-rotate rep. for spherical functions
sums of rotated zonal harmonics
allows dynamic geometry, real-time performance
may be useful in other applications [Zhou2005]
future work: non-local effects
articulated characters
We presented a technique, LDPRT, that models “local” GI effects, soft shadows, inter-reflections and translucency, under low frequency lighting.
We did this by using a representation of low-frequency spherical functions that is easy to rotate, using sums of rotated zonal harmonics.
This allows realistic, real-time shading of dynamic, deforming geometry, which we think is much more interesting than the static objects demonstrated in previous PRT work.
Our representation may be useful in other applications – for example the shadow fields paper from the beginning of this session.
In future work, we would like to model non-local transfer effects from low frequency lighting, for example articulated characters rendered with cast soft shadows from moving limbs.

46. Acknowledgements Demos/Art: John Steed, Shanon Drone, Jason Sandlin
Video: David Thiel
Graphics Cards: Matt Radeki
Light Probes: Paul Debevec