1. Spectacular Specular -LEAN and CLEAN specular highlights
Dan Baker
Firaxis Games

2. Motivation Observations from Movie folks
“The most important aspect of rendering for a movie is anti-aliasing”
“Games today still don’t look as good as animated movies 12 years ago.”
“Why in video games does everything look so shiny?”
Things I have heard over the years

3. Reality Check Have a habit of comparing ourselves against other games
Therefore miss the obvious: Some of our materials don’t look anything like they should
And they alias
And they sparkle

4. Shiny things Water, Metal, commonly seen in games
Used for water in Civilization V, but metals suffer from similar problems
Will tour two common lighting models, Phong and Blinn Phong
Both have huge problems

5. Phong Simple to implement
Since L is constant for environments, can turn specular part into a preconvolved environment

6. Sometimes accurate
Good for perfect reflectors,like still water. But, perfect reflectors = high power
Reflective materials have powers > 10000, but that will cause all sorts of problems for a point light

7. Problems with Phong Aliases Can’t get elongated reflections
Pretty inaccurate – very plastic look to it
No good way to add normals maps together
Anisotropic (grooved) materials will lose there anisotropy at zoom
Can’t use high powers or else!

8. Problems with Phong
Only see a reflection on the very flat part of the beach, even with perfect use of Phong, won’t work.
Classic Scene, sunset. Can’t get this with Phong lighting

9. Blinn Phong Much more accurate if we have real lights in our scene
Can get elongated shapes
Cheap to evaluate
Note, this is the normalized version of Blinn-Phong

10. Blinn Phong problems Aliases Shading becomes very wrong with roughness
Highlights change based on pixel coverage, and become
Anisotropic (groved) materials will lose there anisotropy at zoom
Can’t do environments easily
Can’t use high powers or else!

11. Spectacular Specular fail
Down sampled of the
final render
What Blinn-Phong gives us
A very rough wave filters to a perfectly flat standing water,
An example
The effects are profound. If we take the same image and downsaple it, we get a very different thing then if we filter the normal map. Infact, we are being severely over shiny about things.

12. Bump Filtering
Confirm with imperical data.
The left side would look like the right side by most techinques. As we zoom this image out we notice massive differences in shinyness, even though it’s the same material
Exactly the same substance, but one side is wrinkled. This completely
changes the reflections. (Thanks to Chipotle for the foil)

13. Overviews problems
“The shimmies”, “The speckles” Lots and lots of talks about this problem
The more substantial the normal map, the higher the power, the more noise we get. Lots of artist tweaking, limits to our data
Reflections are just plan wrong at distant scale, makes objects way over-shiny
Can’t add normal maps together easily to get detail maps

14. Why this happens
The integral of a function over a range of inputs isn’t the same as function with inputs integrated over a range
Where R is a region of a texture, F is our shader (in this case, a Phong or Blinn Phong shader) . The second version is a discrete version, where W is the sample weights from our hardware filtering.
Left side is what we want. This is the evaluation at every thoeretical texel that contributes to our scene. But, what we end up doing is the right side. As seen, these two are very different.

15. How do Movies solve this?
Typically use REYES, or more advanced techniques
Roughly equivalent of shading every relevant texel and averaging the results
Very expensive, potentially thousands of shader evals per pixel
Movies typically decouple shading frequency and object rasterization. This gets rid of a huge number of issues, though still not completely ‘correct’. Likely will never do this for games, since objects small on screen can be hugely expensive.

16. Dreaming the dream Ideal lighting model: Can use any power we want
Will deal with zooming in and out correctly
Won’t alias
Easy to use: compatible with our current pipeline
Relatively inexpensive
Can add normals together
Can use all of our MIP hardware

17. Formal definition
What we really want is to build a replacement for Blinn Phong that has this property (where F is basically our shader):
We want a BRDF that is the same when we average all the results at the texel level with the results if we average the inputs. We cannot do the first one feasibly, but we can do the second (and it is already done via MIP maps and hadware).

18. LEAN Mapping Linear Efficient Antialiased Normal Mapping
Considered Fast Antialiased Reflectance Texture Mapping
Fast and flexible solution for bump filtering
Shiny bumps won’t alias
Distant bumps will change surface shading
Directional bumps will become anisotropic highlight
Allows blending layers of bumps
Works with existing Blinn-Phong pipeline

19. Scale problem solved
Normal Map
LEAN Map

20. Prior Work Posed by Kajiya 1985 Monte-Carlo
Cabral et al. 1987, Westin et al. 1992, Becker & Max 1993
Multi-lobed distributions
Fournier 1992, Han et al. 2007
Single Gaussian/Beckmann distribution
Olano & North 1997, Schilling 1997, Toksvig 2005
Diffuse [Kilgard 2000]
Some previous attempts, some more worth mentioning

21. Beckman Shading Model
The math is simpler then it looks. We are raising the power based on the distance of the half angle from the normal. This second formulation rolls the power into a covariance matrix, thereby giving us anisotropic power (e.g. two powers, one for X and one for Y).

22. Probability Distributions in Shading
Distribution of microfacet normals
Perfectly reflective facets
Only facets oriented with reflect to
Look up probability of in distribution
Beckmann distribution
Gaussian of facet tangents = projection

23. Filtering Filter is linear combination over kernel
Linear representation → any linear filter
Summed Area, EWA, …
MIP map, Hardware Anisotropic
We need a BRDF that is linear

24. Filtering: Gaussians Gaussian described by mean and variance
Mean combines linearly
Variance does not, but second moment does

25. Blinn-Phong ↔ Beckmann
Blinn-Phong approximates Gaussian [Lyon 1993]
Better fit as increases
Variance , normalize with
Bechmann is very close a normalized Blinn-Phong. We were using normalization factor of s for Blinn-Phong.

26. Blinn-Phong ↔ Beckmann
Blinn Phong and Bechmann are nearly equivalant
Blinn-Phong
Beckmann

27. LEAN mapping Blinn-Phong ↔ Beckmann Filtering Bumps Sub-facet shading
Layers of bumps

28. Distributions & Bumps
If the normal is changing our surface orientation, is there any way to add them together?
Does that even
meaning?

29. LEAN Mapping
Beckman distribution can be broken into pieces that filter, but doesn’t deal with the normals.
Key insight: We think of the normal instead as a shift of the distribution of microfacets

30. Distributions
Beckman distribution works on a 2d plane. The blue discs represent the distribution of normals. Rather then change the orientation of the surface, we simply shift the center location of the distribution of normals by the x,y component of the normals. Thus, we interpret the normal as a shift in distribution, rather then a change in surface orientation
The green arrow represents the distance used by the beckman distrubtion. It is a vector in 2D, and thus the beckman distribution can be anisotropic since the x and y directions can have what is in effect different components.

31. Filtering Bumps Rather than bump-local frame Use surface tangent frame
Bump normal = mean of off-center distribution

32. Surface-frame Beckmann
Bumps vs. Surface Frame
Bump-frame Beckmann
Surface-frame Beckmann

33. LEAN Data Normal (for diffuse) Bump center in tangent frame
Second moments

34. LEAN Use Pre-process Render-time Seed textures with , and
Build MIP chain
Render-time
Look up with HW filtering
Reconstruct 2D covariance
Compute diffuse & specular per light

35. Sub-facet Shading What about base specularity?
Given base Blinn-Phong exponent,
Base Beckmann distribution
One of these at each facet = convolution
Gaussians convolve by adding ’s
Fold into , or add when reconstructing

36. LEAN Map features Seamless replacement for Blinn-Phong
Specular bump antialiasing
Turns directional bumps into anisotropic microfacets

37. Bump Layers Uses Bump motion (ocean waves) Detail texture Decals
Our approach
Conceptually a linear combination of heights
Equivalent to linear combination of
Even from normal maps

38. Bump Layers: The Tricky Part
What about ?
Expands out to , , and terms
terms are in , terms are in
terms are new:
Total of four new cross terms

39. Layering Options Generate single combined LEAN map
Mix actual heights, or use mixing equations
Time varying: need to generate per-frame
Decal or detail: need high-res LEAN map
Generate mixing texture
One per pair of layers
Decal or detail: need high-res LEAN mixture maps
Approximate cross terms
Use rather than a filtered mixing texture

40. Layer Options Source 1 Source 1 Source 2 Source 2 Mixed
Mixture Texture
Approximation
Single LEAN Map
MIP Biased

41. 1600 x 1200, single full screen object
Performance
Single Layer
Two Layers
Blinn-Phong
LEAN
Per-frame
Mix texture
Approx
ATI Radeon HD 5870
1570 FPS
1540 FPS
917 FPS
1450 FPS
1458 FPS
D3D Instructions
30 ALU 1 TEX
42 ALU 2 TEX
50 ALU 3 TEX
54 ALU
5 TEX
4 TEX
1600 x 1200, single full screen object

42. Converting Blinn-Phong Data
So fast could be done at load time
float3 tn = tex2D(normalMap, coord);
float3 N = float3(2*tn.xy-1, tn.z);
float2 B = N.xy/(ScaleFactor&N.z);
float3 M = float3(B.x*B.x + 1/s, B.y*B.y + 1/s, B.x*B.y)
Output.lean1 = float4(tn, .5*M.z + .5)
Output.lean2 = float4(.5*B + .5, M.xy)
S is the power, ScaleFactor is a reranging factor

43. Texture Compression and Precision
Normal maps get big, painful to compress
Lean MAPs require 5 fields
x,y, x^2, xy, y^2
Caveat: The precision matters. Unlike other techniques, we are using the normal filtering hardware

44. Obligatory Shader Code
float4 f4BaseMeshColor = tex2D(BaseMeshColor, f2BaseTexCoord); float4 f4BaseColor = tex2D(LeanTextureMap1, f2BaseTexCoord); float Var = tex2D(LeanTextureMap2,f2BaseTexCoord).x; float GradientScale = g_fLeanMapScale; float VarianceScale = GradientScale*GradientScale; float2 Gradient = float2(f4BaseColor.x*2-1,f4BaseColor.y*2-1) *GradientScale; float3 Covar = float3(f4BaseColor.zw , Var*2 - 1) * VarianceScale; // turn moments into elements of covariance matrix, matrix is mat4(Covar.x,Covar.z,Covar,z,Covar.y) Covar -= float3(Gradient.xy*Gradient.xy, Gradient.x*Gradient.y);
float3 Half = normalize(ViewDir + LightDir);
//Transform half angle back into tangent space
Half = mul(mTS, Half);
float2 HalfCenter = Half.xy/Half.z - Gradient.xy;
//Now calculate the spec
float Cxx = Covar.x + 1/g_fExp, Cyy = Covar.y + 1/g_fExp, Cxy = Covar.z;
float Cdet = Cxx*Cyy - Cxy*Cxy;
float e = (Cyy*HalfCenter.x*HalfCenter.x + (Cyy*HalfCenter.y - 2*Cxy*HalfCenter.x)*HalfCenter.y)*.5/Cdet;
fExp = (Cdet<=0 || e>=10 | Half.z < 0) ? 0 : exp(-e)/sqrt(Cdet);

45. Typical strategy Remember that our is stored power is 1/s
Simple normalized texture, pow 32 = 4 bits precision, pow 128 = 2 bits!
Can renormalize range, to capture some bits
If we want to use very high powers, e.g. 10,000+, really need 16 bits precision

46. Water For Civilization V
Lots of background, but why did we do this?
Needed to make water that worked at a distance, not a smooth reflection
And, wanted a realistic wave combing effect
Does not use a reflection map, high powers let us use an analytic model istead

47. Civ 5’s water Linear combination of 4 moving bump maps
Allows us to accurate wave directions

48. Can we make a cheaper version?
CLEAN Mapping
An extension to LEAN mapping developed after paper published
Common art problem: Went to 5 values, hard to drop into most pipelines, and need more precision
Can we make it use less values
CLEAN mapping Cheap Linear Efficient Antialiased Normal Mapping.

49. Dropping Anisotropy
Cool feature of LEAN maps, but efficiency might be more important
Let’s examine the Beckmann distribution again
Be really nice if we could make only 1 value instead of 3

50. Dropping Terms
Can just approximate the covariance matrix with a diagonal matrix
Then store just X^2 + Y^2 in addition to X,Y
Remember that we aren’t after X^2 – Y^2, but rather the variance of it

51. CLEAN Mapping
Now we have only 3 terms to store. X, Y, X^2 + Y^2, can store in 3 values
Then, calculating the variance:

52. Combining CLEAN Maps
Combing two maps M1 and M2 with lerp factor t

53. Coming CLEAN Most of the high level benefits of LEAN mapping
About half the data costs
Does not support anisotropy

54. Conclusions Normal map filtering = solved problem
Cheap, easy to make art for
Huge Visual Impact
NO EXCUSE to have messy specular!

55. Thanks
Marc Olano – can find I3D paper on his website
Firaxis Games