Spectacular Specular -LEAN and CLEAN specular highlights Dan Baker Firaxis Games

Motivation Observations from Movie folks “The most important aspect of rendering for a movie is anti-aliasing” “Games today still don’t look as good as animated movies 12 years ago.” “Why in video games does everything look so shiny?” Things I have heard over the years

Reality Check Have a habit of comparing ourselves against other games Therefore miss the obvious: Some of our materials don’t look anything like they should And they alias And they sparkle

Shiny things Water, Metal, commonly seen in games Used for water in Civilization V, but metals suffer from similar problems Will tour two common lighting models, Phong and Blinn Phong Both have huge problems

Phong Simple to implement Since L is constant for environments, can turn specular part into a preconvolved environment

Sometimes accurate Good for perfect reflectors,like still water. But, perfect reflectors = high power Reflective materials have powers > 10000, but that will cause all sorts of problems for a point light

Problems with Phong Aliases Can’t get elongated reflections Pretty inaccurate – very plastic look to it No good way to add normals maps together Anisotropic (grooved) materials will lose there anisotropy at zoom Can’t use high powers or else!

Problems with Phong Only see a reflection on the very flat part of the beach, even with perfect use of Phong, won’t work. Classic Scene, sunset. Can’t get this with Phong lighting

Blinn Phong Much more accurate if we have real lights in our scene Can get elongated shapes Cheap to evaluate Note, this is the normalized version of Blinn-Phong

Blinn Phong problems Aliases Shading becomes very wrong with roughness Highlights change based on pixel coverage, and become Anisotropic (groved) materials will lose there anisotropy at zoom Can’t do environments easily Can’t use high powers or else!

Spectacular Specular fail Down sampled of the final render What Blinn-Phong gives us A very rough wave filters to a perfectly flat standing water, An example The effects are profound. If we take the same image and downsaple it, we get a very different thing then if we filter the normal map. Infact, we are being severely over shiny about things.

Bump Filtering Confirm with imperical data. The left side would look like the right side by most techinques. As we zoom this image out we notice massive differences in shinyness, even though it’s the same material Exactly the same substance, but one side is wrinkled. This completely changes the reflections. (Thanks to Chipotle for the foil)

Overviews problems “The shimmies”, “The speckles” Lots and lots of talks about this problem The more substantial the normal map, the higher the power, the more noise we get. Lots of artist tweaking, limits to our data Reflections are just plan wrong at distant scale, makes objects way over-shiny Can’t add normal maps together easily to get detail maps

Why this happens The integral of a function over a range of inputs isn’t the same as function with inputs integrated over a range Where R is a region of a texture, F is our shader (in this case, a Phong or Blinn Phong shader) . The second version is a discrete version, where W is the sample weights from our hardware filtering. Left side is what we want. This is the evaluation at every thoeretical texel that contributes to our scene. But, what we end up doing is the right side. As seen, these two are very different.

How do Movies solve this? Typically use REYES, or more advanced techniques Roughly equivalent of shading every relevant texel and averaging the results Very expensive, potentially thousands of shader evals per pixel Movies typically decouple shading frequency and object rasterization. This gets rid of a huge number of issues, though still not completely ‘correct’. Likely will never do this for games, since objects small on screen can be hugely expensive.

Dreaming the dream Ideal lighting model: Can use any power we want Will deal with zooming in and out correctly Won’t alias Easy to use: compatible with our current pipeline Relatively inexpensive Can add normals together Can use all of our MIP hardware

Formal definition What we really want is to build a replacement for Blinn Phong that has this property (where F is basically our shader): We want a BRDF that is the same when we average all the results at the texel level with the results if we average the inputs. We cannot do the first one feasibly, but we can do the second (and it is already done via MIP maps and hadware).

LEAN Mapping Linear Efficient Antialiased Normal Mapping Considered Fast Antialiased Reflectance Texture Mapping Fast and flexible solution for bump filtering Shiny bumps won’t alias Distant bumps will change surface shading Directional bumps will become anisotropic highlight Allows blending layers of bumps Works with existing Blinn-Phong pipeline

Scale problem solved Normal Map LEAN Map

Prior Work Posed by Kajiya 1985 Monte-Carlo Cabral et al. 1987, Westin et al. 1992, Becker & Max 1993 Multi-lobed distributions Fournier 1992, Han et al. 2007 Single Gaussian/Beckmann distribution Olano & North 1997, Schilling 1997, Toksvig 2005 Diffuse [Kilgard 2000] Some previous attempts, some more worth mentioning

Beckman Shading Model The math is simpler then it looks. We are raising the power based on the distance of the half angle from the normal. This second formulation rolls the power into a covariance matrix, thereby giving us anisotropic power (e.g. two powers, one for X and one for Y).

Probability Distributions in Shading Distribution of microfacet normals Perfectly reflective facets Only facets oriented with reflect to Look up probability of in distribution Beckmann distribution Gaussian of facet tangents = projection

Filtering Filter is linear combination over kernel Linear representation → any linear filter Summed Area, EWA, … MIP map, Hardware Anisotropic We need a BRDF that is linear

Filtering: Gaussians Gaussian described by mean and variance Mean combines linearly Variance does not, but second moment does

Blinn-Phong ↔ Beckmann Blinn-Phong approximates Gaussian [Lyon 1993] Better fit as increases Variance , normalize with Bechmann is very close a normalized Blinn-Phong. We were using normalization factor of s for Blinn-Phong.

Blinn-Phong ↔ Beckmann Blinn Phong and Bechmann are nearly equivalant Blinn-Phong Beckmann

LEAN mapping Blinn-Phong ↔ Beckmann Filtering Bumps Sub-facet shading Layers of bumps

Distributions & Bumps If the normal is changing our surface orientation, is there any way to add them together? Does that even meaning?

LEAN Mapping Beckman distribution can be broken into pieces that filter, but doesn’t deal with the normals. Key insight: We think of the normal instead as a shift of the distribution of microfacets

Distributions Beckman distribution works on a 2d plane. The blue discs represent the distribution of normals. Rather then change the orientation of the surface, we simply shift the center location of the distribution of normals by the x,y component of the normals. Thus, we interpret the normal as a shift in distribution, rather then a change in surface orientation The green arrow represents the distance used by the beckman distrubtion. It is a vector in 2D, and thus the beckman distribution can be anisotropic since the x and y directions can have what is in effect different components.

Filtering Bumps Rather than bump-local frame Use surface tangent frame Bump normal = mean of off-center distribution

Surface-frame Beckmann Bumps vs. Surface Frame Bump-frame Beckmann Surface-frame Beckmann

LEAN Data Normal (for diffuse) Bump center in tangent frame Second moments

LEAN Use Pre-process Render-time Seed textures with , and Build MIP chain Render-time Look up with HW filtering Reconstruct 2D covariance Compute diffuse & specular per light

Sub-facet Shading What about base specularity? Given base Blinn-Phong exponent, Base Beckmann distribution One of these at each facet = convolution Gaussians convolve by adding ’s Fold into , or add when reconstructing

LEAN Map features Seamless replacement for Blinn-Phong Specular bump antialiasing Turns directional bumps into anisotropic microfacets

Bump Layers Uses Bump motion (ocean waves) Detail texture Decals Our approach Conceptually a linear combination of heights Equivalent to linear combination of Even from normal maps

Bump Layers: The Tricky Part What about ? Expands out to , , and terms terms are in , terms are in terms are new: Total of four new cross terms

Layering Options Generate single combined LEAN map Mix actual heights, or use mixing equations Time varying: need to generate per-frame Decal or detail: need high-res LEAN map Generate mixing texture One per pair of layers Decal or detail: need high-res LEAN mixture maps Approximate cross terms Use rather than a filtered mixing texture

Layer Options Source 1 Source 1 Source 2 Source 2 Mixed Mixture Texture Approximation Single LEAN Map MIP Biased

1600 x 1200, single full screen object Performance Single Layer Two Layers Blinn-Phong LEAN Per-frame Mix texture Approx ATI Radeon HD 5870 1570 FPS 1540 FPS 917 FPS 1450 FPS 1458 FPS D3D Instructions 30 ALU 1 TEX 42 ALU 2 TEX 50 ALU 3 TEX 54 ALU 5 TEX 4 TEX 1600 x 1200, single full screen object

Converting Blinn-Phong Data So fast could be done at load time float3 tn = tex2D(normalMap, coord); float3 N = float3(2*tn.xy-1, tn.z); float2 B = N.xy/(ScaleFactor&N.z); float3 M = float3(B.x*B.x + 1/s, B.y*B.y + 1/s, B.x*B.y) Output.lean1 = float4(tn, .5*M.z + .5) Output.lean2 = float4(.5*B + .5, M.xy) S is the power, ScaleFactor is a reranging factor

Texture Compression and Precision Normal maps get big, painful to compress Lean MAPs require 5 fields x,y, x^2, xy, y^2 Caveat: The precision matters. Unlike other techniques, we are using the normal filtering hardware

Obligatory Shader Code float4 f4BaseMeshColor = tex2D(BaseMeshColor, f2BaseTexCoord); float4 f4BaseColor = tex2D(LeanTextureMap1, f2BaseTexCoord); float Var = tex2D(LeanTextureMap2,f2BaseTexCoord).x; float GradientScale = g_fLeanMapScale; float VarianceScale = GradientScale*GradientScale; float2 Gradient = float2(f4BaseColor.x*2-1,f4BaseColor.y*2-1) *GradientScale; float3 Covar = float3(f4BaseColor.zw , Var*2 - 1) * VarianceScale; // turn moments into elements of covariance matrix, matrix is mat4(Covar.x,Covar.z,Covar,z,Covar.y) Covar -= float3(Gradient.xy*Gradient.xy, Gradient.x*Gradient.y); float3 Half = normalize(ViewDir + LightDir); //Transform half angle back into tangent space Half = mul(mTS, Half); float2 HalfCenter = Half.xy/Half.z - Gradient.xy; //Now calculate the spec float Cxx = Covar.x + 1/g_fExp, Cyy = Covar.y + 1/g_fExp, Cxy = Covar.z; float Cdet = Cxx*Cyy - Cxy*Cxy; float e = (Cyy*HalfCenter.x*HalfCenter.x + (Cyy*HalfCenter.y - 2*Cxy*HalfCenter.x)*HalfCenter.y)*.5/Cdet; fExp = (Cdet<=0 || e>=10 | Half.z < 0) ? 0 : exp(-e)/sqrt(Cdet);

Typical strategy Remember that our is stored power is 1/s Simple normalized texture, pow 32 = 4 bits precision, pow 128 = 2 bits! Can renormalize range, to capture some bits If we want to use very high powers, e.g. 10,000+, really need 16 bits precision

Water For Civilization V Lots of background, but why did we do this? Needed to make water that worked at a distance, not a smooth reflection And, wanted a realistic wave combing effect Does not use a reflection map, high powers let us use an analytic model istead

Civ 5’s water Linear combination of 4 moving bump maps Allows us to accurate wave directions

Can we make a cheaper version? CLEAN Mapping An extension to LEAN mapping developed after paper published Common art problem: Went to 5 values, hard to drop into most pipelines, and need more precision Can we make it use less values CLEAN mapping Cheap Linear Efficient Antialiased Normal Mapping.

Dropping Anisotropy Cool feature of LEAN maps, but efficiency might be more important Let’s examine the Beckmann distribution again Be really nice if we could make only 1 value instead of 3

Dropping Terms Can just approximate the covariance matrix with a diagonal matrix Then store just X^2 + Y^2 in addition to X,Y Remember that we aren’t after X^2 – Y^2, but rather the variance of it

CLEAN Mapping Now we have only 3 terms to store. X, Y, X^2 + Y^2, can store in 3 values Then, calculating the variance:

Combining CLEAN Maps Combing two maps M1 and M2 with lerp factor t

Coming CLEAN Most of the high level benefits of LEAN mapping About half the data costs Does not support anisotropy

Conclusions Normal map filtering = solved problem Cheap, easy to make art for Huge Visual Impact NO EXCUSE to have messy specular!

Thanks Marc Olano – can find I3D paper on his website Firaxis Games