1. Pre-lighting in Resistance 2
Mark Lee

2. Outline Our past approach to dynamic lights.
Deferred lighting and pre-lighting.
Pre-lighting stages.
Implementation tips.
Pros and cons.
Classification of light and geometry types. Here we are talking about how we tackle the problem of dynamic lights on static and dynamic geometry.

3. The problem (a.k.a. multipass lighting)
for each dynamic light
for each mesh light intersects
render mesh with lighting
O(M*L)
Too much redundant work:
Repeat vertex transformation for each light.
Repeat texture lookups for each light.
Hard to optimize, we were often vertex bound.
Each object we render needs to track lights which illuminate it on PPU/SPU.

4. One solution O(M+L) Lighting is decoupled from geometry complexity.
for each mesh
render mesh
for each light
render light
O(M+L)
Lighting is decoupled from geometry complexity.
there are different ways to approach this problem

5. G-Buffer
Caches inputs to lighting pass to multiple buffers (G-buffer):
Depth, normals, specular power, albedo, baked lighting, gloss, etc.
All lighting is performed in screen space.
Nicely separates scene geometry from lighting, once geometry is written into G-Buffer, it is shadowed and lit automatically.
G-buffer also available for post processing.
Geometric buffer.

6. do lighting
Use MRT to render to all input buffers simultaneously.

7. G-Buffer issues for us Prohibitive memory footprint.
1280*720 MSAA buffer is 7.3mb, multiplied by 5 is 38mb.
Unproven technology on the PS3 at the time.
A pretty drastic change to implement.

8. Pre-lighting / Light pre-pass
Like the G-Buffer approach except:
Caches only a subset of material properties (in our case normals and specular power) in an initial geometry pass.
A screen space pre-lighting pass is done before the main geometry pass.
All the other material properties are supplied in a second geometry pass.

9. pre-lighting render scene
Specular power is in alpha channel of normal buffer. This is an implementation detail, store whatever makes sense.
Scene geometry is rendered twice!
Minimal amount of extra memory needed compared to G-buffer.
render
scene

10. Rendering flow in Resistance 2
Render depth, normals, and selected material properties.
Resolve the depth buffer.
Accumulate sun shadows.
Accumulate dynamic lights.
Render scene as before but additionally lookup the sun shadow and prelighting buffers.
Rest of frame...

11. Step 1 – Depth & normals

12. Writing depth and normals
R2 used 2x MSAA.
Write out normals when you are rendering your early depth pass.
Use primary render buffer to store normals.
Write specular power into alpha channel of normal buffer.
Use discard in fragment programs to achieve alpha testing.
Normals are stored in viewspace as 3 8-bit components for simplicity.
No need for MRT.

13. The viewspace normal myth
Store viewspace x and y, and reconstruct z…
i.e. z = sqrt(1 – x*x – y*y)
Common misconception in a lot of past deferred rendering literature.
Z can go negative due to perspective projection.
When z goes negative, errors are subtle and continuous so easy to overlook.
We store the full xyz components for simplicity.
May or may not be a problem – depending on FOV

14. The viewspace normal myth
Fig. A
Fig. B
picture what happens to the viewspace normals as the camera rotates…
normal = (1, 0, 0)
normal = (-1, 0, 0)

15. Viewspace normal error
correct
incorrect

16. Step 2 – Depth resolve

17. Depth resolve Convert MSAA to non-MSAA resolution.
Moved earlier to allow us to do stenciling optimizations on non-MSAA lighting and shadow buffers.
No extra work, the same depth buffer is used for all normal post-resolve rendering.
Quick review of what a resolve does…
Resolve is typically done late in the frame…

18. Step 3 – Accumulate sun shadows

19. Sun shadows All done in screen space.
Treated separately than shadows from dynamic lights due to differences in scale.
Used as a darkening pass to already existing light.
Full screen quad for sun, shadows for individual characters are added separately.
Not concerned about blending in with the scene at this point, 0 = in shadow, 1 = no shadow.
Add extra detail to scene since we are using per pixel normals as our inputs.

20. Sun shadows
All sun shadows from static geometry are precomputed in lightmaps.
We just want to accumulate sun shadows from dynamic casters.
for each dynamic caster
compute OBB // use collision rays
merge OBBs where possible
for each OBB
render sun shadow map
for each sun shadow map
render OBB to stencil buffer
render shadow map to sun shadow buffer
Full screen quad for sun, shadows for individual characters are added separately.

21. Sun shadows Min blend used to choose darkest of all inputs.
Originally used an 8-bit buffer but changed to 32-bits for stencil optimizations.
Use lighting buffer as temporary memory, copy to an 8-bit texture afterwards.
Min blend used to keep the darkest shadow of all inputs.
To allow stencil optimizations we use a 32-bit buffer and copy to 8bit afterwards.

22. Which pixels to shadow?

23. Which pixels to shadow?

24. Which pixels to shadow?
Compute OBB in which surrounds bspheres in the sun direction.
Rays are cast to determine how long to extend OBB. Useful since:
Helps to avoid casting shadows onto lower platforms.
Faster, minimizes area of projected quad.

25. Which pixels to shadow?
Use the stencil hardware to figure out where geometry intersects the obb.
Green is the result of the stencil pass.
We render a quad surrounding the whole box.
Everything apart from the green is early stencil rejected.

26. Step 4 – Accumulate dynamic lights
conceptually we want 2 screen space buffers, diffuse and specular.

27. Accumulating light Similar approach to sun shadow buffer.
Render all spotlight shadow maps using D16 linear depth.
For each light:
Lay down stencil volumes.
Rendered screen space projected quad covering light.
Single buffer vs. MRT, LDR vs. HDR.
We only support point and spot. Only spotlights could cast shadows.
For R2 we used a single LDR lighting buffer. Would be good to split going forward.

28. Accumulating light MSAA vs. non-MSAA
Diffuse, shadowing, projected, etc. are all done at non-MSAA resolution.
Specular is 2x super sampled.
These buffers are available to all subsequent rendering passes for the rest of the frame.

29. Dynamic lights where: l is our set of lights
gp is the limited set of geometric/material properties we choose to store for each pixel
mp is the full set of material properties for each pixel
P is the function evaluated in the pre-lighting pass for each light (step 4)
C is our combination function which evaluates the final result (step 5)
In chronological order:
gp = depth normal pass
P = prelighting pass
C = rendering of the scene

30. Lambertian lighting example
gp = normal
P = function inside of sigma
mp = albedo
C = mp * P
we deal with diffuse and specular light independently.

31. Specular lighting example
gp = normal (n), specular power (p)
P = function inside of sigma
mp = gloss
C = mp * P
again, it works fine to just add specular from multiple lights on top of each other.

32. Limitations Limited range of materials can be factored in this way.
There are ways around the limitations:
Extra storage for extra material properties.
Need to encode material type in depth normal pass (could use bits from the stencil buffer).
Conditionally execute different fragment shader code paths in pre-lighting pass depending on material.
Problematic for blended materials, e.g. fur.
In Resistance 2, more complex material types were done with forward rendering.

33. Step 5 – Render scene

34. Rendering the scene
Scene is rendered identically to before with the addition of the lighting and sun shadow buffer lookups.
Smart shadow compositing function used:
“In shadow” global ambient level defined.
Input light level determined from baked lighting.
Geometry is only shadowed if baked light level is above this threshold.
Amount of shadow is determined by light level at that point. Continuous function.
The remaining missing parts of the lighting model are now available.
We are trying to blur the distinction between what shadows are prebaked and what is runtime.
The brighter the region, the more strongly the shadow will contribute.

35. Implementation tips

36. Reconstructing position
Don't store it in your G-buffer.
Don’t do a full matrix transform per pixel.
Instead interpolate vectors from edges of camera frustum such that view z = 1.
Technique not confined to viewspace.

37. Reconstructing position
Scene
linear depth
z = 1
Multiply by *linear depth*

38. Reconstructing depth W-buffering isn’t supported on the PS3.
Linear shadow map tricks don’t work.
Z-buffer review (D3D conventions):
The z value is 0 at near clip and far at the far clip.
The w value is 0 at viewer and far at the far clip.
z/w is in the range 0 to 1.
This is scaled by 2^16 or 2^24 depending on our depth buffer bit depth.

39. Reconstructing depth z = f(vz - n) / (f - n) w = vz zw = z / w
zb = zw*(pow(2, d)-1)
where:
f = far clip
n = near clip
vz = view z
zb = what is stored
in the z-buffer
d = bit depth of
the z buffer
Z (red) is the function encoded by the z column of our projection matrix
W (blue) is the function encoded by the w column of our projection matrix
z/w is what’s stored in the depth buffer

40. Recovering hyperbolic depth
Alias an argb8 texture to the depth buffer.
Using a 24-bit integer depth buffer, 0 maps to near clip and 0x00ffffff maps to far clip.
To recover z/w in C, we would do
When dealing with floats it becomes R G A B ff 00
what is range?
float(r<<16 + g<<8 + b) / f
(r * f + g * 256.f + b) / f

41. Recovering hyperbolic depth
But….
Texture inputs come in at a (0, 1) range
The texture integer to float conversion isn’t operating at full float precision
We are magnifying error in the red and green channels significantly
Solution: round each component back to integer boundaries
float3 rgb = f3tex2D(g_depth_map, uv).rgb;
rgb = round(rgb * 255.0);
float zw = dot(rgb, float3( , 256.0, 1.0 ));
zw *= 1.0 / ;

42. Recovering linear depth
Recall that:
Solving for vz, we get:
Reconstructed linear depth precision will be very skewed – but it’s skewed to where we want it.
z = f(vz - n) / (f - n)
w = vz
zw = (f(vz - n) / (f - n)) / vz
vz = 1.0 / (zw * a + b)
where:
a = (f - n) / -fn
b = 1.0 / n

43. Stenciling algorithm Stencil shadow hardware does this.
clear stencil buffer
if( front facing and depth test passes )
increment stencil
if( back facing and depth test passes )
decrement stencil
render light only to pixels which have non-zero stencil
Stencil shadow hardware does this.
Set culling and depth write to false when rendering volume.
Same issues as stencil shadows
Make sure light volumes are closed.
This only works if camera is outside light volumes.
We want to run expensive fragment programs only on pixels which it actually affects.

44. Object is inside light volume

45. Object is near side of light volume

46. Object is far side of light volume

47. If the camera goes inside light volume
clear stencil buffer
if( front facing and depth test fails )
wrap increment stencil
if( back facing and depth test fails )
wrap decrement stencil
render light only to pixels which have non-zero stencil
Switch to depth fail stencil test.
Only do this when we have to, this disables z-cull optimizations.
Typically we’ll need some fudge factor here.
Stenciling is skipped for smaller lights.
Standard “outside volume” version:
clear stencil buffer
if( front facing and depth test passes )
increment stencil
if( back facing and depth test passes )
decrement stencil
render light only to pixels which have non-zero stencil

48. If the camera goes inside light volume
Red sections is “back facing AND depth test fails”

49. Pros and cons
G-Buffer
Requires only a single geometry pass. Good for vertex bound games.
More complex materials can be implemented.
Not all buffers need to be updated with matching data, e.g. decal tricks.
Pre-lighting / Light pre-pass
Easier to retrofit into "traditional" rendering pipelines. Can keep all your current shaders.
Lower memory and bandwidth usage.
Can reuse your primary shaders for forward rendering of alpha.

50. Problems common to both approaches
Alpha blending is problematic.
MSAA and alpha to coverage can help.
Encoding different material types is not elegant:
A single light may hit many different types of material.
Coherent fragment program dynamic branching can help.
Define the problem with alpha blending – pixel is in depth buffer or it’s not.

51. Questions?