1. 1 Topics: Rendering Systems Basic 3D Concept Rendering System 3D Graphic Effect

2. 2 Interactive Game Loop Game Input (Keyboard, Mouse) Physics Strategy, AI Render Triangles (one frame) Exit? Cleanup States 30 to 60 frames per second

3. 3 3D Graphics Rendering Pipeline Geometry Pipeline –Processing Vertices –Mainly floating-point operations Rasterization Pipeline –Processing Pixels –Mainly dealing with Integer operations Geometry Processing x y z x y z x y z x y z x y z x y Rasterization Processing x y

4. 4 3D Graphics Rendering Pipeline Geometry Pipeline –Processing Vertices –Mainly floating-point operations Rasterization Pipeline –Processing Pixels –Mainly dealing with Integer operations –MMX was originally designed to accelerate this functionality Geometry Processing x y z x y z x y z x y z x y z x y Rasterization Processing x y MMX ISA

5. 5 3D Graphics Rendering Pipeline Geometry Pipeline –Processing Vertices –Mainly floating-point operations –SSE/SSE2 were designed for this part Rasterization Pipeline –Processing Pixels –Mainly dealing with Integer operations –MMX was originally designed to accelerate this functionality Geometry Processing x y z x y z x y z x y z x y z x y Rasterization Processing x y MMX ISA SSE/SSE2 ISA

6. 6 Fixed Function 3D Graphics Pipeline Geometry Pipeline –Processing Vertices –Mainly floating-point operations –SSE/SSE2 were designed for this part Rasterization Pipeline –Processing Pixels –Mainly dealing with Integer operations –MMX was originally designed to accelerate this functionality Geometry Processing x y z x y z x y z x y z x y z x y Rasterization Processing x y Performed by (GP)GPU

7. 7 3D Coordinates Left-handed system is more common for graphics viewing space Camera or viewer position can be set up using 3D API +z +x +y Right-Handed System +z +x +y Left-Handed System

8. 8 Geometry Format ─ Vertex Coordinates (X1, Y1, Z1) (X2, Y2, Z2) (X3, Y3, Z3) +Z +X +Y X1 Z1 Y1

9. 9 Geometry Format ─ Vertex Normals (NX1, NY1, NZ1) (NX2, NY2, NZ2) (NX3, NY3, NZ3) +Z +X +Y

10. 10 Geometry Format ─ Vertex Colors (R1, G1, B1, A1) (R2, G2, B2, A2) (R3, G3, B3, A3) +Z +X +Y

11. 11 Triangle-based Geometry Representation V5 V4 V3 V2 V1 V2 V3V4 V5 V6 V7 V8 V9 V5 V4 V1 V3 V2 Triangle List (note the vertex order) Triangle StripTriangle Fan

12. 12 Specifying a 3D object Vertex ordering is critical V1 V2V3 V4 V5 V6V7 Triangle list {v1, v3, v2}, {v1, v5, v3}, {v5, v6, v3}, {v4, v3, v6}, {v1, v7, v6}, {v1, v6, v5} Triangle strip {v5, v3, v1, v2}, {v5, v6, v3, v4}, {v7, v6, v1, v5}

13. 13 V8 Specifying a 3D object Vertex ordering is critical We will discuss normal computation later in this lecture V1 V2V3 V4 V6V7 Triangle list {v1, v2, v7}, {v2, v8, v7}, {v2, v3, v4}, {v2, v4, v8}, {v4, v7, v8}, {v4, v6, v7} Triangle strip {v1, v2, v7, v8}, {v3, v4, v2, v8}, {v6, v7, v4, v8}

14. 14 3D Rendering Pipeline ClippingPerspective DivideViewport Transform Rasterization Projection TransformWorld TransformView TransformLightingBackface Culling

15. 15 Transformation Pipeline World Transformation –Model coordinates  World coordinates View Transformation –World coordinates  Camera space Projection Transformation –Camera space  View Plane These are a series of matrix multiplications

16. 16 World Transformation Translation Rotation Scaling +x +z +y World origin World Coordinates Local model coordinates

17. 17 View Transformation +x +z +y World origin World Coordinates +y +x +z Camera position Look vector

18. 18 Projection Transformation Set up camera internals Set up –Field of View (FOV) –View frustum –View planes Will discuss later

19. 19 Homogeneous Coordinates Enable all transformations to be done by “ multiplication ” –Primarily for translation (See next few slides) Add one coordinate (w) to a 3D vector Each vertex has [x, y, z, w] –W will be useful for perspective projection –W should be 1 in a Cartesian Coordinate System

20. 20 Transformation 1: Translation (Offset) +x +z +y +x +z +y (x, y, z) (x t, y t, z t )

21. 21 Translation Matrix

22. 22 Transformation 2: Scaling +x +z +y +x +z +y

23. 23 Scaling Matrix

24. 24 Transformation 3: Rotation +x +z +y +x +z

25. 25 2D Rotation +x +y +x +y  (x, y) (x’, y’) +x +y (x, y)   Rotate along which axis?

26. 26 3D Rotation Matrix Rotation along Z axis Rotation along Y axis Rotation along X axis

27. 27 Non-Commutative Property (1) 1. Counter-clockwise 90 o along y 2. Clockwise 90 o along x +x +z +y   +x +z+y 1. Clockwise 90 o along x 2. Counter-clockwise 90 o along y  

28. 28 Non-Commutative Property (1) +x +z +y   +x +z +y     (x’’, y’’, z’’) = (-z, -x, y) (x’’, y’’, z’’) = (-y, -z, x)

29. 29 Non-Commutative Property (2) 1. Translation by (x, y, z) 2. Scale by 2 times +x +z +y 1. Scale by 2 times 2. Translation by (x, y, z)   +x +z +y 

30. 30 Non-Commutative Property (2) +x +z +y   +x +z +y    (x’’, y’’, z’’) = (x*Rx+Rx*Tx, y*Ry+Ry*Ty, z*Rz+Rz*Tz)  (x’’, y’’, z’’) = (x*Rx+Tx, y*Ry+Ty, z*Rz+Tz) Offsets were scaled as well

31. 31 Non-Commutative Property Ordering matters ! Be careful when performing matrix multiplication

32. 32 Parallel Projection Viewer’s position View plane Project from 3D space to viewer ’ s 2D space

33. 33 Perspective Projection The farther the object is, the smaller it appears Some photo editing software allows you to perform “ Perspective Correction ” Viewer’s position View plane

34. 34 Perspective Projection x y z (x,y,z) (xs,ys,d) d xs ys x y z

35. 35 Perspective Projection x y z (x,y,z) (xs,ys,d) d xs ys x y z

36. 36 Viewing Frustum Far plane Viewer’s position Near plane Think about looking through a window in a dark room

37. 37 Viewing Frustum Near plane Far plane Viewer’s position

38. 38 Define Viewing Frustum (Projection Transformation) Viewer’s position +z Parameters: Fov: Field of View Aspect ratio = Width/Height Near z Far z fov/2 +y Far z Near z Width Height

39. 39 Viewport Transformation The actual 2D projection to the viewer Copy to your back buffer (frame buffer) Can be programmed, scaled,..

40. 40 Other Geometry Optimizations Backface Culling Clipping Hidden surface removal (Occlusion)

41. 41 Backface Culling Determine “ facing direction ” Triangle order matters Check if the normal is facing the camera Surface vectors Eye vector

42. 42 When to Perform Backface Culling? ClippingPerspective DivideViewport Transform Rasterization Projection TransformWorld TransformView TransformLighting Backface Culling ClippingPerspective DivideViewport Transform Rasterization Projection TransformWorld TransformView TransformLightingBackface Culling How? Transform your camera to the world space first!

43. 43 3D Clipping Test 6 planes if a triangle is inside, outside, or partially inside the view frustum Clipping creates new triangles (triangulation) –Interpolate new vertices info

44. 44 Illumination Models It won ’ t look 3D without lighting Part of geometry processing Illumination Types –Ambient –Diffuse –Specular –Emissive

45. 45 Local vs. Global Illumination Local Illumination –Direct illumination: Light shines on all objects without blocking or reflection –Used in most games Global Illumination –Indirect illumination: Light bounces from one object to other objects –Add more realism (non real-time rendering) –Computationally much more expensive –Ray tracing, radiosity ?

46. 46 Common Light Sources Directional Light (Infinitely far away) Point Light (Emit in all directions) Spot Light (Emit within a cone) Inner cone outer cone

47. 47 Light Source Properties Position Range –Specifying the visibility Attenuation –The farther the light source, the dimmer the color

48. 48 Falloff effect Spotlight Effect Similar idea to specular lighting Falloff factor determines the fading effect of a spotlight “ f ” exponentially decreases the cos  value L  d  

49. 49 Rasterization: Shading a Triangle Paint each pixel ’ s color (by calculating light intensity) on your display (Typically object-based) Gouraud Shading: –Intensity Interpolation of vertices RGB(255,0,0) RGB(0,255,0) RGB(0,0,255)

50. 50 Gouraud Shading RGB(255,0,0) RGB(0,255,0) RGB(0,0,255) RGB(127,0,127) RGB(127,127,0) RGB(127,64,64) Scan conversion algorithm Scan line

51. 51 Comparison of Shading Methods Phong shading –requires generating per-pixel normals to compute light intensity for each pixel, not efficient for games –Can be done on GPGPU today using Cg or HLSL Gouraud shading is supported by Graphics hardware Flat shadingGouraud shadingPhong shading Source: Michal Necasek

52. 52 Double Buffering Display refreshes at 60  75 Hz Rendering could be “ faster ” than the refresh period Too fast leads to –Frames not shown Too slow leads to –New and old frame mixed –Flickering Solution: –Double or multiple buffering surface1surface2 Front Buffer Back Buffer surface2 surface1 swap

53. 53 Z-Buffer Also called depth buffer Draw the pixel which is nearest to the viewer Number of the entries corresponding to the screen resolution (e.g. 1024x768 should have a 768k-entry Z-buffer) –Some card perform Z-compression Perform Z test before drawing a pixel z1 z2z3

54. 54 Aliasing Jagged line (or staircase) Can be improved by increasing resolution (i.e. more pixels)

55. 55 Anti-Aliasing by Multisampling GPU samples multiple locations for a pixel Several different methods –e.g. grid, random, GeForce ’ s quincunx Downside –Blurry image –Increased memory (e.g. z-buffer) storage for subpixel information Actual Screen Pixels 3x3 Virtual Pixels (Bartlett window) (255, 159, 159) Example 11 112 2 224 (255,255,255)(255,0,0)(255,255,255) (255,0,0)(255,255,255) (255,0,0)

56. 56 Anti-Aliasing Example No MSAA With MSAA Ideal

57. 57 Visualizing Anti-Aliasing Example No MSAAWith MSAA

58. 58 Texture Mapping Rendering tiny triangles is slow Players won ’ t even look at some certain details –Sky, clouds, walls, terrain, wood patterns, etc. Simple way to add details and enhance realism Use 2D images to map polygons Images are composed of 2D “ texels ” Can be used to substitute or blend the lit color of a texture-mapped surface

59. 59 Texture Mapping Introduce one more component to geometry –Position coordinates –Normal vector –Color –Texture coordinates Texture info –(u, v) coordinates for each vertex –Typically range from 0 to 1 (0,0) from upper left corner

60. 60 Texture Mapping Texture: hunk.jpg (0,0) (1,1) v0 v1 v2 v3 v7v8 (0,1) (1,0) {v1.x, v0.y, v0.z, …, 1, 0}, {v2.x, v1.y, v1.z, …, 1, 1}, {v0.x, v2.y, v2.z, …, 0, 0}, {v3.x, v3.y, v3.z, …, 0, 1}, uv

61. 61 Texture Mapping Texture 2 (0,0) (1,1) v0 v1 v2 v3 v7v8 (0,0) (1,1) (0,1) (1,0) Texture 1

62. 62 Texture Mapping (0,0) (1,1) {v0.x, v0.y, v0.z, …, 0, 0}, {v1.x, v1.y, v1.z, …, 5, 0}, {v2.x, v2.y, v2.z, …, 5, 3}, {v3.x, v3.y, v3.z, …, 0, 3}, uv (1,0) (0,1)

63. 63 Texture Mapping {v0.x, v0.y, v0.z, …, 0, 0}, {v1.x, v1.y, v1.z, …, 6, 0}, {v2.x, v2.y, v2.z, …, 6, 6}, {v3.x, v3.y, v3.z, …, 0, 6}, uv

64. 64 Magnification Texel and pixel mapping is rarely 1-to-1 Mapped triangle is very close to the camera One texel maps to multiple pixels Pixels on screen Texels

65. 65 Nearest Point Sampling (for Magnification) Choose the texel nearest the pixel ’ s center Pixels on screen Texels

66. 66 Bi-linear Filtering (for Magnification) Average for the 2x2 texels surrounding a given pixel Texels R=102 G=102 B=51 R=255 G=204 B=102 R=253 G=230 B=145 R=247 G=237 B=141 Pixels on screen R=214 G=193 B=110

67. 67 Minification Texel and pixel mapping is rarely 1-to-1 Multiple texels map to one pixel Pixels on screen Texels Color?

68. 68 Nearest Point Sampling (for Minification) Choose the texel nearest the pixel ’ s center Pixels on screen

69. 69 Bi-linear Filtering (for Minification) Average for the 2x2 texels surrounding a given pixel Pixels on screen R=135 G=119 B=23 R=252 G=219 B=96 R=0 G=0 B=0 R=234 G=189 B=0 R=155 G=132 B=30

70. 70 Mip-mapping Multiple versions are provided for the same texture Different versions have different levels of details –E.g., 7 LOD maps: 256x256, 128x128, 64x64, 32x32, 16x16, 8x8, 4x4 –Choose the closet maps to render a surface Maps can be automatically generated by 3D API Accelerate texture mapping for far-away polygons More space to store texture maps

71. 71 Mip-mapping API or Hardware can –Generate (lower resolution) mip maps automatically –Choose the right one for the viewer Good performance for far triangles Good LOD for close-by objects –Tri-linearly interpolate

72. 72 Tri-linear Filtering using Mip maps Interpolate between mipmaps Higher Res. Mip Map Screen Pixel R=155 G=132 B=30 R=229 G=208 B=119 R=233 G=227 B=143 R=178 G=179 B=90 R=199 G=187 B=96 Lower Res. Mip Map R=147 G=114 B=117 R=58 G=0 B=0 R=66 G=0 B=0 R=106 G=80 B=74 R=94 G=49 B=48  R=147 G=118 B=72

73. 73 Color Blending and Alpha Blending Transparency effect (e.g. Water, glasses, etc.) Source color blended with destination color Several blending methods –Additive C = SrcPixel  (1,1,1,1) + DstPixel  (1,1,1,1) = SrcPixel + DstPixel –Subtractive C = SrcPixel  (1,1,1,1) ― DstPixel  (1,1,1,1) = SrcPixel ― DstPixel –Multiplicative C = DstPixel  SrcPixel –Using Alpha value in the color (Alpha Blending) C = SrcPixel  ( , , ,  ) + DstPixel  (1- ,1- ,1- ,1-  ) –And many more in the API …

74. 74 Alpha Blending (Inverse Source form) No transparency Src=0.2 (triangle) Dest=0.8 (square) Src=0.5 (triangle) Dest=0.5 (square) Src=0.8 (triangle) Dest=0.2 (square)

75. 75 Another Example Without Transparency

76. 76 Another Alpha Blending Example Src=0.3 (rect) Dest=0.7 (checker) Src=0.5 (orange rect) Dest=0.5 Src=0.6 (Triangle) Dest=0.4

77. 77 Alpha Test Reject pixels by checking their alpha values Model fences, chicken wires, etc. Texture: bar.jpg if (  op val) reject pixel else accept pixel Straightforward texture mapping

78. 78 Multi-Texturing Map multiple texture to a polygon –Common APIs support 8 textures Performance will be reduced Multiple texturing stages in the pipeline Texture color will be calculated by –Multiplication –Addition –Subtraction Operation 1 Texture1 color lit color Operation 2 Texture2 color Operation 3 Texture3 color Operation N TextureN color Final Color

79. 79 Multi-Texturing Example: Light Mapping  Some crumpled paper texture A spotlight map Different alpha blending

80. 80 Stenciling Stencil buffer –To reject certain pixels to be displayed –To create special effect similar to alpha test Mask out part of the screen –Set together with Z-buffer in 3D API –Perform prior to Z-buffer test if ((stencil ref & mask) op (pixel val & mask)) accept pixel else reject pixel

81. 81 Stencil Buffer Example This window area Is set to be drawn by stencil buffer Reject pixels inside this area

82. 82 Apply Fog Effect Provide depth cue –Simulate weather condition –Avoid popping effect Color blending fog sto p fog start Calculate distance Calculate intensity of vertex color based on distance –Color blending –Linear density, exponential density Blending color schemes –Per-vertex (then Interpolate pixels), less expensive –Per-fragment basis (Nvidia hardware), better quality