Gaming Technologies Craig Peeper Software Architect Windows Graphics & Gaming Technologies Microsoft Corporation Craig Peeper Software Architect Windows Graphics & Gaming Technologies Microsoft Corporation

Overview Games Yesterday & Today Game Components PC Platform & WGF 2.0 Game Trends Big Challenges in Game Creation Summary Games Yesterday & Today Game Components PC Platform & WGF 2.0 Game Trends Big Challenges in Game Creation Summary

Games Yesterday & Today Quake Far Cry Prince of Persia Age of Empires III Max Payne Doom

Platform Game Components Content + Engine

Content Components story scripting animation texturing / surfacing lighting Content as well as … user interface modeling effects level design sound visual technologies

Content Creation Tool driven Combination of tools forms “Art Pipeline” Character Pipeline 3D Model/Sculpting Animation/Rigging Skinning Texturing Lighting/Shading An artist often specializes on one part of pipeline Tool driven Combination of tools forms “Art Pipeline” Character Pipeline 3D Model/Sculpting Animation/Rigging Skinning Texturing Lighting/Shading An artist often specializes on one part of pipeline

Game Engine Components AI Physics Audio Graphics Networking + I/O Engine as well as … installation patching resource management multithreading disk i/o state save/restore

Game Engines Studio chooses to build or buy Quake Source, Unreal engines Renderware, Gamebryo middleware Often come with great authoring tools (level editors, etc.) Componentized software May buy specialized components How many people can write a physics engine? Video codecs, etc Engine may be optimized for game genre/style Engine bounds range of content Global illumination, terrain, indoor scenes Studio chooses to build or buy Quake Source, Unreal engines Renderware, Gamebryo middleware Often come with great authoring tools (level editors, etc.) Componentized software May buy specialized components How many people can write a physics engine? Video codecs, etc Engine may be optimized for game genre/style Engine bounds range of content Global illumination, terrain, indoor scenes

Game Platform Components CPU + Memory Storage Audio Graphics Networking + Input SW + HW Platform

Game Platforms Console 10M – 100M installed base “fixed platform” Moderately powerful hardware Standard & High definition video resolution (1080i, 720p) Controller, wheel (& keyboard, mouse) input Refreshed every 4-5 years 10’ experience on TV Console 10M – 100M installed base “fixed platform” Moderately powerful hardware Standard & High definition video resolution (1080i, 720p) Controller, wheel (& keyboard, mouse) input Refreshed every 4-5 years 10’ experience on TV

Game Platforms (cont.) PC 600M installed base (running Microsoft Windows) Non-uniform platform (video cards, memory, CPU) Most powerful hardware Least powerful too Very high resolution (1600x1200, 1920x1080, …) Keyboard, mouse (& controller) input 2’ & 10’ (desk vs living room) Notebook (portable gaming) Upgraded yearly! PC 600M installed base (running Microsoft Windows) Non-uniform platform (video cards, memory, CPU) Most powerful hardware Least powerful too Very high resolution (1600x1200, 1920x1080, …) Keyboard, mouse (& controller) input 2’ & 10’ (desk vs living room) Notebook (portable gaming) Upgraded yearly!

Other Platforms Hand-held Dedicated consoles (PSP, Gameboy) Controller-like input Mobile phones 700M phones/year Non-uniform platform Keypad input Modest resolution (320x200 … 640x480) 1995-class PC graphics Arcade Industrial version of home console? Hand-held Dedicated consoles (PSP, Gameboy) Controller-like input Mobile phones 700M phones/year Non-uniform platform Keypad input Modest resolution (320x200 … 640x480) 1995-class PC graphics Arcade Industrial version of home console?

PC as a Platform Large installed base Least expensive to publish on No platform royalties Large hw variation, rapidly evolving Large number of support calls Supports rapid innovation Profit from selling PC hardware! Games must span variability Influences content & engine Reach versus development cost Challenges Abstract hardware differences in platform Support innovation Large installed base Least expensive to publish on No platform royalties Large hw variation, rapidly evolving Large number of support calls Supports rapid innovation Profit from selling PC hardware! Games must span variability Influences content & engine Reach versus development cost Challenges Abstract hardware differences in platform Support innovation

New Direction for PC Platform Reduce “unnecessary” hardware variation Programmable hardware causes new problems Shading programs are part of content not engine Shading programs more complex Content must be portable & reusable Approach: Specify details that matter Representation, precision, accuracy Don’t specify non-observable implementation details Use virtual machine model Reduce “unnecessary” hardware variation Programmable hardware causes new problems Shading programs are part of content not engine Shading programs more complex Content must be portable & reusable Approach: Specify details that matter Representation, precision, accuracy Don’t specify non-observable implementation details Use virtual machine model

Windows Graphics Foundation 2.0 Maintain consistent feature set Eliminate capability bits (optional features) Well defined behavior Multiple implementations produce same result E.g., IEEE-754 arithmetic Migrate legacy fixed functions to shaders Fixed-function lighting, alpha test, etc Consistent shader programming model Vertex and pixel shaders use common instruction set Obsolete assembly level programming Add optimizations where large benefit Scenario-specific formats (high dynamic range, normal maps) Maintain consistent feature set Eliminate capability bits (optional features) Well defined behavior Multiple implementations produce same result E.g., IEEE-754 arithmetic Migrate legacy fixed functions to shaders Fixed-function lighting, alpha test, etc Consistent shader programming model Vertex and pixel shaders use common instruction set Obsolete assembly level programming Add optimizations where large benefit Scenario-specific formats (high dynamic range, normal maps)

Innovation in WGF 2.0 Integer & floating-point processing More general data flow Stream out to memory after vertex processing Unify textures, render targets, vertex buffers “Arrayed” resources Select array element to render to in shader Enables single-pass render to cube map New pipeline stage (geometry shader) Whole-primitive processing Integer & floating-point processing More general data flow Stream out to memory after vertex processing Unify textures, render targets, vertex buffers “Arrayed” resources Select array element to render to in shader Enables single-pass render to cube map New pipeline stage (geometry shader) Whole-primitive processing

Geometry Shader Shader stage that “sees” whole primitive 3 vertices of a triangle (or 2 vertices of a line) As well as adjacency (6 vertices per triangle) Can compute per-primitive data Pass to pixel shader Barycentric parameters Plane equations Can amplify data Extrude edges, expand points, generate shells Find silhouette edges Limited amplification (not a general tessellator) Shader stage that “sees” whole primitive 3 vertices of a triangle (or 2 vertices of a line) As well as adjacency (6 vertices per triangle) Can compute per-primitive data Pass to pixel shader Barycentric parameters Plane equations Can amplify data Extrude edges, expand points, generate shells Find silhouette edges Limited amplification (not a general tessellator)

WGF 2.0 Pipeline Vertex Buffer Input Assembler Vertex Shader Setup Rasterizer Output Merger Pixel Shader Geometry Shader Index Buffer Texture Render Target Depth Stencil Texture Stream Buffer Stream out Memory memory programmable fixed Sampler Constant

Game Trends Record sales Visually stunning titles Half Life 2, Halo 2, Far Cry, Doom 3 Largest worlds Most detailed levels of realism Not just graphics: AI, physics, animation Longest development times Largest development costs Bulk of cost in producing content Increasing ratio of artists : programmers Record sales Visually stunning titles Half Life 2, Halo 2, Far Cry, Doom 3 Largest worlds Most detailed levels of realism Not just graphics: AI, physics, animation Longest development times Largest development costs Bulk of cost in producing content Increasing ratio of artists : programmers

Trends - Software Demand for larger worlds Demand for more detail Floppies vs CDs vs DVDs → HALO2 4.2GB HD-DVD/Blueray → 20GB Rising development cost Content creation is the bottleneck $10M content budget Art Pipeline is not scaling Amortize cost over multiple platforms Demand for larger worlds Demand for more detail Floppies vs CDs vs DVDs → HALO2 4.2GB HD-DVD/Blueray → 20GB Rising development cost Content creation is the bottleneck $10M content budget Art Pipeline is not scaling Amortize cost over multiple platforms

Trends – Hardware CPU performance Clock speed brick wall Transition to “multi-core” CPUs Games are CPU-limited Games are tuned until they are not GPU-limited PC & consoles GPU performance 2x increase every 2 years CPU performance Clock speed brick wall Transition to “multi-core” CPUs Games are CPU-limited Games are tuned until they are not GPU-limited PC & consoles GPU performance 2x increase every 2 years

Hardware Lots of Room To Grow? Enthusiast GPU today (soon): 1-2 Teraflops nm ~ M transistors ~80 Watts Increasing attention on power efficiency Enthusiast GPU today (soon): 1-2 Teraflops nm ~ M transistors ~80 Watts Increasing attention on power efficiency

Game Development Challenges Creating & Managing Detail Scalable process for content creation CPU overload Multi-core not a panacea Volume of data Effective run-time management of data Tool power & expressiveness Traditional tools not powerful enough Creating & Managing Detail Scalable process for content creation CPU overload Multi-core not a panacea Volume of data Effective run-time management of data Tool power & expressiveness Traditional tools not powerful enough

GPU as Solution How best to use performance increases? CPU bottlenecks Offload more processing to the GPU? Simulations, physics? Content Support greater detail Don’t create more work for artists Procedural detail? How best to use performance increases? CPU bottlenecks Offload more processing to the GPU? Simulations, physics? Content Support greater detail Don’t create more work for artists Procedural detail?

Procedural Detail Techniques used in film production Scene Composition & Post Processing Translucency & Antialiasing Filter & Blend Texture Detail Procedural generation Modeling & Animation Tessellation + Displacement/Normal Maps Techniques used in film production Scene Composition & Post Processing Translucency & Antialiasing Filter & Blend Texture Detail Procedural generation Modeling & Animation Tessellation + Displacement/Normal Maps

Scene Composition & Post “Post” effects becoming common place HDR, bloom, tints, grain, motion blur, depth of field …. Global atmosphere Don’t merge into individual assets Next step, on the fly Apply to individual scene elements Composite scene elements together Use to “break up” the scene – detailing “Post” effects becoming common place HDR, bloom, tints, grain, motion blur, depth of field …. Global atmosphere Don’t merge into individual assets Next step, on the fly Apply to individual scene elements Composite scene elements together Use to “break up” the scene – detailing

Composition Commonly use render target blending Porter-Duff composition More generally use pixel shader to Apply curves and filters (blurs) Limiting factors Performance Transparency & depth compositing Commonly use render target blending Porter-Duff composition More generally use pixel shader to Apply curves and filters (blurs) Limiting factors Performance Transparency & depth compositing

Transparency Traditionally render back to front with blend Sort on CPU Interactions with other algorithms (e.g., shadows) Different algorithms for Objects with feathered edges (trees) Transparent objects (windows) Volumetric effects (particle systems) Opacity versus Coverage Use alpha to represent both Can we solve with hardware? Traditionally render back to front with blend Sort on CPU Interactions with other algorithms (e.g., shadows) Different algorithms for Objects with feathered edges (trees) Transparent objects (windows) Volumetric effects (particle systems) Opacity versus Coverage Use alpha to represent both Can we solve with hardware?

Unsorted Transparency Traditionally solved with a-buffer algorithm Save all of the pixel fragments for each pixel Resolve at end of frame Arbitrary number of fragments per pixel  Lists (linked), dynamic allocation Look at “good enough” solutions Save some maximum number of fragments Fixed-function or programmable solution? Fixed-function more efficient, but extra hardware Traditionally solved with a-buffer algorithm Save all of the pixel fragments for each pixel Resolve at end of frame Arbitrary number of fragments per pixel  Lists (linked), dynamic allocation Look at “good enough” solutions Save some maximum number of fragments Fixed-function or programmable solution? Fixed-function more efficient, but extra hardware

Texture Maps Resolution increasing rapidly 4Kx4K high-resolution maps Nearing the 8Kx8K used for film Too much work to paint this much detail Don’t paint individual skin pores Digital camera capture not the answer 4Kx4K RGBA = 2x compression Resolution increasing rapidly 4Kx4K high-resolution maps Nearing the 8Kx8K used for film Too much work to paint this much detail Don’t paint individual skin pores Digital camera capture not the answer 4Kx4K RGBA = 2x compression

Texture Maps GPU virtual memory helps with size, but… Can we ever really store enough resolution? A better way? Use procedural texture techniques Shaders now expressive enough Do they have enough performance? Artists concentrate on form Algorithmically add detail “paint” algorithm ids GPU virtual memory helps with size, but… Can we ever really store enough resolution? A better way? Use procedural texture techniques Shaders now expressive enough Do they have enough performance? Artists concentrate on form Algorithmically add detail “paint” algorithm ids

Surfaces and Tessellation Increasing character complexity 2K → 5K → 15K triangles/character Increased skinning and morphing complexity k weights for each vertex Hand-tuned for animation Silhouette edges still look poor Compared to lighting and shading quality Alternative – use higher-order surfaces Beziers, Catmull-Clark subdivision, NURBS, … Many studios already model with surfaces Convert to triangles when packaging content Increasing character complexity 2K → 5K → 15K triangles/character Increased skinning and morphing complexity k weights for each vertex Hand-tuned for animation Silhouette edges still look poor Compared to lighting and shading quality Alternative – use higher-order surfaces Beziers, Catmull-Clark subdivision, NURBS, … Many studios already model with surfaces Convert to triangles when packaging content

Tessellation Advantages Animate and skin the control mesh Less computation than per-vertex Tessellate the resulting control mesh Combine with displacement mapping Use for additional geometric detail Adds detail to silhouette edges Use normal maps for fine shading detail Combine with adaptive tessellation? Animate and skin the control mesh Less computation than per-vertex Tessellate the resulting control mesh Combine with displacement mapping Use for additional geometric detail Adds detail to silhouette edges Use normal maps for fine shading detail Combine with adaptive tessellation?

Skinned Control Mesh

Summary Games increasingly more complex World size, detail Bulk of cost in producing content Content creation process not scaling Amortize cost over multiple platforms Try to reduce cost of additional detail Run-time generation of detail Improved tools for controlling detail Good areas for additional research Games increasingly more complex World size, detail Bulk of cost in producing content Content creation process not scaling Amortize cost over multiple platforms Try to reduce cost of additional detail Run-time generation of detail Improved tools for controlling detail Good areas for additional research

Games Tomorrow

Questions