1. computer graphics & visualization The programmable (D3D 10) Pipeline

2. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Pixel Stage Vertex Stage User / Driver First: the fixed function pipeline Transform & Lighting Rasterizer Texturing Blending/Ops Texture 3 Texture 2 Texture 1 Texture 0 Vertex Stream Fragment Stream

3. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Fixed Function Pipeline Properties States can be changedStates can be changed States out of a fixed number of alternativesStates out of a fixed number of alternatives Same principal operations always performed on stream elementsSame principal operations always performed on stream elements Can only discard certain operationsCan only discard certain operations Action on stream elements (vertices/polygons/fragments) can not be programmedAction on stream elements (vertices/polygons/fragments) can not be programmed

4. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Fixed Function Pipeline To In the following: To In the following: – Non-OpenGL effects using fixed function pipeline – Multipass Combine pixel results of multiple rendering passes Combine pixel results of multiple rendering passes – Pre-computed texture maps Pre-compute results of certain non-standard operations and store in texture maps Pre-compute results of certain non-standard operations and store in texture maps

5. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Advanced shading The Phong lighting model – – N: surface normal – L: light source direction – R: reflection vector – V: direction to the viewpoint – n: materials specular reflection exponent Defines the highlights falloff from the direction of reflection Defines the highlights falloff from the direction of reflection – Specular reflections show the color of the light source Not valid for all materials Not valid for all materials

6. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Advanced shading The Phong/Blinn lighting model – – The Halfway vector H points into the direction of a surfaces´ normal that reflects light into V – If light source and view point are considered to be at infinity, H is a constant – The angle between R and V is twice the angle between N and H if all above vectors are complanar Highlights appear sharper in the Phong model Highlights appear sharper in the Phong model

7. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Advanced shading In OpenGL, per-vertex Phong lighting is employed to simulate specular reflections In OpenGL, per-vertex Phong lighting is employed to simulate specular reflections But, during scan-conversion vertex color is interpolated But, during scan-conversion vertex color is interpolated – OpenGL uses Gouraud shading instead of Phong shading – Texture color affects highlight color – Sharp highlights might be missed if the surface is tesselated coarsely

8. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Advanced shading To get separate highlight color highlights have to be added to the diffusely lit textured surface To get separate highlight color highlights have to be added to the diffusely lit textured surface – OpenGL computes C t (C d + C s ) – Instead Compute C d and C s separately Compute C d and C s separately Modulate C d with C t Modulate C d with C t Add result to C s Add result to C s – Solution can be obtained by  -blending

9. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Advanced shading Multipass technique to get separate highlight color Multipass technique to get separate highlight color – Render the textured object without specular light  C t C d – Render the untextured object with specular light only  C s – Combine the results in the frame buffer by additive blend glBlendFunc(GL_ONE, GL_ONE) glBlendFunc(GL_ONE, GL_ONE) glBlendEquation(GL_ADD) glBlendEquation(GL_ADD)

10. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Advanced shading Textured with diffuse light Untextured with specular light Combined images In the original image the highlight can hardly be seen

11. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Advanced shading Using OpenGL 1.2 Using OpenGL 1.2 glLightModel(GL_LIGHT_MODEL_COLOR_CONTROL, GL_SEPARATE_SPECULAR_COLOR);

12. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group A few words on OpenGL Extensions The concept of extensions The concept of extensions – Naming conventions ARB vs. cross vendor vs. vendor specific extensions ARB vs. cross vendor vs. vendor specific extensions – Querying: compile time vs. run time compile time vs. run time

13. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group OpenGL Extensions Mechanism for providing access to non- standard features of the hardware Mechanism for providing access to non- standard features of the hardware Every vendor may define own extensions without having to ask anybody else Every vendor may define own extensions without having to ask anybody else Multiple vendors may cooperate to define more common extensions to ensure compatibility across platforms Multiple vendors may cooperate to define more common extensions to ensure compatibility across platforms

14. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Naming Conventions for Extensions ARB: Architecture Review Board ARB: Architecture Review Board – Ext. that is likely to become part of core OpenGL in the future EXT: Multi-vendor extensions EXT: Multi-vendor extensions – Ext. that multiple vendors have agreed upon NV, ATI, SGI, SUN...: vendor-specific ext. NV, ATI, SGI, SUN...: vendor-specific ext. – Ext. supported only by one vendor, sometimes experimental and not available on all systems of that vendor (e.g. NVX, SGIS, SGIX)

15. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Dealing with Extensions Since all extensions are optional, applications cannot rely on their presence Since all extensions are optional, applications cannot rely on their presence Have to check both at compile time and at run time! Have to check both at compile time and at run time! Compile time check: Compile time check: #ifdef GL_ARB_multitexture // use multitexture ext. in here...#endif

16. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Dealing with Extensions (2) Run time check: const Glubyte *extstring = glGetString(GL_EXTENSIONS); strcpy(exts, extstring); extsupported= 0; next= strtok(exts, “ “); while(next) { if(!strcmp(next,“GL_ARB_multitexture”)) extsupported= 1; break; next= strtok(exts,NULL); }

17. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Advanced shading How to get sharp highlights without adaptive refinement (tesselation) – Using the Blinn/Phong model specular highlights depend on L,N and V – By fixing L and V highlights depend only on N – N can be used to index into a texture map in which the reflection for R = f(N,L) are stored – Texture map looks like reflection from chrome hemisphere

18. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Texture coordinates computed from the normal vector Texture coordinates map into sphere map texture Texel contain pre- computed Phong lighting Advanced shading Spherical texture maps

19. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Advanced shading Phong shading with highlight texture – Render the object with diffuse light  C t C d – Render the unlit, untextured white object and modulate it with the highlight texture  C ht – Combine the results in the frame buffer by additive blend – … or use multi-textures in one rendering pass

20. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Advanced shading Phong shading with highlight texture Textured with diffuse light White object modulated by highlight texture Blended images

21. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group No more “Hacks” … The Programmable pipeline Direct3D 9 / OpenGL 2.0 Model – Still used in many existing games – Supported by any graphics card of the last 5 years – Supported by every mayor OS (DX only on Windows) Direct3D 10 / OpenGL 2.0 + Ext / OpenGL 3.0 – Supported only by the latest GPUs NVIDIA 8xxx or ATI Radeon 2xxxx – OpenGL 3.0 not out yet  must be used via extensions – Direct3D 10.0 only supported by Windows Vista (Direct3D 10.1 requires Vista SP1)

22. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group The Direct3D 9.0c Pipeline Input Data Input Assembler Stage (IA) Vertex Shader Stage (VS) Rasterizer Stage (RS) Pixel Shader Stage (PS) Output Merger Stage (OM) Output Data Memory Resources: Buffers, Textures Buffer Texture, Constants Texture, Constants States Texture

23. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Features of the D3D 9.0c pipeline Programmable vertex and pixel shader with limited shader program length (and many other limitations) Vertex and pixel shader texture fetch capabilities (some limitations apply) Configurable output merger (Blending/Ops/Tests) Fixed function pipeline vertex and/or pixel shader can be used Floating point through the entire pipeline (except for the blending)

24. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Benefits of the programmable Pipeline No more “hacks”  more sophisticated “hacks” Straight forward program flow without the necessity of cryptic configuration changes Decent knowledge of the transformation and lighting required to program the pipeline  everything has to be done “by hand” Shaders can be programmed in a high level language HLSL/GLSL

25. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Shaders: a first look (HLSL) “Basic HLSL”-Sample from the DirectX SDK: //-------------------------------------------------------------------------------------- // Global variables //-------------------------------------------------------------------------------------- float4 g_MaterialAmbientColor; // Material's ambient color float4 g_MaterialDiffuseColor; // Material's diffuse color int g_nNumLights; float3 g_LightDir[3]; // Light's direction in world space float4 g_LightDiffuse[3]; // Light's diffuse color float4 g_LightAmbient; // Light's ambient color texture g_MeshTexture; // Color texture for mesh float g_fTime; // App's time in seconds float4x4 g_mWorld; // World matrix for object float4x4 g_mWorldViewProjection; // World * View * Projection matrix //-------------------------------------------------------------------------------------- // Texture samplers //-------------------------------------------------------------------------------------- sampler MeshTextureSampler = sampler_state { Texture = ; MipFilter = LINEAR; MinFilter = LINEAR; MagFilter = LINEAR; }; //-------------------------------------------------------------------------------------- // Vertex shader output structure //-------------------------------------------------------------------------------------- struct VS_OUTPUT { float4 Position : POSITION; // vertex position float4 Diffuse : COLOR0; // vertex diffuse color (note that COLOR0 is clamped from 0..1) float2 TextureUV : TEXCOORD0; // vertex texture coords }; //-------------------------------------------------------------------------------------- // This shader computes standard transform and lighting //-------------------------------------------------------------------------------------- VS_OUTPUT RenderSceneVS( float4 vPos : POSITION, float3 vNormal : NORMAL, float2 vTexCoord0 : TEXCOORD0, uniform int nNumLights, uniform bool bTexture, uniform bool bAnimate ) { VS_OUTPUT Output; float3 vNormalWorldSpace; float4 vAnimatedPos = vPos; // Animation the vertex based on time and the vertex's object space position if( bAnimate ) vAnimatedPos += float4(vNormal, 0) * (sin(g_fTime+5.5)+0.5)*5; // Transform the position from object space to homogeneous projection space Output.Position = mul(vAnimatedPos, g_mWorldViewProjection); // Transform the normal from object space to world space vNormalWorldSpace = normalize(mul(vNormal, (float3x3)g_mWorld)); // normal (world space) // Compute simple directional lighting equation float3 vTotalLightDiffuse = float3(0,0,0); for(int i=0; i<nNumLights; i++ ) vTotalLightDiffuse += g_LightDiffuse[i] * max(0,dot(vNormalWorldSpace, g_LightDir[i])); Output.Diffuse.rgb = g_MaterialDiffuseColor * vTotalLightDiffuse + g_MaterialAmbientColor * g_LightAmbient; Output.Diffuse.a = 1.0f; // Just copy the texture coordinate through if( bTexture ) Output.TextureUV = vTexCoord0; else Output.TextureUV = 0; return Output; } struct PS_OUTPUT { float4 RGBColor : COLOR0; // Pixel color }; PS_OUTPUT RenderScenePS( VS_OUTPUT In, uniform bool bTexture ) { PS_OUTPUT Output; // Lookup mesh texture and modulate it with diffuse if( bTexture ) Output.RGBColor = tex2D(MeshTextureSampler, In.TextureUV) * In.Diffuse; else Output.RGBColor = In.Diffuse; return Output; } technique RenderSceneWithTexture1Light { pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 1, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true ); // trivial pixel shader (could use FF instead if desired) } technique RenderSceneWithTexture2Light { pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 2, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true ); // trivial pixel shader (could use FF instead if desired) } technique RenderSceneWithTexture3Light { pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 3, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true ); // trivial pixel shader (could use FF instead if desired) } technique RenderSceneNoTexture { pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 1, false, false ); PixelShader = compile ps_2_0 RenderScenePS( false ); // trivial pixel shader (could use FF instead if desired) } }

26. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Vertex Shaders: a closer look VS_OUTPUT RenderSceneVS( float4 vPos : POSITION, float3 vNormal : NORMAL, float2 vTexCoord0 : TEXCOORD0, uniform int nNumLights, uniform bool bTexture, uniform bool bAnimate ) { VS_OUTPUT Output; float3 vNormalWorldSpace; float4 vAnimatedPos = vPos; // Animation the vertex based on time and the vertex's object space position if( bAnimate ) vAnimatedPos += float4(vNormal, 0) * (sin(g_fTime+5.5)+0.5)*5; // Transform the position from object space to homogeneous projection space Output.Position = mul(vAnimatedPos, g_mWorldViewProjection); // Transform the normal from object space to world space vNormalWorldSpace = normalize(mul(vNormal, (float3x3)g_mWorld)); // normal (world space) // Compute simple directional lighting equation float3 vTotalLightDiffuse = float3(0,0,0); for(int i=0; i<nNumLights; i++ ) vTotalLightDiffuse += g_LightDiffuse[i] * max(0,dot(vNormalWorldSpace, g_LightDir[i])); Output.Diffuse.rgb = g_MaterialDiffuseColor * vTotalLightDiffuse + g_MaterialAmbientColor * g_LightAmbient; Output.Diffuse.a = 1.0f; // Just copy the texture coordinate through if( bTexture ) Output.TextureUV = vTexCoord0; else Output.TextureUV = 0; return Output; }

27. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Pixel Shaders: a closer look PS_OUTPUT RenderScenePS( VS_OUTPUT In, uniform bool bTexture ) { PS_OUTPUT Output; // Lookup mesh texture and modulate it with diffuse if( bTexture ) Output.RGBColor = tex2D(MeshTextureSampler, In.TextureUV) * In.Diffuse; else Output.RGBColor = In.Diffuse; return Output; }

28. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Putting it all together: Effect Files Store many vertex and pixel shaders as well as pipeline stages in a single file Also Store variables, textures, and samplers Allow for grouping of shaders into techniques and passes Can be accessed easily from the main program Can be created/modified easily from a content creation tool (AMD RenderMonkey/Nvidia FX Composer)AMD RenderMonkeyNvidia FX Composer

29. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group FX Composer Showcase

30. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Effects in the “Basic HLSL”-Sample //-------------------------------------------------------------------------------------- // Global variables //-------------------------------------------------------------------------------------- float4 g_MaterialAmbientColor; // Material's ambient color float4 g_MaterialDiffuseColor; // Material's diffuse color int g_nNumLights; float3 g_LightDir[3]; // Light's direction in world space float4 g_LightDiffuse[3]; // Light's diffuse color float4 g_LightAmbient; // Light's ambient color float g_fTime; // App's time in seconds float4x4 g_mWorld; // World matrix for object float4x4 g_mWorldViewProjection; // World * View * Projection matrix

31. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Effects in the “Basic HLSL”-Sample texture g_MeshTexture; // Color texture for mesh //-------------------------------------------------------------------------------------- // Texture samplers //-------------------------------------------------------------------------------------- sampler MeshTextureSampler = sampler_state { Texture = ; MipFilter = LINEAR; MinFilter = LINEAR; MagFilter = LINEAR; };

32. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Effects in the “Basic HLSL”-Sample //-------------------------------------------------------------------------------------- // Renders scene to render target //-------------------------------------------------------------------------------------- technique RenderSceneWithTexture1Light { pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 1, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true ); // trivial pixel shader (could use FF instead if desired) }

33. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Effects in the “Basic HLSL”-Sample technique RenderSceneWithTexture2Light { pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 2, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true ); // trivial pixel shader (could use FF instead if desired) }

34. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group The DX 10.0 Pipeline Input Data Input Assembler Stage (IA) Vertex Shader Stage (VS) Geometry Shader Stage (GS) Stream Output Stage (SO) Rasterizer Stage (RS) Pixel Shader Stage (PS) Output Merger Stage (OM) Output Data Memory Resources: Buffers, Textures, Constant Buffers Buffer Texture, Constant Buffer Buffer States Buffer, Texture,Constant Buffer

35. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Features of the D3D 10.1 pipeline Programmable vertex, pixel, and geometry shader with practically unlimited shader program length Vertex, pixel, and geometry shader texture fetch capabilities Configurable output merger (Blending/Ops/Tests) Fixed function pipeline vertex and/or pixel shader can NOT be used anymore Floating point through the entire pipeline including blending API Support for double precision (first hardware supporting double expected to ship in late 2007)

36. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group “Basic HLSL” D3D 10 /-------------------------------------------------------------------------------------- // This shader outputs the pixel's color by modulating the texture's // color with diffuse material color //-------------------------------------------------------------------------------------- PS_OUTPUT RenderScenePS( VS_OUTPUT In, uniform bool bTexture ) { PS_OUTPUT Output; // Lookup mesh texture and modulate it with diffuse if( bTexture ) Output.RGBColor = g_MeshTexture.Sample(MeshTextureSampler, In.TextureUV) * In.Diffuse; else Output.RGBColor = In.Diffuse; return Output; } //-------------------------------------------------------------------------------------- // Renders scene to render target using D3D10 Techniques //-------------------------------------------------------------------------------------- technique10 RenderSceneWithTexture1Light { pass P0 { SetVertexShader( CompileShader( vs_4_0, RenderSceneVS( 1, true, true ) ) ); SetGeometryShader( NULL ); SetPixelShader( CompileShader( ps_4_0, RenderScenePS( true ) ) ); SetDepthStencilState( EnableDepth, 0 ); }

37. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group “Basic HLSL” D3D 10 /-------------------------------------------------------------------------------------- // This shader outputs the pixel's color by modulating the texture's // color with diffuse material color //-------------------------------------------------------------------------------------- PS_OUTPUT RenderScenePS( VS_OUTPUT In, uniform bool bTexture ) { PS_OUTPUT Output; // Lookup mesh texture and modulate it with diffuse if( bTexture ) Output.RGBColor = g_MeshTexture.Sample(MeshTextureSampler, In.TextureUV) * In.Diffuse; else Output.RGBColor = In.Diffuse; return Output; } //-------------------------------------------------------------------------------------- // Renders scene to render target using D3D10 Techniques //-------------------------------------------------------------------------------------- technique10 RenderSceneWithTexture1Light { pass P0 { SetVertexShader( CompileShader( vs_4_0, RenderSceneVS( 1, true, true ) ) ); SetGeometryShader( NULL ); SetPixelShader( CompileShader( ps_4_0, RenderScenePS( true ) ) ); SetDepthStencilState( EnableDepth, 0 ); }

38. computer graphics & visualization Image Synthesis – WS 07/08 Dr. Jens Krüger – Computer Graphics and Visualization Group Coming up