Further Programming for 3D applications CE Introduction to Further Programming for 3D application Bob Hobbs Faculty of Computing, Engineering and Technology Staffordshire University

Outline Module Details Module Details What is 3D programming? What is 3D programming? Typical Processing Steps Typical Processing Steps Modelling and Rendering Modelling and Rendering Windows and Direct X Windows and Direct X Programmable Pipeline Programmable Pipeline Summary Summary

Module Details Teaching Team Teaching Team –Bob –Steve Foster Semester 215 cats Semester 215 cats 3 Hours per week 3 Hours per week –1 Hour Lecture Thu 4pm –2 Hours Practical Mon 9-11am& 2-4pm K106 Fri 1-3pm & 3-5pm K106

Module Details Course Handbook & Lecture Notes Blackboard Blackboard Assignment Details 50% assignment work 50% assignment work 50% two hour exam 50% two hour exam

Program of Study Week 01 Introduction to Module and DirectX11 Week 01 Introduction to Module and DirectX11 Week 02 Culling, Clipping and Line drawing Week 02 Culling, Clipping and Line drawing Week 03 Lighting and Shading using Programmable pipeline Week 03 Lighting and Shading using Programmable pipeline Week 04Textures & Texture Mapping Week 04Textures & Texture Mapping Week 05Blending, Stencilling and use of Geometry Shader Week 05Blending, Stencilling and use of Geometry Shader Week 06 Meshing and Instancing Week 06 Meshing and Instancing Week 07Data structures for graphics engines Week 07Data structures for graphics engines Week 08Picking, Interaction and Motion Cues Week 08Picking, Interaction and Motion Cues Week 09Tessellation and Terrain Week 09Tessellation and Terrain Week 10 Cube, Normal and Bump mapping Week 10 Cube, Normal and Bump mapping Week 11Other SFX Week 11Other SFX Week 12Review and Assessment Week 12Review and Assessment

What is 3D rendering? Object Viewer Generally deals with graphical display of 3D objects as seen by viewer Synthetic image will vary according to: viewing direction, viewer position, illumination, object properties,... Projection onto 2D surface

What is 3D Computer Graphics? 3D graphics: generation of graphical display (rendering) of 3D object(s) from specification (model(s)) 3D graphics: generation of graphical display (rendering) of 3D object(s) from specification (model(s)) Specification Modelling Graphical display Rendering such as

Wireframe polygonal modelSolid object Vertices Viewpoint Typical Processing Steps Facets Transformation Hidden surface removal Light source Shading

Object model(s) Illumination model Viewing and projection specification Graphics engine Graphical display Rendering: operations that produce view of scene projected onto view surface Modelling: numerical description of scene objects, illumination, and viewer Typical Processing Steps

15-Sep D picture Light source Surface material Typical Processing Steps xmxm zmzm ymym xmxm zmzm ymym 3D world coordinates 3D modelling coordinates 2D device coordinates 3D viewing coordinates & Culling xwxw zwzw ywyw xvxv zvzv yvyv Graphics pipeline: generation of picture involves sequence of coordinate transformations, hidden surface removal, shading/texturing and etc...

15-Sep-1511 Typical graphics pipeline Typical Processing Steps Pos., orient. of display area Geometry models Shapes World pos., orient., sizes Viewpoint Pos., orient. Field of view Projection type Viewport Object & back-face culling View mapping onto display device Generation / configuration of world Mesh operations Geometry processing: shape, world geometry, viewpoint 2D picture Lights & materials Shading, texturing,... Light pos., orient. Optical char. Pixel operations Material processing: colour, texture, transparency/occlusion, … Hidden surface removal Viewpoint

15-Sep-1512 Typical Processing Steps Loading or generation of object (shape, surface properties) model Loading or generation of object (shape, surface properties) model Model coordinate transformation: translation, rotation, projection,... Model coordinate transformation: translation, rotation, projection,... Hidden surface removal Hidden surface removal Shading Shading Display Display

15-Sep-1513 Viewport transformation 3D Models Modelling transformation 2D Display Back-face culling & Vertex lighting Viewing transformation Object culling Clipping could also be performed before viewing transformation, or during rasterisation Polygon clipping Typical Processing Steps Projection Scan conv. HSR, Texturing, & Shading Typical graphics pipeline

Modelling Human Head Model 1438 facets

Modelling Human Head Model 7258 facets

Modelling Teacher and Board Model 2074 facets

Rendering Shaded Human Head 1438 facets

Rendering Shaded Human Head 7258 facets

Rendering Shaded Teacher and Board

15-Sep-1520 Triangle scan conversion Rendering requires scan conversion Rendering requires scan conversion Scan conversion of graphics primitive: Scan conversion of graphics primitive: – point-by- point traversal of primitive along (horizontal) scan lines along (horizontal) scan lines coherence often exploited (e.g. incremental computation) coherence often exploited (e.g. incremental computation) å reduces computational cost Rendering

15-Sep-1521 x y Rasterisation of filled triangle Point grid (raster) Scan line Point / Pixel External pointRendering b Common approach: polygon interior filled line by line, one point at a time (rasterisation) linear interpolation (i.e. line equation) often used for colour shade, depth and other calculations at each pixellinear interpolation (i.e. line equation) often used for colour shade, depth and other calculations at each pixel

15-Sep-1522 General procedure for filled triangles: find intersections of scan line with polygon edge find intersections of scan line with polygon edge – use equation of line joining two vertices: interpolation edge coherence, hence use incremental computation edge coherence, hence use incremental computation fill / shade / texture interior points between intersection pairs along scan line fill / shade / texture interior points between intersection pairs along scan line Rendering

15-Sep-1523 General procedure for filled triangles: find intersections of scan line with polygon edge find intersections of scan line with polygon edge – use equation of line joining two vertices: interpolation edge coherence, hence use incremental computation edge coherence, hence use incremental computation fill / shade / texture interior points between intersection pairs along scan line fill / shade / texture interior points between intersection pairs along scan line Rendering

Windows Programming Rendering takes place in a window Rendering takes place in a window Application must generate window in which to render graphics Application must generate window in which to render graphics IP3DA used C and GLUT to handle windows. OpenGL can run in GLUT or other windowing framework including MS windows IP3DA used C and GLUT to handle windows. OpenGL can run in GLUT or other windowing framework including MS windows DirectX is integrated into windows programming API DirectX is integrated into windows programming API

What is a “window?” The most obvious windows are the application windows on your desktop. The most obvious windows are the application windows on your desktop. The term “window” also refers to any child window or control (such as a button) within a window. The term “window” also refers to any child window or control (such as a button) within a window. Windows are organized in a hierarchy. Windows are organized in a hierarchy.

Example - Calculator

Event Driven Programming Windows programming is event driven. Windows programming is event driven. An event driven program sits and waits for an event to process. An event driven program sits and waits for an event to process. Events include moving or sizing a window, clicking the mouse button over a window, or typing keys on the keyboard. Events include moving or sizing a window, clicking the mouse button over a window, or typing keys on the keyboard. Windows programs are notified of events by messages sent from the OS. Windows programs are notified of events by messages sent from the OS.

Win32 API We will use the Win32 Platform API to write a windows program. We will use the Win32 Platform API to write a windows program. –Typically more tedious, but we only need to create one window. –Faster and Smaller executable. –Written in C++ Other options include: Other options include: –MFC, Qt, wxWindows, SDL,.NET Forms

WinMain Instead of main, windows programs use WinMain Instead of main, windows programs use WinMain WinMain has different parameters: WinMain has different parameters: – HINSTANCE hInstance – a handle to the program – HINSTANCE hPrevInstance – no longer used. – LPSTR lpCmdLine – unparsed command line. Doesn’t include executable’s filename. – int nCmdShow – Specifies how the window should be initially shown (ie. Minimized, Maximized, Normal) WinMain must have a WINAPI modifier. WinMain must have a WINAPI modifier. – int WINAPI WinMain (…) WinMain still returns an integer like main. WinMain still returns an integer like main.

Headers To use the Win32 API we need to #include windows.h To use the Win32 API we need to #include windows.h If we #define WIN32_LEAN_AND_MEAN before #including windows.h, the compiler will skip compiling the more rarely used windows code. If we #define WIN32_LEAN_AND_MEAN before #including windows.h, the compiler will skip compiling the more rarely used windows code.

Windows Program Outline Create Window Class Create Window Class Create Window Create Window Show the Window Show the Window Enter Message Loop Enter Message Loop

Window Classes A window class acts as a template for window creation. A window class acts as a template for window creation. Window classes are not classes in the C++ sense. Window classes are not classes in the C++ sense. We can set such properties as: We can set such properties as: –Name of the window class – important for identifying it later! –Window style – How it looks and its default behaviors. –Window procedure – a pointer to function that handles messages for a window. –Default cursor, icon, and menu.

Creating a Window Class Fill out a WNDCLASS or WNDCLASSEX structure. Fill out a WNDCLASS or WNDCLASSEX structure. Use the structure to register the class with a call to RegisterClass or RegisterClassEx, respectively. Use the structure to register the class with a call to RegisterClass or RegisterClassEx, respectively.

Create the Window Create a Window with a call to CreateWindow or CreateWindowEx. Create a Window with a call to CreateWindow or CreateWindowEx. Some of the parameters you will need to enter include: Some of the parameters you will need to enter include: –The name of a window class. –The name of the window. –A handle to a parent window. ( NULL if no parent) –Initial size and position of the window. CreateWindow and CreateWindowEx will return a window handle ( HWND ) if successful or NULL if failed. CreateWindow and CreateWindowEx will return a window handle ( HWND ) if successful or NULL if failed.

Showing the Window When we create a window it starts off hidden. When we create a window it starts off hidden. To show the window you must call ShowWindow (hWnd, nCmdShow) To show the window you must call ShowWindow (hWnd, nCmdShow) – hWnd – returned from CreateWindow / Ex. – nCmdShow – parameter from WinMain. –This will send a message to the window telling it to show. Call UpdateWindow (hWnd) to force the window to process the show message immediately. Call UpdateWindow (hWnd) to force the window to process the show message immediately.

Message Loops A message loop is needed to forward messages to their windows. A message loop is needed to forward messages to their windows. There are different ways to make a message loop but here is the most common one: There are different ways to make a message loop but here is the most common one: MSG msg = {0}; // returns FALSE when message is WM_QUIT while (GetMessage (&msg, NULL, 0, 0)) { TranslateMessage (&msg); // translate keyboard messages. DispatchMessage (&msg); // send message to its window. }

Message Loop (Cont’d) The loop uses: The loop uses: –A MSG structure – Holds message information. – GetMessage – Retrieves the next message. Waits for a message if the message queue is empty. – TranslateMessage – Translate some keyboard messages. – DispatchMessage – Dispatches the message to the appropriate windows procedure.

Windows Procedure Every window has an associate procedure for handling messages. Every window has an associate procedure for handling messages. Windows procedures have the following format: Windows procedures have the following format: – LRESULT CALLBACK ProcName (HWND hwnd, UINT msg, WPARAM wp, LPARAM lp);

Window Procedure Example Here is an example procedure: Here is an example procedure: LRESULT CALLBACK WndProc (HWND hwnd, UINT msg, WPARAM wp, LPARAM lp) { switch (msg) { case WM_DESTROY: PostQuitMessage (0); return 0; } return DefWindowProc (hwnd, msg, wp, lp); }

Window Procedure Parameters The procedure has 4 parameters: The procedure has 4 parameters: – HWND hwnd – handle to the window receiving the message. This is useful when multiple windows share the same procedure. You can think of it sort of like a this pointer. – UINT msg – The message. – WPARAM wp and LPARAM lp – These are parameters to the message and may represent different things.

Window Procedure Return When the procedure completely handles a message, it should return 0. When the procedure completely handles a message, it should return 0. The procedure should let the OS handle all other messages by calling DefWindowProc and then return its result. The procedure should let the OS handle all other messages by calling DefWindowProc and then return its result.

WM_DESTROY and WM_QUIT Initially we only want to handle one message: WM_DESTROY. Initially we only want to handle one message: WM_DESTROY. –This message is called when a window is closed. –Note that just because the window is destroyed, the program will still run. –To end the program we send WM_QUIT by calling PostQuitMessage.

WM_* Other messages you may want to handle include: Other messages you may want to handle include: – WM_CREATE – WM_SIZE – WM_LBUTTONDOWN – WM_LBUTTONUP – WM_MOUSEMOVE – WM_KEYDOWN – WM_KEYUP – WM_PAINT Check MSDN literature for information about these messages: Check MSDN literature for information about these messages:

Direct X The window has to obtain a handle to a Direct 3D and swap chain to render graphics objects in the windows object created. The window has to obtain a handle to a Direct 3D and swap chain to render graphics objects in the windows object created. In DirectX 11 this is based on the Programmable Pipeline. In DirectX 11 this is based on the Programmable Pipeline.

15-Sep-1545 Shader: custom program segment inserted into graphics pipeline and executed on GPU Shader: custom program segment inserted into graphics pipeline and executed on GPU – replaces specific parts of fixed-function pipeline comprises instructions that operate solely on: comprises instructions that operate solely on: – vertex data (vertex shader) – or pixel data (pixel shader) – geometric primitive data (geometry shader) Introduction to PPL

15-Sep-1546 Operations Vertex shader: vertex transformation and lighting calculations Vertex shader: vertex transformation and lighting calculations Pixel shader: e.g. pixel lighting, texturing, and image processing calculations Pixel shader: e.g. pixel lighting, texturing, and image processing calculations Geometry shader: primitive processing and creation Geometry shader: primitive processing and creation Introduction to PPL

15-Sep-1547 Shaders offer code development flexibility and processing speed: Shaders offer code development flexibility and processing speed: – flexibility: shader implements customised vertex, polygon, or pixel processing – speed: shader runs on fast specialised chips many current graphics cards support shaders many current graphics cards support shaders – programmable hardware pipeline on most modern GPUs Shaders often used for better visual quality in real time Shaders often used for better visual quality in real time Introduction to PPL

15-Sep-1548 Feature of DirectX Graphics and OpenGL: Feature of DirectX Graphics and OpenGL: – support for programmable pipeline: API provides high-level or assembly language interface to graphics processing hardware API provides high-level or assembly language interface to graphics processing hardware specialised programming language for vertex and pixel processing specialised programming language for vertex and pixel processing – developer can program special code for real-time graphics applications  e.g. rich graphical effects such as simulation of cloth, hair or fur Programmable Graphics Pipeline

15-Sep-1549 Graphics processing unit (GPU) Graphics cards used to be fixed-function: Graphics cards used to be fixed-function: – graphics processors hard-wired for specific functions graphics limited by implementation of device driver and underlying hardware graphics limited by implementation of device driver and underlying hardware – fixed-function constraints overcome by running code on: general-purpose CPU (has limited support for graphics, hence slow) general-purpose CPU (has limited support for graphics, hence slow) programmable graphics processor(s) known as GPU (fast): programmable graphics processor(s) known as GPU (fast): – code downloaded onto GPU during rendering Programmable Graphics Hardware

15-Sep-1550 Layered architecture of fixed-function graphics system D3D D3DX Device driver interface ApplicationHAL Hardware Programmable Graphics Hardware DirectX Graphics

15-Sep-1551 Graphics processing unit (GPU) Special purpose microprocessor(s) Special purpose microprocessor(s) – multiple programmable processing units for concurrent processing on single chip optimised for graphics processing, e.g. vector / matrix manipulations optimised for graphics processing, e.g. vector / matrix manipulations – used in computer graphics (accelerator) card for manipulation and display of graphics data for manipulation and display of graphics data – can perform some tasks of pipeline  evolving hardware: capability improving with time Programmable Graphics Hardware

15-Sep-1552 ATI Radeon HD 5670 (supports DirectX 11) Source: Don Woligroski, The Radeon HD 5670 Architecture, Tom's Hardware, January 14, 2010 ( Programmable Graphics Hardware

15-Sep-1553 DirectX 9 graphics pipeline Programmable Graphics Hardware

15-Sep-1554 Application Vertex Processor Fragment Processor Assembly & Rasterization Frame buffer Operations Frame buffer GPU CPU Textures GPU programming model Source: Bill Mark, “NVIDIA Programmable Graphics Technology”, SIGGRAPH 2002 course) (Source: Bill Mark, “NVIDIA Programmable Graphics Technology”, SIGGRAPH 2002 course) Programmable Graphics Hardware

15-Sep-1555 DirectX 10 graphics pipeline ( see ) Programmable Graphics Hardware

15-Sep-1556 Input assembler: fixed functions to Input assembler: fixed functions to – read vertices from memory, – assemble them into geometry primitives, – create other data needed for pipeline processing e.g. vertex or primitive identifiers (shown flowing along dotted line in previous figure) to allow identifier-specific processing e.g. vertex or primitive identifiers (shown flowing along dotted line in previous figure) to allow identifier-specific processing Vertex shader: Vertex shader: – processes vertices – receives one vertex and outputs one vertex Programmable Graphics Hardware

15-Sep-1557 Geometry shader: Geometry shader: – processes entire primitives primitive consists of: 3 vertices for triangle, 2 vertices for line, or 1 vertex for point primitive consists of: 3 vertices for triangle, 2 vertices for line, or 1 vertex for point primitive data can also include vertex data for edge-adjacent primitives (e.g. 3 vertices for triangle or 2vertices for line) primitive data can also include vertex data for edge-adjacent primitives (e.g. 3 vertices for triangle or 2vertices for line) – receives one primitive and outputs zero, one, or multiple primitives Stream output: Stream output: – concatenates (streams out) primitives output by geometry shader to output buffers (stream output and geometry shader are programmed together) Programmable Graphics Hardware

15-Sep-1558 Rasterizer: Rasterizer: – clips primitives, performs perspective divide on primitives, implements viewport and scissor selection, and etc... Pixel shader: Pixel shader: – receives one pixel and outputs one pixel at same position or no pixel Output merger: Output merger: – performs fixed function render-target blend, depth and stencil operations Programmable Graphics Hardware

15-Sep-1559 Vertex shader Vertex shader bypasses fixed-function ‘transformation and lighting’ pipeline segment Vertex shader bypasses fixed-function ‘transformation and lighting’ pipeline segment – processes vertex runs on programmable vertex processor runs on programmable vertex processor example functionality: vertex displacements for mesh deformation (e.g. for fish-eye lens effect, flag animation (for ripple)) example functionality: vertex displacements for mesh deformation (e.g. for fish-eye lens effect, flag animation (for ripple)) Shaders

15-Sep-1560 Vertex shader Vertex shader outputs attributes that can be interpolated during rasterisation Vertex shader outputs attributes that can be interpolated during rasterisation – pixel / fragment processing done in subsequent stages of graphics pipeline e.g. light reflected at vertex can be calculated in vertex shader for use in pixel shader e.g. light reflected at vertex can be calculated in vertex shader for use in pixel shader Shaders

15-Sep-1561 Geometry shader Allows manipulation of mesh primitive Allows manipulation of mesh primitive – e.g. single vertex, line segment (two vertices), or triangle (three vertices) benefits: per-primitive processing (e.g. calculation of triangle normal, creation of new primitives (e.g. modified copies of single triangle) benefits: per-primitive processing (e.g. calculation of triangle normal, creation of new primitives (e.g. modified copies of single triangle) – can be used for generating intricate surface details such as fur – geometry shader sits between vertex shader and pixel shader in graphics pipeline Shaders

15-Sep-1562 Pixel shader Also known as fragment shader Also known as fragment shader – processes pixel / fragment runs on pixel / fragment processor during rasterisation runs on pixel / fragment processor during rasterisation – input to pixel / fragment processor is typically result of interpolation of triangle vertex data output by vertex shader – typical output is colour of current pixel example functionality: shading / texturing effects, per-pixel lighting example functionality: shading / texturing effects, per-pixel lighting – e.g. for accurate lighting models, simulation of complex surface properties or natural phenomena Shaders

15-Sep-1563 Definition: fragment corresponds to pixel Definition: fragment corresponds to pixel – i.e. fragment processor = pixel processor Fragment Fragment – has attributes such as colour, depth, texture coordinates – fragments are generated by rasterisation Shaders

15-Sep-1564 Shader programming requires language specialised for graphics Shader programming requires language specialised for graphics – needs graphics-specific data types (e.g. vector, matrix) and operations (e.g. dot product, matrix multiply) – language often tied to specific graphics API Actual executable shader code is machine code Actual executable shader code is machine code – but high-level language can be compiled to machine code compilers available compilers available Shader Languages

15-Sep-1565 High-level languages historically preceded by various low-level languages such as Microsoft’s ASM High-level languages historically preceded by various low-level languages such as Microsoft’s ASM – low-level programming has: low productivity low productivity limited portability limited portability low code readability / maintainability low code readability / maintainability Shader Languages

15-Sep-1566 Variety of high-level languages, e.g. Variety of high-level languages, e.g. – ‘C’-like languages jointly developed by Microsoft and NVidia (essentially same language but different brand names): High-Level Shader Language (HLSL) (Microsoft) High-Level Shader Language (HLSL) (Microsoft) – for use with DirectX: HLSL compiler is part of D3DX library  the only language supported intrinsically by XNA and DirectX Cg (NVIDIA) Cg (NVIDIA) – support for DirectX Graphics and OpenGL – OpenGL shading language (GLSL or glslang) for use with OpenGL. for use with OpenGL. Shader Languages

15-Sep-1567 HLSL Sample intrinsic data types: Sample intrinsic data types: – vector types: contain from 1 to 4 components (arranged as single row or column), e.g. int1: vector containing 1 int; float3: vector containing 3 floats; double4: vector containing 4 doubles int1: vector containing 1 int; float3: vector containing 3 floats; double4: vector containing 4 doubles – matrix types: contain rows and columns of same- type data, e.g. int2x1: integer matrix with 2 rows and 1 column; float4x4: float matrix with 4 rows and 4 columns int2x1: integer matrix with 2 rows and 1 column; float4x4: float matrix with 4 rows and 4 columns Shader Languages

15-Sep-1568 HLSL Sample intrinsic functions: Sample intrinsic functions: – dot(a, b): returns dot product of vectors a and b – mul(A, B): returns product of matrices A and B – reflect(i, n): returns reflection vector given incident ray direction i and surface normal n – round(x): returns x rounded to nearest integer Shader Languages

15-Sep-1569 Sample code: simple HLSL shaders Example (from DirectX Graphics documentation ( us/directx9_c/HLSL_Shaders.asp)) /* Vertex shader Functionality: transforms vertex position from object space to projection space transforms vertex position from object space to projection space assigns per-vertex colour from vertex input */ assigns per-vertex colour from vertex input */ // Note: ‘: POSITION’ and ‘: COLOR0’ are semantics that tell compiler usage of variable float4x4 mWorldViewProj; // World * View * Projection transformation // (global variable set by application) // shader output structure struct VS_OUTPUT { float4 Position : POSITION; // vertex position float4 Position : POSITION; // vertex position float4 Diffuse : COLOR0; // vertex diffuse color float4 Diffuse : COLOR0; // vertex diffuse color}; Shader Languages

15-Sep-1570 Sample code: simple HLSL shaders Example (ctd.) VS_OUTPUT Vertex_Shader_Transform( in float4 vPosition : POSITION, in float4 vColor : COLOR0) { VS_OUTPUT Output; VS_OUTPUT Output; // Transform the vertex into projection space. // Transform the vertex into projection space. Output.Position = mul( vPosition, mWorldViewProj ); Output.Position = mul( vPosition, mWorldViewProj ); // Set the diffuse color for output // Set the diffuse color for output Output.Diffuse = vColor; Output.Diffuse = vColor; return Output; return Output;} Shader Languages

15-Sep-1571 Sample code: simple HLSL shaders Example (ctd.) /* Pixel shader */ Functionality: copies interpolated vertex colour data, and outputs pixel colour float4 Pixel_Shader( VS_OUTPUT in ) : COLOR0 { float4 color = in. Diffuse;// copy input colour float4 color = in. Diffuse;// copy input colour /* float4 color = in.Color; */// note error in original example code return color; } Shader Languages Input parameter Output semantic

15-Sep-1572 Suggested Reading (Microsoft DirectX Graphics Documentation) (Microsoft DirectX Graphics Documentation) (Cg homepage) (Cg homepage)

15-Sep-1573 Programmable hardware pipeline on GPU Programmable hardware pipeline on GPU – concurrent processing on single chip Shaders offer code development flexibility and processing speed Shaders offer code development flexibility and processing speed Shader: program segment inserted into graphics pipeline and executed on GPU Shader: program segment inserted into graphics pipeline and executed on GPU – replaces section of fixed-function pipeline vertex shader, pixel shader, geometry shader vertex shader, pixel shader, geometry shader Variety of shader languages Variety of shader languages Summary