Klas Skogmar, Lund Institute of Technology Real-time Video Effects Using Programmable Graphics Cards Master of Science Thesis Klas Skogmar

Klas Skogmar, Lund Institute of Technology Introduction Graphics cards have much computing power but are only used by 3D applications Video and image editing programs often needs to perform per pixel operations that are ideal for graphics cards

Klas Skogmar, Lund Institute of Technology Problems How can graphics cards be used for displaying and altering video? What kinds of effects are suitable for a 3D environment? Is it possible to speed up effects by using the graphics card? How are the transfer rates affecting the system? How much disc space is required and how does this affect the system?

Klas Skogmar, Lund Institute of Technology Programmable graphics cards Graphics cards can do massive parallel calculations 3D developers required more flexibility 3D card producers introduced “Vertex shaders” and “Pixel shaders” Makes it possible to program graphics cards in a similar way to processors

Klas Skogmar, Lund Institute of Technology Effects on graphics cards Color correction Masking Color keying Compositing Transitions Painting 3D effects

Klas Skogmar, Lund Institute of Technology High resolution challenges HDTV standard 1080p requires throughput of 156 Mb/sec uncompressed. Current broadcasts takes approximately 30 Mb/sec uncompressed Challenges: Disc space Transfer speed Other problems (e.g. dynamic range)

Klas Skogmar, Lund Institute of Technology Working practice Real-time Expensive Scanning film Many formats High resolution Proxies and AOI Specialized hardware

Klas Skogmar, Lund Institute of Technology Available technologies Dedicated hardware OpenGL (Imaging subset) DirectX (Direct3D and DirectShow) Quicktime ATI’s and Nvidia’s

Klas Skogmar, Lund Institute of Technology DirectShow Microsoft’s media development environment Built up of filters – each with its own task Filters are assembled in a filter graph Is supported by many video and image formats

Klas Skogmar, Lund Institute of Technology My program Built using MFC Uses a DirectShow filter Shows how DirectShow can be used

Klas Skogmar, Lund Institute of Technology Performance My program was slow ATI has developed a program that is fast (uses VMR-9) Serious Magic’s Texture Download Benchmark Matt Craighead’s program

Klas Skogmar, Lund Institute of Technology Performance Results: ATI’s program shows the potential – adding effects do not give visible frame drops The other programs shows the transfer speed from the graphics card to the processor – it is not enough, but driver updates enhances the performance

Klas Skogmar, Lund Institute of Technology Conclusions It is already a technique that can be used! Drivers has to be enhanced to allow the modifications to be written back to memory Graphics cards are more flexible than dedicated hardware Effects have to be rewritten Many effects are more suitable to do using graphics cards

Klas Skogmar, Lund Institute of Technology Conclusions The transfer to and from disc can be a problem There is a need for modules (e.g. DirectShow filters), that makes it easy to use programmable graphics cards