Pixmotor: A Pixel Motion Integrator Ivan Neulander Rhythm & Hues

What is Pixmotor? A 2D motion blur solution at Rhythm & Hues Makes motion-blurred images based on static images generated by 3D renderer Night At The Museum Garfield: A Tail of Two Kitties

What is Pixmotor? A 2D motion blur solution at Rhythm & Hues Makes motion-blurred images based on static images generated by 3D renderer Not a physically correct but plausible – Incorporates heuristics to minimize artifacts Night At The Museum Garfield: A Tail of Two Kitties

What is Pixmotor? A 2D motion blur solution at Rhythm & Hues Makes motion-blurred images based on static images generated by 3D renderer Not a physically correct but plausible – Incorporates heuristics to minimize artifacts Fits well into our production pipeline – Render objects separately, combine in comp – Zero-centered shutter minimizes interpolation distance from static frame Night At The Museum Garfield: A Tail of Two Kitties

What is Pixmotor? A 2D motion blur solution at Rhythm & Hues Makes motion-blurred images based on static images generated by 3D renderer Not a physically correct but plausible – Incorporates heuristics to minimize artifacts Fits well into our production pipeline – Render objects separately, combine in comp – Zero-centered shutter minimizes interpolation distance from static frame Night At The Museum Garfield: A Tail of Two Kitties

Motivation Processing Speed – Invariant of geometric complexity: Typical 2k element completes in under 1 minute – Many speed/quality controls for interactive testing

Motivation Processing Speed – Invariant of geometric complexity: Typical 2k element completes in under 1 minute – Many speed/quality controls for interactive testing Art Direction – Tweak shutter per-frame without rerendering – Vary motion blur within a frame shuttershutter/2shutter x 2shutter x 3

Motivation In-Compositor Shading – For static frames: Renderer outputs images of normals, positions, etc.; compositor does shading Diffuse Reflection Shader NormalsOutput

Motivation In-Compositor Shading – For static frames: Renderer outputs images of normals, positions, etc.; compositor does shading – For motion-blurred frames: Motion blurring data images in renderer is incorrect Ground truth Diffuse Reflection Shader NormalsOutput

Motivation In-Compositor Shading – For static frames: Renderer outputs images of normals, positions, etc.; compositor does shading – For motion-blurred frames: Motion blurring data images in renderer is incorrect Using pixmotor allows doing shading on static images Ground truth Diffuse Reflection Shader NormalsOutput Pixmotor

Pixmotor Data Flow Pixmotor Color Input Motion Input Depth Input Settings

Computing Motion Vectors When shading a triangle at barycentric point ( , ,  ): Compute NDC motion vector as P NDC ( , ,  ){ShutterClose} - P NDC ( , ,  ){ShutterOpen} Shutter OpenShading MomentShutter Close

Implementation: Overview Discretize motion into a set of slices – Each slice = particular moment within shutter span – Total number of slices depends on maximum motion vector User sets maximum pixel separation between slices – Compute each slice and add it to accumulation buffer

Implementation: Overview Discretize motion into a set of slices – Each slice = particular moment within shutter span – Total number of slices depends on maximum motion vector User sets maximum pixel separation between slices – Compute each slice and add it to accumulation buffer 1 slice3 slices7 slices52 slices

Implementation: Main Loop For each slice: – Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector

Implementation: Main Loop For each slice: – Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector Resolve occlusion for overlapping pixels using depth info

Implementation: Main Loop For each slice: – Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector Resolve occlusion for overlapping pixels using depth info Holes form due to divergent motion  Use hole-filling heuristics

Implementation: Main Loop For each slice: – Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector Resolve occlusion for overlapping pixels using depth info Holes form due to divergent motion  Use hole-filling heuristics Static Image Single-res WB no hole fill Double-res WB, no hole fill Double-res WB, with hole fill Pixmotor Single-slice interpolation

Implementation: Hole-Filling Heuristics Before Main Loop: – Blur motion image (optionally) Filter kernel must ignore empty pixels

Implementation: Hole-Filling Heuristics Before Main Loop: – Blur motion image (optionally) Filter kernel must ignore empty pixels – Compute motion image gradient Force high gradient at image edges

Implementation: Hole-Filling Heuristics Before Main Loop: – Blur motion image (optionally) Filter kernel must ignore empty pixels – Compute motion image gradient Force high gradient at image edges Blur = 0Blur = 3 pixelsBlur = 10 pixelsBlur = 60 pixels Motion Gradient Result

Implementation: Hole-Filling Heuristics During Main Loop: – Identify holes as empty pixels near high-gradient regions – Fill holes with nearby solid neighbors along motion path

Implementation: Hole-Filling Heuristics During Main Loop: – Identify holes as empty pixels near high-gradient regions – Fill holes with nearby solid neighbors along motion path – Solid pixels at image edges are always presumed holes

Implementation: Hole-Filling Heuristics During Main Loop: – Identify holes as empty pixels near high-gradient regions – Fill holes with nearby solid neighbors along motion path – Solid pixels at image edges are always presumed holes Static Image Motion Gradient Hole Fill: off Hole Fill: on [demo]

Implementation: Hole-Filling Heuristics After Main Loop: – Option to fix matte for remaining holes:  Unpremultiply partial pixels with solid neighbors (distance along motion vector times user constant) Helps create a solid matte where needed for compositing But: can harden some soft edges

Implementation: Hole-Filling Heuristics After Main Loop: – Option to fix matte for remaining holes:  Unpremultiply partial pixels with solid neighbors (distance along motion vector times user constant) Helps create a solid matte where needed for compositing But: can harden some soft edges FixMatte: 0.0 FixMatte: 1.0

Results No overlap or transparency: excellent 3D Motion Blur Pixmotor

Results No overlap or transparency: excellent With overlap or transparency: not so good 3D Motion Blur Pixmotor

Results: Linear 3D MB (easy case)

Results: Pixmotor (easy case)

Results: Linear 3D MB (hard case)

Results: Pixmotor (hard case)

Motion-Blurred Shadows: Our Approach Need to store partial occlusion in depth map  use matte channel MatteDepth

Motion-Blurred Shadows: Our Approach Need to store partial occlusion in depth map  use matte channel Process depth, matte channels using Pixmotor Pixmotor MatteDepth Motion

Motion-Blurred Shadows: Our Approach Need to store partial occlusion in depth map  use matte channel Process depth, matte channels using Pixmotor Use unpremultiplied values for depth Depth map camera settings are unmodified Pixmotor MatteDepth Motion MatteDepth

Motion-Blurred Shadows: Results

Frustrations & Future Work Transparency, occlusion artifacts  use more fragments per pixel But: speed will suffer Nonlinear motion (e.g. propeller)  Use acceleration + velocity for motion Memory-hungry on deep inputs  Process channels in batches But: counteracts multithreading Apply to reflections, refractions – Compute motion vectors for these

Demo Video Real-time capture of interactive Icy session on a quad-CPU Opteron 2200 demo demo

Thank You Jonathan Robbins Martin Ryan