2. Pixmotor: A Pixel Motion Integrator Ivan Neulander Rhythm & Hues

3. What is Pixmotor? A 2D motion blur solution at Rhythm & Hues Makes motion-blurred images based on static images generated by 3D renderer Night At The Museum Garfield: A Tail of Two Kitties

4. What is Pixmotor? A 2D motion blur solution at Rhythm & Hues Makes motion-blurred images based on static images generated by 3D renderer Not a physically correct but plausible – Incorporates heuristics to minimize artifacts Night At The Museum Garfield: A Tail of Two Kitties

5. What is Pixmotor? A 2D motion blur solution at Rhythm & Hues Makes motion-blurred images based on static images generated by 3D renderer Not a physically correct but plausible – Incorporates heuristics to minimize artifacts Fits well into our production pipeline – Render objects separately, combine in comp – Zero-centered shutter minimizes interpolation distance from static frame Night At The Museum Garfield: A Tail of Two Kitties

6. What is Pixmotor? A 2D motion blur solution at Rhythm & Hues Makes motion-blurred images based on static images generated by 3D renderer Not a physically correct but plausible – Incorporates heuristics to minimize artifacts Fits well into our production pipeline – Render objects separately, combine in comp – Zero-centered shutter minimizes interpolation distance from static frame Night At The Museum Garfield: A Tail of Two Kitties

7. Motivation Processing Speed – Invariant of geometric complexity: Typical 2k element completes in under 1 minute – Many speed/quality controls for interactive testing

8. Motivation Processing Speed – Invariant of geometric complexity: Typical 2k element completes in under 1 minute – Many speed/quality controls for interactive testing Art Direction – Tweak shutter per-frame without rerendering – Vary motion blur within a frame shuttershutter/2shutter x 2shutter x 3

9. Motivation In-Compositor Shading – For static frames: Renderer outputs images of normals, positions, etc.; compositor does shading Diffuse Reflection Shader NormalsOutput

10. Motivation In-Compositor Shading – For static frames: Renderer outputs images of normals, positions, etc.; compositor does shading – For motion-blurred frames: Motion blurring data images in renderer is incorrect Ground truth Diffuse Reflection Shader NormalsOutput

11. Motivation In-Compositor Shading – For static frames: Renderer outputs images of normals, positions, etc.; compositor does shading – For motion-blurred frames: Motion blurring data images in renderer is incorrect Using pixmotor allows doing shading on static images Ground truth Diffuse Reflection Shader NormalsOutput Pixmotor

12. Pixmotor Data Flow Pixmotor Color Input Motion Input Depth Input Settings

13. Computing Motion Vectors When shading a triangle at barycentric point ( , ,  ): Compute NDC motion vector as P NDC ( , ,  ){ShutterClose} - P NDC ( , ,  ){ShutterOpen} Shutter OpenShading MomentShutter Close

14. Implementation: Overview Discretize motion into a set of slices – Each slice = particular moment within shutter span – Total number of slices depends on maximum motion vector User sets maximum pixel separation between slices – Compute each slice and add it to accumulation buffer

15. Implementation: Overview Discretize motion into a set of slices – Each slice = particular moment within shutter span – Total number of slices depends on maximum motion vector User sets maximum pixel separation between slices – Compute each slice and add it to accumulation buffer 1 slice3 slices7 slices52 slices

16. Implementation: Main Loop For each slice: – Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector

17. Implementation: Main Loop For each slice: – Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector Resolve occlusion for overlapping pixels using depth info

18. Implementation: Main Loop For each slice: – Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector Resolve occlusion for overlapping pixels using depth info Holes form due to divergent motion  Use hole-filling heuristics

19. Implementation: Main Loop For each slice: – Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector Resolve occlusion for overlapping pixels using depth info Holes form due to divergent motion  Use hole-filling heuristics Static Image Single-res WB no hole fill Double-res WB, no hole fill Double-res WB, with hole fill Pixmotor Single-slice interpolation

20. Implementation: Hole-Filling Heuristics Before Main Loop: – Blur motion image (optionally) Filter kernel must ignore empty pixels

21. Implementation: Hole-Filling Heuristics Before Main Loop: – Blur motion image (optionally) Filter kernel must ignore empty pixels – Compute motion image gradient Force high gradient at image edges

22. Implementation: Hole-Filling Heuristics Before Main Loop: – Blur motion image (optionally) Filter kernel must ignore empty pixels – Compute motion image gradient Force high gradient at image edges Blur = 0Blur = 3 pixelsBlur = 10 pixelsBlur = 60 pixels Motion Gradient Result

23. Implementation: Hole-Filling Heuristics During Main Loop: – Identify holes as empty pixels near high-gradient regions – Fill holes with nearby solid neighbors along motion path

24. Implementation: Hole-Filling Heuristics During Main Loop: – Identify holes as empty pixels near high-gradient regions – Fill holes with nearby solid neighbors along motion path – Solid pixels at image edges are always presumed holes

25. Implementation: Hole-Filling Heuristics During Main Loop: – Identify holes as empty pixels near high-gradient regions – Fill holes with nearby solid neighbors along motion path – Solid pixels at image edges are always presumed holes Static Image Motion Gradient Hole Fill: off Hole Fill: on [demo]

26. Implementation: Hole-Filling Heuristics After Main Loop: – Option to fix matte for remaining holes:  Unpremultiply partial pixels with solid neighbors (distance along motion vector times user constant) Helps create a solid matte where needed for compositing But: can harden some soft edges

27. Implementation: Hole-Filling Heuristics After Main Loop: – Option to fix matte for remaining holes:  Unpremultiply partial pixels with solid neighbors (distance along motion vector times user constant) Helps create a solid matte where needed for compositing But: can harden some soft edges FixMatte: 0.0 FixMatte: 1.0

28. Results No overlap or transparency: excellent 3D Motion Blur Pixmotor

29. Results No overlap or transparency: excellent With overlap or transparency: not so good 3D Motion Blur Pixmotor

30. Results: Linear 3D MB (easy case)

31. Results: Pixmotor (easy case)

32. Results: Linear 3D MB (hard case)

33. Results: Pixmotor (hard case)

34. Motion-Blurred Shadows: Our Approach Need to store partial occlusion in depth map  use matte channel MatteDepth

35. Motion-Blurred Shadows: Our Approach Need to store partial occlusion in depth map  use matte channel Process depth, matte channels using Pixmotor Pixmotor MatteDepth Motion

36. Motion-Blurred Shadows: Our Approach Need to store partial occlusion in depth map  use matte channel Process depth, matte channels using Pixmotor Use unpremultiplied values for depth Depth map camera settings are unmodified Pixmotor MatteDepth Motion MatteDepth

37. Motion-Blurred Shadows: Results

39. Frustrations & Future Work Transparency, occlusion artifacts  use more fragments per pixel But: speed will suffer Nonlinear motion (e.g. propeller)  Use acceleration + velocity for motion Memory-hungry on deep inputs  Process channels in batches But: counteracts multithreading Apply to reflections, refractions – Compute motion vectors for these

40. Demo Video Real-time capture of interactive Icy session on a quad-CPU Opteron 2200 demo demo

41. Thank You Jonathan Robbins Martin Ryan www.rhythm.com/~ivan/pdfs/sketch2007.pdf www.rhythm.com/~ivan/pub/sketch2007.ppt