Frequency Analysis and Sheared Reconstruction for Rendering Motion Blur Kevin Egan Yu-Ting Tseng Nicolas Holzschuch Frédo Durand Ravi Ramamoorthi Columbia University Grenoble University MIT CSAIL University of California, Berkeley

Overview  Introduction Overview of Frequency Analysis Sampling and Sheared Filter Implementation Results

Motion Blur Objects move while camera shutter is open – Image is “blurred” over time Expensive for special effects Necessary to remove “strobing” in animations

Garfield: A Tail of Two Kitties Rhythm & Hues Studios Twentieth Century-Fox Film Corporation

The Incredibles Pixar Animation Studios Walt Disney Pictures

A Simple Approach For each pixel – Sample many different moments in time t = 1.00 t = 0.75 t = 0.50 t = 0.25 t = 0.00 t  [0.0, 1.0]

A Simple Approach The simple approach is expensive Can we do better?

Observation Motion blur is expensive Motion blur removes spatial complexity

Standard Method Use axis-aligned pixel filter at each pixel Requires many samples SPACE TIME space-time samples axis-aligned filter

Our Method Use a different filter shape at each pixel Filter sheared in space-time Fewer samples and faster renders SPACE TIME space-time samples sheared filter

Previous Work Plenoptic Sampling Chai et al., 2000 Sheared reconstruction filter in space-angle A Frequency Analysis of Light Transport Durand et al., 2005 Analysis of transport, reflection and occlusion We extend both to space-time

Previous Work Multidimensional Adaptive Sampling Hachisuka et al., 2008 Anisotropic reconstruction based on contrast Our method based on local freq estimates Spatial Anti-Aliasing for Animation Sequences with Spatio-Temporal Filtering Shinya, 1993 Sheared filter across multiple images We use speed bounds, derive sampling rates

Overview Introduction  Overview of Frequency Analysis Sampling and Sheared Filter Implementation Results

Space-Time Analysis Goals Uniform motion leads to sparse freq content Analyze shutter filter’s effect on freq content Design a filter customized to freq content

Basic Example Object not moving x y SPACE f(x, y)f(x, t) Space-time graph TIME

Basic Example x y t x f(x, t) Low velocity, t  [ 0.0, 1.0 ] f(x, y)

Basic Example x y t x f(x, t) High velocity, t  [ 0.0, 1.0 ] f(x, y)

Shear in Space-Time x y t x f(x, t) Object moving with low velocity f(x, y) shear

Large Shear in Space-Time x y t x Object moving with high velocity f(x, y)f(x, t)

Shear in Space-Time Object moving away from camera x y t x f(x, y)f(x, t)

Camera Shutter Filter Applying shutter blurs across time x y t x f(x, y)f(x, t)

Basic Example – Fourier Domain Fourier spectrum, zero velocity t x f(x, t)F(Ω x, Ω t ) texture bandwidth ΩtΩt ΩxΩx

Basic Example – Fourier Domain Low velocity, small shear in both domains f(x, t)F(Ω x, Ω t ) t x slope = -speed ΩtΩt ΩxΩx

Basic Example – Fourier Domain Large shear f(x, t)F(Ω x, Ω t ) t x ΩtΩt ΩxΩx

Basic Example – Fourier Domain Non-linear motion, wedge shaped spectra f(x, t) ΩtΩt ΩxΩx F(Ω x, Ω t ) t x shutter bandlimits in time -min speed -max speed shutter applies blur across time indirectly bandlimits in space

Main Insights Common case = double wedge spectra Shutter indirectly removes spatial freqs Moving reflections and shadows in paper ΩtΩt ΩxΩx double wedge spectrummoving reflection

Overview Introduction and Simple Example Overview of Frequency Analysis  Sampling and Sheared Filter Implementation Results

Sampling and Filtering Goals Minimal sampling rates to prevent aliasing Derive shape of new reconstruction filter

Sampling in Fourier Domain ΩtΩt ΩxΩx t x Sampling produces replicas in Fourier domain Sparse sampling produces dense replicas Fourier DomainSpace-Time Domain

Standard Reconstruction Filtering Standard filter, dense sampling (slow) ΩtΩt no aliasing ΩxΩx Fourier Domain replicas

Standard Reconstruction Filter Standard filter, sparse sampling (fast) ΩtΩt aliasing ΩxΩx Fourier Domain

Sheared Reconstruction Filter Our sheared filter, sparse sampling (fast) ΩtΩt ΩxΩx No aliasing! Fourier Domain

Sheared Reconstruction Filter Compact shape in Fourier = wide space-time t x Space-Time Domain ΩtΩt ΩxΩx Fourier Domain

Main Insights Sheared filter allows for many fewer samples Paper derives sampling rates – for constant velocity same cost as a static image

Overview Introduction Overview of Frequency Analysis Sampling and Sheared Filter  Implementation Results

Our Method 1. Compute bounds for signal speeds 2. Compute filter shapes and sampling rates 3. Render samples and reconstruct image

Car Example staticmotion blurred

Implementation: Stage 1 Sparse sampling to compute velocity bounds max speedmin speed

Implementation: Stage 2 Calculate filter widths and sampling rates filter width max speed min speed

Implementation: Stage 2 Uniform velocities, wide filter, low samples filter width max speed min speed

Implementation: Stage 2 Static surface, small filter, low samples filter width max speed min speed

Implementation: Stage 2 Varying velocities, small filter, high samples filter width max speed min speed

Implementation: Stage 2 Then compute sampling densities samples per pixel Uniform velocities = low sample count

Implementation: Stage 2 Then compute sampling densities samples per pixel Varying velocities = high sample count

Implementation: Stage 3 Render sample locations in space-time Apply wide sheared filters to nearby samples t x sheared filter overlaps samples in multiple pixels pixels

Implementation: Stage 3 Filters stretched along direction of motion Preserve frequencies orthogonal to motion filter shapes

Overview Introduction Overview of Frequency Analysis Sheared Filter Implementation  Results

Car Scene Stratified Sampling 4 samples per pixel Our Method, 4 samples per pixel

Car Scene Stratified Sampling 4 samples per pixel Our Method, 4 samples per pixel

Car Scene Inset MDAS 4 samples / pix Our Method 4 samples / pix Ground Truth

Car Scene Inset MDAS 4 samples / pix Our Method 4 samples / pix Ground Truth

Ballerina Video

Ballerina Stills

Ballerina Inset Stratified 16 samples / pix 4 min 2 sec Our Method 8 samples / pix 3 min 57 sec Stratified 64 samples / pix 14 min 25 sec Equal TimeEqual Quality

Teapot Scene Our Method 8 samples / pix motion blurred reflection

Limitations Currently filter along line segments Initial sampling may miss motion Need to calculate speed and bandlimits Multiple texture / reflection / shadow signals

Conclusion Derivation of filter shape and sampling rates conservative min/max bounds for speed packing spectra as tightly as possible Implementation of sheared filter No explicit representation for spectrum Easily added to existing rendering pipelines Faster render times Space-time Fourier theory for rendering Double wedge shape using min/max bounds Analysis of shutter filter

Acknowledgements Funding Agencies ONR, NSF, Microsoft, Sloan Foundation, INRIA, LJK Equipment and Licenses Pixar, Intel, NVIDIA

Questions? t x sampling rate

Previous Work Distributed Ray Tracing, Cook et al., 1984 Sampling across image, time and lens The Reyes Image Rendering Architecture, Cook et al., 1987 Separate shading from visibility

Implementation: Stage 3 t x Narrow filter, low sample count filter width

Implementation: Stage 3 t x Wide filter, low sample count filter width

Implementation: Stage 3 t x Narrow filter, high sample count filter width