1. Frequency Analysis and Sheared Reconstruction for Rendering Motion Blur Kevin Egan Yu-Ting Tseng Nicolas Holzschuch Frédo Durand Ravi Ramamoorthi Columbia University Grenoble University MIT CSAIL University of California, Berkeley

2. Overview  Introduction Overview of Frequency Analysis Sampling and Sheared Filter Implementation Results

3. Motion Blur Objects move while camera shutter is open – Image is “blurred” over time Expensive for special effects Necessary to remove “strobing” in animations

4. Garfield: A Tail of Two Kitties Rhythm & Hues Studios Twentieth Century-Fox Film Corporation

5. The Incredibles Pixar Animation Studios Walt Disney Pictures

6. A Simple Approach For each pixel – Sample many different moments in time t = 1.00 t = 0.75 t = 0.50 t = 0.25 t = 0.00 t  [0.0, 1.0]

7. A Simple Approach The simple approach is expensive Can we do better?

8. Observation Motion blur is expensive Motion blur removes spatial complexity

9. Standard Method Use axis-aligned pixel filter at each pixel Requires many samples SPACE TIME space-time samples axis-aligned filter

10. Our Method Use a different filter shape at each pixel Filter sheared in space-time Fewer samples and faster renders SPACE TIME space-time samples sheared filter

11. Previous Work Plenoptic Sampling Chai et al., 2000 Sheared reconstruction filter in space-angle A Frequency Analysis of Light Transport Durand et al., 2005 Analysis of transport, reflection and occlusion We extend both to space-time

12. Previous Work Multidimensional Adaptive Sampling Hachisuka et al., 2008 Anisotropic reconstruction based on contrast Our method based on local freq estimates Spatial Anti-Aliasing for Animation Sequences with Spatio-Temporal Filtering Shinya, 1993 Sheared filter across multiple images We use speed bounds, derive sampling rates

13. Overview Introduction  Overview of Frequency Analysis Sampling and Sheared Filter Implementation Results

14. Space-Time Analysis Goals Uniform motion leads to sparse freq content Analyze shutter filter’s effect on freq content Design a filter customized to freq content

15. Basic Example Object not moving x y SPACE f(x, y)f(x, t) Space-time graph TIME

16. Basic Example x y t x f(x, t) Low velocity, t  [ 0.0, 1.0 ] f(x, y)

17. Basic Example x y t x f(x, t) High velocity, t  [ 0.0, 1.0 ] f(x, y)

18. Shear in Space-Time x y t x f(x, t) Object moving with low velocity f(x, y) shear

19. Large Shear in Space-Time x y t x Object moving with high velocity f(x, y)f(x, t)

20. Shear in Space-Time Object moving away from camera x y t x f(x, y)f(x, t)

21. Camera Shutter Filter Applying shutter blurs across time x y t x f(x, y)f(x, t)

22. Basic Example – Fourier Domain Fourier spectrum, zero velocity t x f(x, t)F(Ω x, Ω t ) texture bandwidth ΩtΩt ΩxΩx

23. Basic Example – Fourier Domain Low velocity, small shear in both domains f(x, t)F(Ω x, Ω t ) t x slope = -speed ΩtΩt ΩxΩx

24. Basic Example – Fourier Domain Large shear f(x, t)F(Ω x, Ω t ) t x ΩtΩt ΩxΩx

25. Basic Example – Fourier Domain Non-linear motion, wedge shaped spectra f(x, t) ΩtΩt ΩxΩx F(Ω x, Ω t ) t x shutter bandlimits in time -min speed -max speed shutter applies blur across time indirectly bandlimits in space

26. Main Insights Common case = double wedge spectra Shutter indirectly removes spatial freqs Moving reflections and shadows in paper ΩtΩt ΩxΩx double wedge spectrummoving reflection

27. Overview Introduction and Simple Example Overview of Frequency Analysis  Sampling and Sheared Filter Implementation Results

28. Sampling and Filtering Goals Minimal sampling rates to prevent aliasing Derive shape of new reconstruction filter

29. Sampling in Fourier Domain ΩtΩt ΩxΩx t x Sampling produces replicas in Fourier domain Sparse sampling produces dense replicas Fourier DomainSpace-Time Domain

30. Standard Reconstruction Filtering Standard filter, dense sampling (slow) ΩtΩt no aliasing ΩxΩx Fourier Domain replicas

31. Standard Reconstruction Filter Standard filter, sparse sampling (fast) ΩtΩt aliasing ΩxΩx Fourier Domain

32. Sheared Reconstruction Filter Our sheared filter, sparse sampling (fast) ΩtΩt ΩxΩx No aliasing! Fourier Domain

33. Sheared Reconstruction Filter Compact shape in Fourier = wide space-time t x Space-Time Domain ΩtΩt ΩxΩx Fourier Domain

34. Main Insights Sheared filter allows for many fewer samples Paper derives sampling rates – for constant velocity same cost as a static image

35. Overview Introduction Overview of Frequency Analysis Sampling and Sheared Filter  Implementation Results

36. Our Method 1. Compute bounds for signal speeds 2. Compute filter shapes and sampling rates 3. Render samples and reconstruct image

37. Car Example staticmotion blurred

38. Implementation: Stage 1 Sparse sampling to compute velocity bounds max speedmin speed

39. Implementation: Stage 2 Calculate filter widths and sampling rates filter width max speed min speed

40. Implementation: Stage 2 Uniform velocities, wide filter, low samples filter width max speed min speed

41. Implementation: Stage 2 Static surface, small filter, low samples filter width max speed min speed

42. Implementation: Stage 2 Varying velocities, small filter, high samples filter width max speed min speed

43. Implementation: Stage 2 Then compute sampling densities samples per pixel Uniform velocities = low sample count

44. Implementation: Stage 2 Then compute sampling densities samples per pixel Varying velocities = high sample count

45. Implementation: Stage 3 Render sample locations in space-time Apply wide sheared filters to nearby samples t x sheared filter overlaps samples in multiple pixels pixels

46. Implementation: Stage 3 Filters stretched along direction of motion Preserve frequencies orthogonal to motion filter shapes

47. Overview Introduction Overview of Frequency Analysis Sheared Filter Implementation  Results

48. Car Scene Stratified Sampling 4 samples per pixel Our Method, 4 samples per pixel

49. Car Scene Stratified Sampling 4 samples per pixel Our Method, 4 samples per pixel

50. Car Scene Inset MDAS 4 samples / pix Our Method 4 samples / pix Ground Truth

51. Car Scene Inset MDAS 4 samples / pix Our Method 4 samples / pix Ground Truth

52. Ballerina Video

53. Ballerina Stills

56. Ballerina Inset Stratified 16 samples / pix 4 min 2 sec Our Method 8 samples / pix 3 min 57 sec Stratified 64 samples / pix 14 min 25 sec Equal TimeEqual Quality

57. Teapot Scene Our Method 8 samples / pix motion blurred reflection

58. Limitations Currently filter along line segments Initial sampling may miss motion Need to calculate speed and bandlimits Multiple texture / reflection / shadow signals

59. Conclusion Derivation of filter shape and sampling rates conservative min/max bounds for speed packing spectra as tightly as possible Implementation of sheared filter No explicit representation for spectrum Easily added to existing rendering pipelines Faster render times Space-time Fourier theory for rendering Double wedge shape using min/max bounds Analysis of shutter filter

60. Acknowledgements Funding Agencies ONR, NSF, Microsoft, Sloan Foundation, INRIA, LJK Equipment and Licenses Pixar, Intel, NVIDIA

61. Questions? t x sampling rate

64. Previous Work Distributed Ray Tracing, Cook et al., 1984 Sampling across image, time and lens The Reyes Image Rendering Architecture, Cook et al., 1987 Separate shading from visibility

65. Implementation: Stage 3 t x Narrow filter, low sample count filter width

66. Implementation: Stage 3 t x Wide filter, low sample count filter width

67. Implementation: Stage 3 t x Narrow filter, high sample count filter width