1. Fourier Depth of Field Cyril Soler, Kartic Subr, Frédo Durand, Nicolas Holzschuch, François Sillion INRIA, UC Irvine, MIT CSAIL

2. Defocus blur is important in photography

3. Defocus is due to integration over aperture Image Aperture Pixel p Lens

4. Defocus Image Aperture Pixel p Lens Scene

5. Monte Carlo estimate of aperture integral Image Aperture N A primary rays per pixel Integrate at p

6. Aperture integration is costly Image Aperture N P pixels N P x N A Primary rays N A Aperture samples

7. 64 x #primary rays of the pinhole image Aperture integration is costly Paradox: More blurry image is costlier to compute!

8. Key observations

9. Observation 1: Image sampling Blurry regions should not require dense sampling of the image

10. Observation 2: Lens sampling Regions in focus should not require profuse sampling of the lens for diffuse objects

11. Observation 2: Aperture sampling Plane in focus At “sharp” pixels, rays are from same scene point Lens Image Regions in focus should not require profuse sampling of the lens for diffuse objects

12. Observation 2: Aperture sampling Plane in focus Variance depends on reflectance Lens Image Regions in focus should not require profuse sampling of the lens for diffuse objects

13. Goal: Adaptive sampling  Reduce number of primary rays  Adapt image and lens sampling rates based on Fourier bandwidth prediction

14. Sampling: 1) Image Sample blurry image regions sparsely Reference Our image samples

15. Sampling: 2) Aperture 100 4 20 Sample aperture sparsely for objects in focus

16. Contributions  Fourier analysis of depth of field for image synthesis – Account for different transport phenomena  Mechanism for propagating local frequency content  Adaptive sampling of image and lens

17. Related work

18. Related work: Sampling approach [Cook et al. 84]  Trace multiple rays per pixel  Correctly account for phenomena  Costly [Cook et al. 87]

19. Related work: Image space approach [Kraus and Strengert 07] [Kass et al. 06]  Post process pinhole image using depth map  Fast although approximate  Correct handling of occlusion is a challenge [Potmesil and Chakravarty 81]

20. Related work: Frequency domain analysis [Chai et al. 2000] [Durand et al. 05] [Ramamoorthi and Hanrahan 04] [Ng. 05]

21. Algorithm

22. Typical Algorithm for estimating defocus for each pixel x in P for each sample y in L Sum ← Sum + EstimatedRadiance(x, y) P = {uniformly distributed image samples} N A // number of aperture samples Image (x) = Sum / N A L ← SampleLens(N A )

23. Our adaptive sampling for each pixel x in P P = {uniformly distributed image samples} N A (P, A ) ← BandwidthEstimation() P = {bandwidth dependent image samples} A = {aperture variance estimate} N A proportional to A(x) Reconstruct (Image, P) L ← SampleLens(N A ) for each sample y in L Image (x) = Sum / N A Sum ← Sum + EstimatedRadiance(x, y)

24. Algorithm Bandwidth Estimation Sampling rates over image and lens Estimate radiance rays through image and lens samples Reconstruct image from scattered radiance estimates

25. Theory: Propagation of light field spectra

26. Review: Local light field parametrization Space Angle [Durand05]

27. Local light field [Durand05] Review: Local light field as a density Angle Space 1D Lambertian emitter

28. Review: Local light field spectrum Local light field Power spectrum Fourier Transform [Durand05] Angular frequencies Spatial frequencies

29. Review: Transport & Local light field spectra [Durand05] Transport (free space) Processes Occlusion Reflectance Shear (angle) Operations Convolution Product

30. Our sampled representation  Samples in frequency space  Updated through light transport  Provides  bandwidth (max frequency)  Variance (sum of square frequencies) Spatial frequencies Angular frequencies Light field spectrum Sampled Light field spectrum

31. Our sampled representation  Samples in frequency space  Updated through light transport  Provides  bandwidth (max frequency)  Variance (sum of square frequencies) High spatial frequency High angular frequency Sampled Light field spectrum Spatial freq. Angular freq. Light field spectrum

32. Our sampled representation  Samples in frequency space  Updated through light transport  Provides  bandwidth (max frequency)  Variance (sum of square frequencies) Sampled Light field spectrum High spatial frequency Low angular frequency Angular freq. Light field spectrum

33. Our sampled representation  Samples in frequency space  Updated through light transport  Provides  bandwidth (max frequency)  Variance (sum of square frequencies) Sampled Light field spectrum Light field spectrum Spatial freq. Angular freq. Max angular freq.

34. Propagating light field spectra Aperture Sensor First intersection point P Primary ray through lens center

35. Propagating light field spectra Aperture Sensor Coarse depth image Scene Propagate spectra

36. Propagating light field spectra Aperture Sensor Coarse depth image Propagate spectra Scene Aperture variance Image –space bandwidth

37. Propagating light field spectra Aperture Sensor Coarse depth image Propagate spectra Aperture variance Image –space bandwidth Scene Sparse radiance Trace rays

38. Propagating light field spectra Aperture Sensor Primary ray through lens center Light field incident at P First intersection point P Local image bandwidth Variance over aperture

39. Propagating light field spectra Aperture Sensor Primary ray through lens center First intersection point P Reflection Light field incident at P

40. Propagating light field spectra Aperture Sensor Primary ray through lens center First intersection point P Reflection Transport through free space Light field incident at P

41. Propagating light field spectra Aperture Sensor Primary ray through lens center First intersection point P Reflection Transport through free space Aperture effect Light field incident at P

42. Propagating light field spectra Aperture Sensor Primary ray through lens center First intersection point P Reflection Transport through free space Aperture effect Light field incident at P

43. Propagating light field spectra

44. Incident light field  Assume full spectrum  Conservatively expect all frequencies  Simple, no illumination dependence Spatial frequency Angular frequency

45. Reflection: Last bounce to the eye  Convolution by BRDF  Fourier domain: Product of spectra Incident light field spectrum BRDF spectrum =x Light field spectrum after reflection

46. Reflection: Last bounce to the eye  Convolution by BRDF  Fourier domain: Product of spectra Incident light field spectrum BRDF spectrum =x Light field spectrum after reflection

47. Transport to aperture  Transport through free space: angular shear of the light field spectrum [Durand05] Spatial frequency Angular frequency

48. Transport to aperture Occluder  Transport through free space  Occlusion: Convolution with blocker spectrum [Durand05] = * Light field spectrum after occlusion Light field before occlusion Blocker spectrum

49. Occlusion test Occluder  To find occluders for ray through pixel p  Test if depth value at q is in cone of rays Image p

50. Occlusion test Occluder  To find occluders for ray through pixel p  Test if depth value at q is in cone of rays Image p q

51. Occlusion test Occluder  To find occluders for ray through pixel p  Test if depth value at q is in cone of rays Image p q

52. Occlusion test Occluder  To find occluders for ray through pixel p  Test if depth value at q is in cone of rays Image p q

53. Operations on sampled spectra X Y X+Y Draw Samples f(x)g(x)

54. Operations on sampled spectra X+Y f(x) g(x) * X+Y X Y Draw Samples f(x)g(x) Simply add frequency samples and sampled occluder spectra

55. Occlusion test Occluder  To find occluders for ray through pixel p  Test if depth value at q is in cone of rays Image p q

56. Transport to aperture Occluder  Transport through free space  Occlusion: Convolution with blocker spectrum [Durand05] = * Light field spectrum after occlusion Light field before occlusion Blocker spectrum

57. Transport to aperture Occluder  Transport through free space  Occlusion  Transport through free space Angular frequency Spatial frequency

58. Effect of finite aperture  Model integration of rays at the aperture as convolution in ray space

59. Fourier depth of field analysis Ray integration modeled as convolution ImageApertureLensPlane in focus

60.  Model integration of rays at the aperture as convolution in ray space Fourier depth of field analysis ImageApertureLensPlane in focus Ray integration modeled as convolution Dirac in space Box in angle Ray space

61.  Model integration of rays at the aperture as convolution in ray space Fourier depth of field analysis ImageApertureLensPlane in focus Ray integration modeled as convolution Dirac in space Box in angle Ray space Fourier Constant in space Sinc in angle Parametrization at plane in focus

62.  Model integration of rays at the aperture as convolution in ray space Fourier depth of field analysis ImageApertureLensPlane in focus Ray integration modeled as convolution Dirac in space Box in angle Ray space Fourier Constant in space Sinc in angle Shear by distance to Lens

63. Effect of finite aperture  Model integration as convolution in ray space  Hence product in Fourier space  See paper for details = x Incident spectrum at aperture Aperture response spectrum Light field spectrum after DOF effect

64. Transport to sensor  Angular shear [Durand05]  Usually small

65. Estimating sampling rates Local image bandwidth? Variance over aperture?

66. Propagating light field spectra Aperture Sensor Coarse depth image Propagate spectra Image –space bandwidth Scene ? Aperture variance ?

67. Spatial frequency Angular frequency

68. Aperture variance  Project obliquely onto angular axis

69. Aperture variance  Variance = (power spectrum) 2 – (DC) 2 [Parseval]

70. Local image-space bandwidth  Project horizontally onto angular axis

71. Local image-space bandwidth  See paper for details Bandwidth Max angular frequency

72. Summary: Operations on light field spectra Reflection Transport Aperture Transport Processes

73. Operations on light field spectra Product Shear (angle) Convolution Product Shear (angle) Reflection Transport Aperture Transport ProcessesOperations on spectra

74. Operations on sampled spectra Product (Band-limit) Shear (angle) Convolution Shear (angle) Product (Band-limit) Band-limit

75. Operations on sampled spectra Product (Band-limit) Shear (angle) Convolution Shear (angle) Product (Band-limit) Reject samples

76. Operations on sampled spectra Product Shear (angle) Convolution Product Shear (angle) P P’  Angular shear, distance s  Update P(x, t) to P’(x, t-sx)

77. Operations on sampled spectra Product Shear (angle) Convolution Product Shear (angle) X Y X+Y Draw Samples f(x)g(x)

78. Operations on sampled spectra Product Shear (angle) Convolution Product Shear (angle) X Y X+Y Draw Samples f(x)g(x) f(x) g(x) *

79. Bandwidth estimation is useful  Easy modification to existing algorithms  Efficient allocation of samples  Independent of method used to estimate radiance

80. Typical Algorithm for estimating defocus for each pixel x in P for each sample y in L Sum ← Sum + EstimatedRadiance(x, y) P = {uniformly distributed image samples} N A Image (x) = Sum / N A Algorithm SimDOF (P, N A ) L ← SampleLens(N A )

81. Simple modification to sampling for each pixel x in P P = {uniformly distributed image samples} N A Algorithm SimDOF (P, N A ) (P, A ) ← BandwidthEstimation() (P, A ) P = {bandwidth dependent image samples} A = {aperture variance estimate} N A proportional to A(x) Reconstruct (Image, P) L ← SampleLens(N A ) for each sample y in L Image (x) = Average / N A Sum ← Sum + EstimatedRadiance(x, y)

82. Summary Bandwidth Estimation Sample generation over image and lens Estimate radiance rays through image and lens samples Reconstruct image from scattered radiance estimates

83. Summary Bandwidth Estimation Sample generation over image and lens Estimate radiance rays through image and lens samples Reconstruct image from scattered radiance estimates < 2% of total time ~ 0.5% of total time

84. Bandwidth estimation is fast for each pixel x in P P = {uniformly distributed image samples} N A Algorithm SimDOF (P, N A ) (P, A ) ← BandwidthEstimation() (P, A ) P = {bandwidth dependent image samples} A = {aperture variance estimate} N A proportional to A(x) Reconstruct (Image, P) L ← SampleLens(N A ) for each sample y in L Image (x) = Average / N A Sum ← Sum + EstimatedRadiance(x, y) < 2% of total time ~ 0.5 % of total time

85. Results Local bandwidth (image space) Aperture variance

86. Results: Computation time (seconds) Bandwidth estimation Raytracing Image reconstruction 904560 450031507401 1038

87. Results: Quality comparison (similar cost) Stratified lens sampling (70 lens samp/pixel) Our algorithm Adaptive sampling

88. Results: Cost comparison (similar quality) Speedup = 17.3 Speedup = 14.7 Speedup = 24.0 #Primary rays using standard technique #Primary rays using bandwidth prediction Speedup =

89. Results: Variance estimate Our estimated variance Computed reference variance

90. Results: Variance estimate Our estimated variance Computed reference variance

91. Summary of phenomena Reference Aperture variance Image-space bandwidth  Defocus  Reflectance  Occlusion

92. Reference Aperture variance Image-space bandwidth  Defocus  Reflectance  Occlusion Defocus Summary of phenomena

93. Reference Aperture variance Image-space bandwidth  Defocus  Reflectance  Occlusion Reflection Summary of phenomena

94. Reference  Defocus  Reflectance  Occlusion Aperture variance Image-space bandwidth Occlusion Summary of phenomena

95. Limitations  Approximate bandwidth – We do not account for illumination – Coarse approximation of band-limiting operations

96. Conclusion  Reduced number of primary rays for depth of field  Fourier analysis  New mechanism for propagating local bandwidth  Considered several transport phenomena

97. Acknowledgements   ANR "HFIBMR" (ANR-07-BLAN-0331)   INRIA Equipe Associée with MIT "Flexible Rendering“   INRIA post-doctoral program   NSF   Microsoft New Faculty Fellowship   Sloan fellowship

98. Thank you Reference Aperture variance Image-space bandwidth

100. Results: Similar cost Without bandwidth prediction With our bandwidth prediction

101. Thin lens system: Object in focus Plane in focus LensAperture Image

102. Thin lens system: Object out of focus Plane in focus Image LensAperture Blurry image

103. Thin lens system: Object out of focus Plane in focus Image LensAperture Circle of confusion

104. Depth of field (DOF) Image LensAperture Maximum acceptable circle of confusion Depth of Field

105. Blurry pixels

106. Focusing on a subject

107. Depth of field  Range of depths that appear acceptably sharp Aperture Sensor Scene Depth of field

108. Physically based image synthesis  Mimic the effect of a shallow depth of field  Costly affair

109. Our speedup 17.3x 24.0x

110. Pinhole camera Image Pinhole aperture Pixel p

111. Aperture integration is costly Image Aperture N P pixels N P x N A Primary rays Pinhole image N P primary rays

112. Sampling: 1) Image Image Aperture Which pixels to trace primary rays through? ?

113. Sampling: 2) Aperture How many primary rays through a given p? Image Aperture Pixel p ?

114. Review: Local light field [Durand05] 4D light field around a central ray Central ray [Durand05]

115. Our approach Aperture Sensor Scene Transport local-light field spectral information

116. Our approach Aperture Sensor Scene Local bandwidth Aperture variance

117. Our approach  Transport spectra of local light field to sensor  Predict image-space bandwidth and aperture variance  Derive sampling rates Aperture Sensor Scene Estimate sampling rates