Fourier Depth of Field Cyril Soler, Kartic Subr, Frédo Durand, Nicolas Holzschuch, François Sillion INRIA, UC Irvine, MIT CSAIL

Defocus blur is important in photography

Defocus is due to integration over aperture Image Aperture Pixel p Lens

Defocus Image Aperture Pixel p Lens Scene

Monte Carlo estimate of aperture integral Image Aperture N A primary rays per pixel Integrate at p

Aperture integration is costly Image Aperture N P pixels N P x N A Primary rays N A Aperture samples

64 x #primary rays of the pinhole image Aperture integration is costly Paradox: More blurry image is costlier to compute!

Key observations

Observation 1: Image sampling Blurry regions should not require dense sampling of the image

Observation 2: Lens sampling Regions in focus should not require profuse sampling of the lens for diffuse objects

Observation 2: Aperture sampling Plane in focus At “sharp” pixels, rays are from same scene point Lens Image Regions in focus should not require profuse sampling of the lens for diffuse objects

Observation 2: Aperture sampling Plane in focus Variance depends on reflectance Lens Image Regions in focus should not require profuse sampling of the lens for diffuse objects

Goal: Adaptive sampling  Reduce number of primary rays  Adapt image and lens sampling rates based on Fourier bandwidth prediction

Sampling: 1) Image Sample blurry image regions sparsely Reference Our image samples

Sampling: 2) Aperture Sample aperture sparsely for objects in focus

Contributions  Fourier analysis of depth of field for image synthesis – Account for different transport phenomena  Mechanism for propagating local frequency content  Adaptive sampling of image and lens

Related work

Related work: Sampling approach [Cook et al. 84]  Trace multiple rays per pixel  Correctly account for phenomena  Costly [Cook et al. 87]

Related work: Image space approach [Kraus and Strengert 07] [Kass et al. 06]  Post process pinhole image using depth map  Fast although approximate  Correct handling of occlusion is a challenge [Potmesil and Chakravarty 81]

Related work: Frequency domain analysis [Chai et al. 2000] [Durand et al. 05] [Ramamoorthi and Hanrahan 04] [Ng. 05]

Algorithm

Typical Algorithm for estimating defocus for each pixel x in P for each sample y in L Sum ← Sum + EstimatedRadiance(x, y) P = {uniformly distributed image samples} N A // number of aperture samples Image (x) = Sum / N A L ← SampleLens(N A )

Our adaptive sampling for each pixel x in P P = {uniformly distributed image samples} N A (P, A ) ← BandwidthEstimation() P = {bandwidth dependent image samples} A = {aperture variance estimate} N A proportional to A(x) Reconstruct (Image, P) L ← SampleLens(N A ) for each sample y in L Image (x) = Sum / N A Sum ← Sum + EstimatedRadiance(x, y)

Algorithm Bandwidth Estimation Sampling rates over image and lens Estimate radiance rays through image and lens samples Reconstruct image from scattered radiance estimates

Theory: Propagation of light field spectra

Review: Local light field parametrization Space Angle [Durand05]

Local light field [Durand05] Review: Local light field as a density Angle Space 1D Lambertian emitter

Review: Local light field spectrum Local light field Power spectrum Fourier Transform [Durand05] Angular frequencies Spatial frequencies

Review: Transport & Local light field spectra [Durand05] Transport (free space) Processes Occlusion Reflectance Shear (angle) Operations Convolution Product

Our sampled representation  Samples in frequency space  Updated through light transport  Provides  bandwidth (max frequency)  Variance (sum of square frequencies) Spatial frequencies Angular frequencies Light field spectrum Sampled Light field spectrum

Our sampled representation  Samples in frequency space  Updated through light transport  Provides  bandwidth (max frequency)  Variance (sum of square frequencies) High spatial frequency High angular frequency Sampled Light field spectrum Spatial freq. Angular freq. Light field spectrum

Our sampled representation  Samples in frequency space  Updated through light transport  Provides  bandwidth (max frequency)  Variance (sum of square frequencies) Sampled Light field spectrum High spatial frequency Low angular frequency Angular freq. Light field spectrum

Our sampled representation  Samples in frequency space  Updated through light transport  Provides  bandwidth (max frequency)  Variance (sum of square frequencies) Sampled Light field spectrum Light field spectrum Spatial freq. Angular freq. Max angular freq.

Propagating light field spectra Aperture Sensor First intersection point P Primary ray through lens center

Propagating light field spectra Aperture Sensor Coarse depth image Scene Propagate spectra

Propagating light field spectra Aperture Sensor Coarse depth image Propagate spectra Scene Aperture variance Image –space bandwidth

Propagating light field spectra Aperture Sensor Coarse depth image Propagate spectra Aperture variance Image –space bandwidth Scene Sparse radiance Trace rays

Propagating light field spectra Aperture Sensor Primary ray through lens center Light field incident at P First intersection point P Local image bandwidth Variance over aperture

Propagating light field spectra Aperture Sensor Primary ray through lens center First intersection point P Reflection Light field incident at P

Propagating light field spectra Aperture Sensor Primary ray through lens center First intersection point P Reflection Transport through free space Light field incident at P

Propagating light field spectra Aperture Sensor Primary ray through lens center First intersection point P Reflection Transport through free space Aperture effect Light field incident at P

Propagating light field spectra Aperture Sensor Primary ray through lens center First intersection point P Reflection Transport through free space Aperture effect Light field incident at P

Propagating light field spectra

Incident light field  Assume full spectrum  Conservatively expect all frequencies  Simple, no illumination dependence Spatial frequency Angular frequency

Reflection: Last bounce to the eye  Convolution by BRDF  Fourier domain: Product of spectra Incident light field spectrum BRDF spectrum =x Light field spectrum after reflection

Reflection: Last bounce to the eye  Convolution by BRDF  Fourier domain: Product of spectra Incident light field spectrum BRDF spectrum =x Light field spectrum after reflection

Transport to aperture  Transport through free space: angular shear of the light field spectrum [Durand05] Spatial frequency Angular frequency

Transport to aperture Occluder  Transport through free space  Occlusion: Convolution with blocker spectrum [Durand05] = * Light field spectrum after occlusion Light field before occlusion Blocker spectrum

Occlusion test Occluder  To find occluders for ray through pixel p  Test if depth value at q is in cone of rays Image p

Occlusion test Occluder  To find occluders for ray through pixel p  Test if depth value at q is in cone of rays Image p q

Occlusion test Occluder  To find occluders for ray through pixel p  Test if depth value at q is in cone of rays Image p q

Occlusion test Occluder  To find occluders for ray through pixel p  Test if depth value at q is in cone of rays Image p q

Operations on sampled spectra X Y X+Y Draw Samples f(x)g(x)

Operations on sampled spectra X+Y f(x) g(x) * X+Y X Y Draw Samples f(x)g(x) Simply add frequency samples and sampled occluder spectra

Occlusion test Occluder  To find occluders for ray through pixel p  Test if depth value at q is in cone of rays Image p q

Transport to aperture Occluder  Transport through free space  Occlusion: Convolution with blocker spectrum [Durand05] = * Light field spectrum after occlusion Light field before occlusion Blocker spectrum

Transport to aperture Occluder  Transport through free space  Occlusion  Transport through free space Angular frequency Spatial frequency

Effect of finite aperture  Model integration of rays at the aperture as convolution in ray space

Fourier depth of field analysis Ray integration modeled as convolution ImageApertureLensPlane in focus

 Model integration of rays at the aperture as convolution in ray space Fourier depth of field analysis ImageApertureLensPlane in focus Ray integration modeled as convolution Dirac in space Box in angle Ray space

 Model integration of rays at the aperture as convolution in ray space Fourier depth of field analysis ImageApertureLensPlane in focus Ray integration modeled as convolution Dirac in space Box in angle Ray space Fourier Constant in space Sinc in angle Parametrization at plane in focus

 Model integration of rays at the aperture as convolution in ray space Fourier depth of field analysis ImageApertureLensPlane in focus Ray integration modeled as convolution Dirac in space Box in angle Ray space Fourier Constant in space Sinc in angle Shear by distance to Lens

Effect of finite aperture  Model integration as convolution in ray space  Hence product in Fourier space  See paper for details = x Incident spectrum at aperture Aperture response spectrum Light field spectrum after DOF effect

Transport to sensor  Angular shear [Durand05]  Usually small

Estimating sampling rates Local image bandwidth? Variance over aperture?

Propagating light field spectra Aperture Sensor Coarse depth image Propagate spectra Image –space bandwidth Scene ? Aperture variance ?

Spatial frequency Angular frequency

Aperture variance  Project obliquely onto angular axis

Aperture variance  Variance = (power spectrum) 2 – (DC) 2 [Parseval]

Local image-space bandwidth  Project horizontally onto angular axis

Local image-space bandwidth  See paper for details Bandwidth Max angular frequency

Summary: Operations on light field spectra Reflection Transport Aperture Transport Processes

Operations on light field spectra Product Shear (angle) Convolution Product Shear (angle) Reflection Transport Aperture Transport ProcessesOperations on spectra

Operations on sampled spectra Product (Band-limit) Shear (angle) Convolution Shear (angle) Product (Band-limit) Band-limit

Operations on sampled spectra Product (Band-limit) Shear (angle) Convolution Shear (angle) Product (Band-limit) Reject samples

Operations on sampled spectra Product Shear (angle) Convolution Product Shear (angle) P P’  Angular shear, distance s  Update P(x, t) to P’(x, t-sx)

Operations on sampled spectra Product Shear (angle) Convolution Product Shear (angle) X Y X+Y Draw Samples f(x)g(x)

Operations on sampled spectra Product Shear (angle) Convolution Product Shear (angle) X Y X+Y Draw Samples f(x)g(x) f(x) g(x) *

Bandwidth estimation is useful  Easy modification to existing algorithms  Efficient allocation of samples  Independent of method used to estimate radiance

Typical Algorithm for estimating defocus for each pixel x in P for each sample y in L Sum ← Sum + EstimatedRadiance(x, y) P = {uniformly distributed image samples} N A Image (x) = Sum / N A Algorithm SimDOF (P, N A ) L ← SampleLens(N A )

Simple modification to sampling for each pixel x in P P = {uniformly distributed image samples} N A Algorithm SimDOF (P, N A ) (P, A ) ← BandwidthEstimation() (P, A ) P = {bandwidth dependent image samples} A = {aperture variance estimate} N A proportional to A(x) Reconstruct (Image, P) L ← SampleLens(N A ) for each sample y in L Image (x) = Average / N A Sum ← Sum + EstimatedRadiance(x, y)

Summary Bandwidth Estimation Sample generation over image and lens Estimate radiance rays through image and lens samples Reconstruct image from scattered radiance estimates

Summary Bandwidth Estimation Sample generation over image and lens Estimate radiance rays through image and lens samples Reconstruct image from scattered radiance estimates < 2% of total time ~ 0.5% of total time

Bandwidth estimation is fast for each pixel x in P P = {uniformly distributed image samples} N A Algorithm SimDOF (P, N A ) (P, A ) ← BandwidthEstimation() (P, A ) P = {bandwidth dependent image samples} A = {aperture variance estimate} N A proportional to A(x) Reconstruct (Image, P) L ← SampleLens(N A ) for each sample y in L Image (x) = Average / N A Sum ← Sum + EstimatedRadiance(x, y) < 2% of total time ~ 0.5 % of total time

Results Local bandwidth (image space) Aperture variance

Results: Computation time (seconds) Bandwidth estimation Raytracing Image reconstruction

Results: Quality comparison (similar cost) Stratified lens sampling (70 lens samp/pixel) Our algorithm Adaptive sampling

Results: Cost comparison (similar quality) Speedup = 17.3 Speedup = 14.7 Speedup = 24.0 #Primary rays using standard technique #Primary rays using bandwidth prediction Speedup =

Results: Variance estimate Our estimated variance Computed reference variance

Results: Variance estimate Our estimated variance Computed reference variance

Summary of phenomena Reference Aperture variance Image-space bandwidth  Defocus  Reflectance  Occlusion

Reference Aperture variance Image-space bandwidth  Defocus  Reflectance  Occlusion Defocus Summary of phenomena

Reference Aperture variance Image-space bandwidth  Defocus  Reflectance  Occlusion Reflection Summary of phenomena

Reference  Defocus  Reflectance  Occlusion Aperture variance Image-space bandwidth Occlusion Summary of phenomena

Limitations  Approximate bandwidth – We do not account for illumination – Coarse approximation of band-limiting operations

Conclusion  Reduced number of primary rays for depth of field  Fourier analysis  New mechanism for propagating local bandwidth  Considered several transport phenomena

Acknowledgements   ANR "HFIBMR" (ANR-07-BLAN-0331)   INRIA Equipe Associée with MIT "Flexible Rendering“   INRIA post-doctoral program   NSF   Microsoft New Faculty Fellowship   Sloan fellowship

Thank you Reference Aperture variance Image-space bandwidth

Results: Similar cost Without bandwidth prediction With our bandwidth prediction

Thin lens system: Object in focus Plane in focus LensAperture Image

Thin lens system: Object out of focus Plane in focus Image LensAperture Blurry image

Thin lens system: Object out of focus Plane in focus Image LensAperture Circle of confusion

Depth of field (DOF) Image LensAperture Maximum acceptable circle of confusion Depth of Field

Blurry pixels

Focusing on a subject

Depth of field  Range of depths that appear acceptably sharp Aperture Sensor Scene Depth of field

Physically based image synthesis  Mimic the effect of a shallow depth of field  Costly affair

Our speedup 17.3x 24.0x

Pinhole camera Image Pinhole aperture Pixel p

Aperture integration is costly Image Aperture N P pixels N P x N A Primary rays Pinhole image N P primary rays

Sampling: 1) Image Image Aperture Which pixels to trace primary rays through? ?

Sampling: 2) Aperture How many primary rays through a given p? Image Aperture Pixel p ?

Review: Local light field [Durand05] 4D light field around a central ray Central ray [Durand05]

Our approach Aperture Sensor Scene Transport local-light field spectral information

Our approach Aperture Sensor Scene Local bandwidth Aperture variance

Our approach  Transport spectra of local light field to sensor  Predict image-space bandwidth and aperture variance  Derive sampling rates Aperture Sensor Scene Estimate sampling rates