Spectral Focal Sweep (SFS): Extended Depth of Field from Chromatic Aberrations ICCP 2010 Ollie Cossairt, Shree Nayar Columbia University

Conventional Camera (F/4) BLURRY Spectral Focal Sweep: ICCP 2010

Conventional Camera (F/16) NOISY Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 EDOF Camera (F/4) CAPTURED Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 EDOF Camera (F/4) DEBLURRED Spectral Focal Sweep: ICCP 2010

Conventional Camera (F/16) NOISY Spectral Focal Sweep: ICCP 2010

Conventional Camera (F/4) BLURRY Spectral Focal Sweep: ICCP 2010

Conventional Camera (F/16) NOISY Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 EDOF Camera (F/4) CODED Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 EDOF Camera (F/4) DEBLURRED Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 Related Work Coded Aperture [Welford ’60] [Levin et al. ‘07] [Veeraraghavan et al. ’07] [Zhou and Nayar ’09] Phase Plates [Dowski and Cathey ’95] [Chi and George ’01] [Levin et al. ’09] Focus Sweep [Nagahara et al. ’08] [Hausler ’72] Lens Focal Plane Sensor Color [Guichard et al. ’09] Lens Focal Planes RGB Sensor Red Green Blue Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep Idea Broadband reflectance spectra Poorly Corrected Lens + B&W Sensor + Focus Sweep Lens Focal Plane Sensor = No moving parts or custom elements Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep Idea Broadband reflectance spectra Works exactly for black and white scenes Many naturally occurring reflectance spectra are also broadband [Parkkinen ‘89] Poorly corrected lens Simpler lens design optimization Black & White Sensor Can also approximate with luminance channel from color sensor Spectral Focal Sweep: ICCP 2010

Conventional Focus Sweep PSF u = object depth (fixed) v = sensor distance (varying) Instantaneous PSF Lens EFL = f Focal Plane Sensor v1 u Scene Depth Range Spectral Focal Sweep: ICCP 2010

Conventional Focus Sweep PSF u = object depth (fixed) v = sensor distance (varying) Instantaneous PSF Lens EFL = f Focal Plane Sensor d2 v2 u Scene Depth Range Spectral Focal Sweep: ICCP 2010

Conventional Focus Sweep PSF u = object depth (fixed) v = sensor distance (varying) Instantaneous PSF Lens EFL = f Focal Plane Sensor d3 v3 u Scene Depth Range Spectral Focal Sweep: ICCP 2010

Conventional Focus Sweep PSF Defocus disk diameter (A = aperture diameter) Integrate over time to get final PSF Circ/Top-hat function Varying either object/sensor distance, or focal length will produce the same PSF Spectral Focal Sweep: ICCP 2010

Refractive Dispersion (prism) Snell’s Law of refraction Refraction depends on wavelength Spectral Focal Sweep: ICCP 2010

Refractive Dispersion (lens) For an uncorrected lens, Focal length depends on wavelength R1 , R2 = lens radii of curvature u v Sensor PSF Focal Planes Scene Depth Range Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep PSF Each wavelength produces a different defocus disk A B&W Sensor integrates over wavelength, weighted by the spectral reflectance of the scene For white objects, a SFS PSF is identical to a mechanical focal sweep PSF Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 SFS Lens Design Design Criteria Wavelength range 400-700nm 75mm EFL F/4 1/3” Sensor, 10um pixel size Optimization Criteria Zemax Optical Design Software Maximize chromatic defocus across wavelength range Minimize spot size at 550nm Fit to off-the-shelf lenses Maximum defocus diameter 100um at 400nm and 700nm Spectral Focal Sweep: ICCP 2010

Zemax SFS PSFs (white spectrum) Corrected lens SFS Lens field 15x depth Spectral Focal Sweep: ICCP 2010

Zemax SFS PSFs (white spectrum) Corrected lens field SFS Lens 1 15 x depth Spectral Focal Sweep: ICCP 2010

Simulating Real Spectra Munsell database of 1250 measured reflectance spectra Zemax simulation of SFS PSF at 12 depth locations (1.5m-3m, 100um max defocus dia.) For each Munsell spectrum Calculate average deblurring error when a white spectrum is used for deblurring Spectral Focal Sweep: ICCP 2010

Average Deblurring Error Input Image: Munsell color PSFs: White PSF at center depth: PSF Distance Metric: [Zhou and Nayar ‘09] Average Deblurring Error: Averaged over many images Spectral Focal Sweep: ICCP 2010

Munsell Spectra PSFs Most Munsell spectra show little PSF variation Corrected lens 1 15 x depth Most Munsell spectra show little PSF variation Spectral Focal Sweep: ICCP 2010

Average Deblurring Error (%95 of Munsell spectra) Most Munsell PSFs introduce negligible deblurring artifacts Spectral Focal Sweep: ICCP 2010

Doublet SFS Implementation Zemax Design Implementation Edmund Optics #48184 1” Dia. 100mm FL Plano-convex asphere Thorlabs #LE1929 1” Dia. 300mm FL positive meniscus Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 Experimental Setup Corrected Lens SFS Doublet Lens B&W Examples (Basler 1/3” sensor) F/4 SFS 75mm EFL Doublet EFL Cosmicar F/1.4 75mm Color Examples (Canon 450D sensor) F/4 SFS 75mm EFL Doublet F/2.8 100mm EFL Cannon Spectral Focal Sweep: ICCP 2010

Measured SFS PSFs (white spectrum) depth Cosmicar SFS lens Measured SFS PSF approximately depth-invariant Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 Deblurring Deblur with measured white PSF B&W Images Wiener deconvolution Color Images Transform from RGB to YUV space Wiener deconvolution of Y image Transform back to RGB space Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 EDOF Examples Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 Corrected Lens 8ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 SFS Camera (Captured) 8ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 SFS Camera (Deblurred) 8ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 Corrected Lens Background Foreground 8ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 SFS Lens (Captured) 8ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 SFS Lens (Deblurred) 8ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 Corrected Lens Foreground Background 16ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 SFS Lens (Captured) 16ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 SFS Lens (Deblurred) 16ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 Corrected Lens Background Foreground 16ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 SFS Lens (Captured) 16ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 SFS Lens (Deblurred) 16ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 Corrected Lens Background Foreground 16ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 SFS Lens (Captured) 16ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 SFS Lens (Deblurred) 16ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 Corrected Lens Background Foreground 16ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 SFS Lens (Captured) 16ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 F/4 SFS Lens (Deblurred) 16ms exposure Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 Limitations Broadband assumption does not always hold (e.g. %5 of Munsell Spectra) deblurring artifacts introduced Color deblurring method does not correct blur in chrominance channels Spectral Focal Sweep: ICCP 2010

Spectral Focal Sweep: ICCP 2010 Conclusion Broadband spectra Chromatic aberrations B&W sensor Focus sweep + = Works exactly for B&W scenes Works well for many naturally occurring spectra Results for color images “look” good Easy to build EDOF camera Spectral Focal Sweep: ICCP 2010