3. Spectral Focal Sweep (SFS): Extended Depth of Field from Chromatic Aberrations ICCP Ollie Cossairt, Shree Nayar Columbia University

4. Conventional Camera (F/4)
BLURRY
Spectral Focal Sweep: ICCP 2010

5. Conventional Camera (F/16)
NOISY
Spectral Focal Sweep: ICCP 2010

6. Spectral Focal Sweep: ICCP 2010
EDOF Camera (F/4)
CAPTURED
Spectral Focal Sweep: ICCP 2010

7. Spectral Focal Sweep: ICCP 2010
EDOF Camera (F/4)
DEBLURRED
Spectral Focal Sweep: ICCP 2010

8. Conventional Camera (F/16)
NOISY
Spectral Focal Sweep: ICCP 2010

9. Conventional Camera (F/4)
BLURRY
Spectral Focal Sweep: ICCP 2010

10. Conventional Camera (F/16)
NOISY
Spectral Focal Sweep: ICCP 2010

11. Spectral Focal Sweep: ICCP 2010
EDOF Camera (F/4)
CODED
Spectral Focal Sweep: ICCP 2010

12. Spectral Focal Sweep: ICCP 2010
EDOF Camera (F/4)
DEBLURRED
Spectral Focal Sweep: ICCP 2010

13. Spectral Focal Sweep: ICCP 2010
Related Work
Coded Aperture
[Welford ’60] [Levin et al. ‘07]
[Veeraraghavan et al. ’07] [Zhou and Nayar ’09]
Phase Plates
[Dowski and Cathey ’95]
[Chi and George ’01] [Levin et al. ’09]
Focus Sweep
[Nagahara et al. ’08]
[Hausler ’72]
Lens
Focal Plane
Sensor
Color
[Guichard et al. ’09]
Lens
Focal Planes
RGB Sensor
Red
Green
Blue
Spectral Focal Sweep: ICCP 2010

14. Spectral Focal Sweep Idea
Broadband
reflectance
spectra
Poorly
Corrected
Lens +
B&W
Sensor +
Focus Sweep
Lens
Focal
Plane
Sensor =
No moving parts or custom elements
Spectral Focal Sweep: ICCP 2010

15. Spectral Focal Sweep Idea
Broadband reflectance spectra
Works exactly for black and white scenes
Many naturally occurring reflectance spectra are also broadband [Parkkinen ‘89]
Poorly corrected lens
Simpler lens design optimization
Black & White Sensor
Can also approximate with luminance channel from color sensor
Spectral Focal Sweep: ICCP 2010

16. Conventional Focus Sweep PSF
u = object depth (fixed)
v = sensor distance (varying)
Instantaneous
PSF
Lens
EFL = f
Focal
Plane
Sensor v1 u
Scene Depth Range
Spectral Focal Sweep: ICCP 2010

17. Conventional Focus Sweep PSF
u = object depth (fixed)
v = sensor distance (varying)
Instantaneous
PSF
Lens
EFL = f
Focal
Plane
Sensor d2 v2 u
Scene Depth Range
Spectral Focal Sweep: ICCP 2010

18. Conventional Focus Sweep PSF
u = object depth (fixed)
v = sensor distance (varying)
Instantaneous
PSF
Lens
EFL = f
Focal
Plane
Sensor d3 v3 u
Scene Depth Range
Spectral Focal Sweep: ICCP 2010

19. Conventional Focus Sweep PSF
Defocus disk diameter (A = aperture diameter)
Integrate over time to get final PSF
Circ/Top-hat
function
Varying either object/sensor distance, or focal length will produce the same PSF
Spectral Focal Sweep: ICCP 2010

20. Refractive Dispersion (prism)
Snell’s Law of refraction
Refraction depends on wavelength
Spectral Focal Sweep: ICCP 2010

21. Refractive Dispersion (lens)
For an uncorrected lens, Focal length depends on wavelength
R1 , R2 = lens radii of curvature u v
Sensor
PSF
Focal
Planes
Scene Depth Range
Spectral Focal Sweep: ICCP 2010

22. Spectral Focal Sweep PSF
Each wavelength produces a different defocus disk
A B&W Sensor integrates over wavelength, weighted by the spectral reflectance of the scene
For white objects, a SFS PSF is identical
to a mechanical focal sweep PSF
Spectral Focal Sweep: ICCP 2010

23. Spectral Focal Sweep: ICCP 2010
SFS Lens Design
Design Criteria
Wavelength range nm
75mm EFL F/4
1/3” Sensor, 10um pixel size
Optimization Criteria
Zemax Optical Design Software
Maximize chromatic defocus across wavelength range
Minimize spot size at 550nm
Fit to off-the-shelf lenses
Maximum defocus diameter 100um at 400nm and 700nm
Spectral Focal Sweep: ICCP 2010

24. Zemax SFS PSFs (white spectrum)
Corrected lens
SFS Lens
field
15x
depth
Spectral Focal Sweep: ICCP 2010

25. Zemax SFS PSFs (white spectrum)
Corrected lens
field
SFS Lens 1 15 x
depth
Spectral Focal Sweep: ICCP 2010

26. Simulating Real Spectra
Munsell database of 1250 measured reflectance spectra
Zemax simulation of SFS PSF at 12 depth locations (1.5m-3m, 100um max defocus dia.)
For each Munsell spectrum
Calculate average deblurring error when a white spectrum is used for deblurring
Spectral Focal Sweep: ICCP 2010

27. Average Deblurring Error
Input Image:
Munsell color PSFs:
White PSF at center depth:
PSF Distance Metric:
[Zhou and Nayar ‘09]
Average Deblurring Error:
Averaged over many images
Spectral Focal Sweep: ICCP 2010

28. Munsell Spectra PSFs Most Munsell spectra show little PSF variation
Corrected lens 1 15 x
depth
Most Munsell spectra show little PSF variation
Spectral Focal Sweep: ICCP 2010

29. Average Deblurring Error (%95 of Munsell spectra)
Most Munsell PSFs introduce negligible deblurring artifacts
Spectral Focal Sweep: ICCP 2010

30. Doublet SFS Implementation
Zemax Design
Implementation
Edmund Optics #48184
1” Dia. 100mm FL
Plano-convex asphere
Thorlabs #LE1929
1” Dia. 300mm FL
positive meniscus
Spectral Focal Sweep: ICCP 2010

31. Spectral Focal Sweep: ICCP 2010
Experimental Setup
Corrected Lens
SFS Doublet Lens
B&W Examples
(Basler 1/3” sensor)
F/4 SFS 75mm
EFL Doublet
EFL Cosmicar
F/1.4 75mm
Color Examples
(Canon 450D sensor)
F/4 SFS 75mm
EFL Doublet
F/ mm
EFL Cannon
Spectral Focal Sweep: ICCP 2010

32. Measured SFS PSFs (white spectrum)
depth
Cosmicar
SFS lens
Measured SFS PSF approximately depth-invariant
Spectral Focal Sweep: ICCP 2010

33. Spectral Focal Sweep: ICCP 2010
Deblurring
Deblur with measured white PSF
B&W Images
Wiener deconvolution
Color Images
Transform from RGB to YUV space
Wiener deconvolution of Y image
Transform back to RGB space
Spectral Focal Sweep: ICCP 2010

34. Spectral Focal Sweep: ICCP 2010
EDOF Examples
Spectral Focal Sweep: ICCP 2010

35. Spectral Focal Sweep: ICCP 2010
F/4 Corrected Lens
8ms exposure
Spectral Focal Sweep: ICCP 2010

36. Spectral Focal Sweep: ICCP 2010
F/4 SFS Camera
(Captured)
8ms exposure
Spectral Focal Sweep: ICCP 2010

37. Spectral Focal Sweep: ICCP 2010
F/4 SFS Camera
(Deblurred)
8ms exposure
Spectral Focal Sweep: ICCP 2010

38. Spectral Focal Sweep: ICCP 2010
F/4 Corrected Lens
Background
Foreground
8ms exposure
Spectral Focal Sweep: ICCP 2010

39. Spectral Focal Sweep: ICCP 2010
F/4 SFS Lens
(Captured)
8ms exposure
Spectral Focal Sweep: ICCP 2010

40. Spectral Focal Sweep: ICCP 2010
F/4 SFS Lens
(Deblurred)
8ms exposure
Spectral Focal Sweep: ICCP 2010

41. Spectral Focal Sweep: ICCP 2010
F/4 Corrected Lens
Foreground
Background
16ms exposure
Spectral Focal Sweep: ICCP 2010

42. Spectral Focal Sweep: ICCP 2010
F/4 SFS Lens
(Captured)
16ms exposure
Spectral Focal Sweep: ICCP 2010

43. Spectral Focal Sweep: ICCP 2010
F/4 SFS Lens
(Deblurred)
16ms exposure
Spectral Focal Sweep: ICCP 2010

44. Spectral Focal Sweep: ICCP 2010
F/4 Corrected Lens
Background
Foreground
16ms exposure
Spectral Focal Sweep: ICCP 2010

45. Spectral Focal Sweep: ICCP 2010
F/4 SFS Lens
(Captured)
16ms exposure
Spectral Focal Sweep: ICCP 2010

46. Spectral Focal Sweep: ICCP 2010
F/4 SFS Lens
(Deblurred)
16ms exposure
Spectral Focal Sweep: ICCP 2010

47. Spectral Focal Sweep: ICCP 2010
F/4 Corrected Lens
Background
Foreground
16ms exposure
Spectral Focal Sweep: ICCP 2010

48. Spectral Focal Sweep: ICCP 2010
F/4 SFS Lens
(Captured)
16ms exposure
Spectral Focal Sweep: ICCP 2010

49. Spectral Focal Sweep: ICCP 2010
F/4 SFS Lens
(Deblurred)
16ms exposure
Spectral Focal Sweep: ICCP 2010

50. Spectral Focal Sweep: ICCP 2010
F/4 Corrected Lens
Background
Foreground
16ms exposure
Spectral Focal Sweep: ICCP 2010

51. Spectral Focal Sweep: ICCP 2010
F/4 SFS Lens
(Captured)
16ms exposure
Spectral Focal Sweep: ICCP 2010

52. Spectral Focal Sweep: ICCP 2010
F/4 SFS Lens
(Deblurred)
16ms exposure
Spectral Focal Sweep: ICCP 2010

53. Spectral Focal Sweep: ICCP 2010
Limitations
Broadband assumption does not always hold (e.g. %5 of Munsell Spectra)
deblurring artifacts introduced
Color deblurring method does not correct blur in chrominance channels
Spectral Focal Sweep: ICCP 2010

54. Spectral Focal Sweep: ICCP 2010
Conclusion
Broadband
spectra
Chromatic
aberrations
B&W
sensor
Focus
sweep + =
Works exactly for B&W scenes
Works well for many naturally occurring spectra
Results for color images “look” good
Easy to build EDOF camera
Spectral Focal Sweep: ICCP 2010