1. Frequency domain methods for demosaicking of Bayer sampled color images Eric Dubois

2. Frequency-domain Bayer demosaicking
Problem Statement
Problem: Most digital color cameras capture only one color component at each spatial location. The remaining components must be reconstructed by interpolation from the captured samples. Cameras provide hardware or software to do this, but the quality may be inadequate.
Objective: Develop new algorithms to interpolate each color plane (called demosaicking) with better quality reconstruction, and with minimal computational complexity.
Frequency-domain Bayer demosaicking

3. Frequency-domain Bayer demosaicking
Retinal Cone Mosaic
The human visual system must solve a similar problem!
Frequency-domain Bayer demosaicking

4. Construction of color image from color planes+
Frequency-domain Bayer demosaicking

5. Lighthouse
original

6. Lighthouse
red original

7. Lighthouse
green original

8. Lighthouse
blue original

9. Formation of Color planes
Frequency-domain Bayer demosaicking

10. Lighthouse
red subsampled

11. Lighthouse
green subsampled

12. Lighthouse
blue subsampled

13. Lighthouse
Bayer CFA image

14. Color plane interpolation
Green channel: bilinear interpolation GA GL GR GI GB
Frequency-domain Bayer demosaicking

15. Color plane interpolation
Red channel: bilinear interpolation
RNW
RNE RC
RSW
RSE RS
Frequency-domain Bayer demosaicking

16. Lighthouse
red interpolated

17. Lighthouse
green interpolated

18. Lighthouse
blue interpolated

19. Lighthouse
Interpolated color image

20. Lighthouse
original

21. Frequency-domain Bayer demosaicking
Can we do better?
Color planes have severe aliasing. Better interpolation of the individual planes has little effect.
Frequency-domain Bayer demosaicking

22. Lighthouse
red interpolated
with bilinear interpolator

23. Lighthouse
red interpolated
with bicubic interpolator

24. Frequency-domain Bayer demosaicking
Can we do better?
Color planes have severe aliasing. Better interpolation of the individual planes has little effect.
We could optically prefilter the image (blur it) so that aliasing is less severe.
Frequency-domain Bayer demosaicking

25. Lighthouse
red interpolated
with bilinear interpolator

26. Lighthouse
prefiltered red interpolated
with bilinear interpolator

27. Lighthouse
Interpolated color image

28. Lighthouse Prefiltered & Interpolated color image

29. Lighthouse original

30. Frequency-domain Bayer demosaicking
Can we do better?
Color planes have severe aliasing. Better interpolation of the individual planes has little effect.
We could optically prefilter the image (blur it) so that aliasing is less severe.
We can process the three color planes together to gather details from all three components.
Frequency-domain Bayer demosaicking

31. Frequency-domain Bayer demosaicking
Can we do better?
There have been numerous papers and patents describing different algorithms to interpolate the color planes – they all work on the three planes together, exploiting the correlation between the three components.
Gunturk et al. published an extensive survey in March The best methods were the projection on convex sets (POCS) algorithm (lowest MSE) and the adaptive homogeneity directed (AHD) algorithm (best subjective quality).
We present here a novel frequency-domain algorithm.
Frequency-domain Bayer demosaicking

32. Spatial multiplexing model
subsampling
multiplexing
Frequency-domain Bayer demosaicking

33. Spatial multiplexing model
Frequency-domain Bayer demosaicking

34. Frequency-domain multiplexing model
Re-arranging the spatial multiplexing expression
Frequency-domain Bayer demosaicking

35. Frequency-domain multiplexing model
David Alleysson, EPFL
Frequency-domain Bayer demosaicking

36. Luma and chrominance components
Frequency-domain Bayer demosaicking

37. Luma and chrominance components
Luma fL
Chroma_1 fC1
Chroma_2 fC2
Frequency-domain Bayer demosaicking

38. Lighthouse Bilinearly
Interpolated color image

39. Frequency-domain demosaicking algorithm
Extract modulated C1 using a band-pass filter at (0.5,0.5) and demodulate to baseband
Extract modulated C2 using band-pass filters at (0.5,0.0) and (0.0, 0.5), demodulate to baseband, and combine in some suitable fashion (the key)
Subtract modulated C1 and remodulated C2 from the CFA to get the estimated luma component L.
Matrix the L, C1 and C2 components to get the RGB representation.
Frequency-domain Bayer demosaicking

40. Frequency-domain Bayer demosaicking
Spectrum of CFA signal b a
Frequency-domain Bayer demosaicking

41. Using C2a only
Using C2b only

42. Frequency-domain Bayer demosaicking
Demosaicking using C2a only or C2b only -- details
Original
From C2a only
From C2b only
Frequency-domain Bayer demosaicking

43. Demosaicking Block Diagram
h2a
h2b + -
fCFA 
(-1)n1+n2
-(-1)n2
(-1)n1 h1
combine
(-1)n1-(-1)n2
matrix fR fG fB
fC2am
fC2bm
fC1m
fC1
fC2a
fC2b
fC2 fL
Frequency-domain Bayer demosaicking

44. Frequency-domain Bayer demosaicking
Spectrum of CFA signal b a
Frequency-domain Bayer demosaicking

45. Frequency-domain Bayer demosaicking
Design Issues
How to choose the filters h1, h2a and h2b
Frequency domain design methods
Least-squares design methods
Size of the filters
How to combine the two estimates and
Choice of features to guide weighting
The two above issues may be inter-related.
Frequency-domain Bayer demosaicking

46. Frequency-domain Bayer demosaicking
Filter design
Gaussian filters (Alleysson)
Window design or minimax design
Define ideal response, with pass, stop and transition bands
Approximate using the window design method
Refine using minimax or least pth optimization
Can design low-pass filters and modulate to the center frequency
Frequency-domain Bayer demosaicking

47. Frequency-domain Bayer demosaicking
Filter specification
u v val
Frequency-domain Bayer demosaicking

48. Ideal response – perspective view
Frequency-domain Bayer demosaicking

49. Ideal response – contour plot
Frequency-domain Bayer demosaicking

50. Window design – perspective view
Frequency-domain Bayer demosaicking

51. Window design – contour plot
Frequency-domain Bayer demosaicking

52. Least pth filter – perspective view
Frequency-domain Bayer demosaicking

53. Least pth filter – contour plot
Frequency-domain Bayer demosaicking

54. 21 x 21 filters in SPL published algorithm
h2a
h2b h1 u v

55. Adaptive weighting of C2a and C2b
We want to form the estimate of C2 by choosing the best between C2a and C2b, or perhaps by a weighted average. We have used
should be near 1 when C2a is the best choice, and near 0 when C2b is the best choice
Frequency-domain Bayer demosaicking

56. Typical scenarios for local spectrum
C2b C1
C2b C1 C1 C1 L L
C2a
C2a
C2a
C2a u u C1 C1 C1
C2b C1
C2b v v
A: C2a is better estimate
B: C2b is better estimate
Frequency-domain Bayer demosaicking

57. Scenario A
Scenario B

58. Typical scenarios for local spectrum
C2b C1
C2b C1 C1 C1
C2a L L
C2a
C2a
C2a u u C1 C1 C1
C2b C1
C2b v v
A: C2a is better estimate
B: C2b is better estimate
Frequency-domain Bayer demosaicking

59. Weight selection strategy
Scenario A: average local energy near (fm, 0) is smaller than near (0, fm ).
Scenario B: average local energy near (0, fm ) is smaller than near (fm, 0).
Let be a measure of the average local energy near (fm, 0), and be a measure of the average local energy near (0, fm ).
Frequency-domain Bayer demosaicking

60. Gaussian filters for local energy measurement
fm = 0.375 v
Frequency-domain Bayer demosaicking

61. Frequency-domain Bayer demosaicking
Results
Results with this adaptive frequency-domain demosaicking method were published in IEEE Signal Processing Letters in Dec All filters were of size 21 x 21. Filters h1, h2a and h2b were designed with the window method, with band parameters determined by trial and error. The method gave the lowest mean-square reconstruction error on the standard set of Kodak test images compared to other published methods.
Frequency-domain Bayer demosaicking

62. Mean square error comparison
Frequency-domain Bayer demosaicking