Frequency domain methods for demosaicking of Bayer sampled color images Eric Dubois

Frequency-domain Bayer demosaicking Problem Statement Problem: Most digital color cameras capture only one color component at each spatial location. The remaining components must be reconstructed by interpolation from the captured samples. Cameras provide hardware or software to do this, but the quality may be inadequate. Objective: Develop new algorithms to interpolate each color plane (called demosaicking) with better quality reconstruction, and with minimal computational complexity. Frequency-domain Bayer demosaicking

Frequency-domain Bayer demosaicking Retinal Cone Mosaic The human visual system must solve a similar problem! Frequency-domain Bayer demosaicking

Construction of color image from color planes + Frequency-domain Bayer demosaicking

Lighthouse original

Lighthouse red original

Lighthouse green original

Lighthouse blue original

Formation of Color planes Frequency-domain Bayer demosaicking

Lighthouse red subsampled

Lighthouse green subsampled

Lighthouse blue subsampled

Lighthouse Bayer CFA image

Color plane interpolation Green channel: bilinear interpolation GA GL GR GI GB Frequency-domain Bayer demosaicking

Color plane interpolation Red channel: bilinear interpolation RNW RNE RC RSW RSE RS Frequency-domain Bayer demosaicking

Lighthouse red interpolated

Lighthouse green interpolated

Lighthouse blue interpolated

Lighthouse Interpolated color image

Lighthouse original

Frequency-domain Bayer demosaicking Can we do better? Color planes have severe aliasing. Better interpolation of the individual planes has little effect. Frequency-domain Bayer demosaicking

Lighthouse red interpolated with bilinear interpolator

Lighthouse red interpolated with bicubic interpolator

Frequency-domain Bayer demosaicking Can we do better? Color planes have severe aliasing. Better interpolation of the individual planes has little effect. We could optically prefilter the image (blur it) so that aliasing is less severe. Frequency-domain Bayer demosaicking

Lighthouse red interpolated with bilinear interpolator

Lighthouse prefiltered red interpolated with bilinear interpolator

Lighthouse Interpolated color image

Lighthouse Prefiltered & Interpolated color image

Lighthouse original

Frequency-domain Bayer demosaicking Can we do better? Color planes have severe aliasing. Better interpolation of the individual planes has little effect. We could optically prefilter the image (blur it) so that aliasing is less severe. We can process the three color planes together to gather details from all three components. Frequency-domain Bayer demosaicking

Frequency-domain Bayer demosaicking Can we do better? There have been numerous papers and patents describing different algorithms to interpolate the color planes – they all work on the three planes together, exploiting the correlation between the three components. Gunturk et al. published an extensive survey in March 2005. The best methods were the projection on convex sets (POCS) algorithm (lowest MSE) and the adaptive homogeneity directed (AHD) algorithm (best subjective quality). We present here a novel frequency-domain algorithm. Frequency-domain Bayer demosaicking

Spatial multiplexing model subsampling multiplexing Frequency-domain Bayer demosaicking

Spatial multiplexing model Frequency-domain Bayer demosaicking

Frequency-domain multiplexing model Re-arranging the spatial multiplexing expression Frequency-domain Bayer demosaicking

Frequency-domain multiplexing model David Alleysson, EPFL Frequency-domain Bayer demosaicking

Luma and chrominance components Frequency-domain Bayer demosaicking

Luma and chrominance components Luma fL Chroma_1 fC1 Chroma_2 fC2 Frequency-domain Bayer demosaicking

Lighthouse Bilinearly Interpolated color image

Frequency-domain demosaicking algorithm Extract modulated C1 using a band-pass filter at (0.5,0.5) and demodulate to baseband Extract modulated C2 using band-pass filters at (0.5,0.0) and (0.0, 0.5), demodulate to baseband, and combine in some suitable fashion (the key) Subtract modulated C1 and remodulated C2 from the CFA to get the estimated luma component L. Matrix the L, C1 and C2 components to get the RGB representation. Frequency-domain Bayer demosaicking

Frequency-domain Bayer demosaicking Spectrum of CFA signal b a Frequency-domain Bayer demosaicking

Using C2a only Using C2b only

Frequency-domain Bayer demosaicking Demosaicking using C2a only or C2b only -- details Original From C2a only From C2b only Frequency-domain Bayer demosaicking

Demosaicking Block Diagram h2a h2b + - fCFA  (-1)n1+n2 -(-1)n2 (-1)n1 h1 combine (-1)n1-(-1)n2 matrix fR fG fB fC2am fC2bm fC1m fC1 fC2a fC2b fC2 fL Frequency-domain Bayer demosaicking

Frequency-domain Bayer demosaicking Spectrum of CFA signal b a Frequency-domain Bayer demosaicking

Frequency-domain Bayer demosaicking Design Issues How to choose the filters h1, h2a and h2b Frequency domain design methods Least-squares design methods Size of the filters How to combine the two estimates and Choice of features to guide weighting The two above issues may be inter-related. Frequency-domain Bayer demosaicking

Frequency-domain Bayer demosaicking Filter design Gaussian filters (Alleysson) Window design or minimax design Define ideal response, with pass, stop and transition bands Approximate using the window design method Refine using minimax or least pth optimization Can design low-pass filters and modulate to the center frequency Frequency-domain Bayer demosaicking

Frequency-domain Bayer demosaicking Filter specification u v val 0.000 0.00 1.0 0.110 0.00 1.0 0.110 0.02 1.0 0.000 0.10 1.0 0.030 0.10 1.0 0.070 0.06 1.0 0.338 0.00 0.0 0.338 0.05 0.0 0.050 0.36 0.0 0.000 0.36 0.0 0.184 .205 0.0 0.500 0.00 0.0 0.000 0.50 0.0 0.500 0.50 0.0 Frequency-domain Bayer demosaicking

Ideal response – perspective view Frequency-domain Bayer demosaicking

Ideal response – contour plot Frequency-domain Bayer demosaicking

Window design – perspective view Frequency-domain Bayer demosaicking

Window design – contour plot Frequency-domain Bayer demosaicking

Least pth filter – perspective view Frequency-domain Bayer demosaicking

Least pth filter – contour plot Frequency-domain Bayer demosaicking

21 x 21 filters in SPL published algorithm h2a h2b h1 u v

Adaptive weighting of C2a and C2b We want to form the estimate of C2 by choosing the best between C2a and C2b, or perhaps by a weighted average. We have used should be near 1 when C2a is the best choice, and near 0 when C2b is the best choice Frequency-domain Bayer demosaicking

Typical scenarios for local spectrum C2b C1 C2b C1 C1 C1 L L C2a C2a C2a C2a u u C1 C1 C1 C2b C1 C2b v v A: C2a is better estimate B: C2b is better estimate Frequency-domain Bayer demosaicking

Scenario A Scenario B

Typical scenarios for local spectrum C2b C1 C2b C1 C1 C1 C2a L L C2a C2a C2a u u C1 C1 C1 C2b C1 C2b v v A: C2a is better estimate B: C2b is better estimate Frequency-domain Bayer demosaicking

Weight selection strategy Scenario A: average local energy near (fm, 0) is smaller than near (0, fm ). Scenario B: average local energy near (0, fm ) is smaller than near (fm, 0). Let be a measure of the average local energy near (fm, 0), and be a measure of the average local energy near (0, fm ). Frequency-domain Bayer demosaicking

Gaussian filters for local energy measurement fm = 0.375 v Frequency-domain Bayer demosaicking

Frequency-domain Bayer demosaicking Results Results with this adaptive frequency-domain demosaicking method were published in IEEE Signal Processing Letters in Dec. 2005. All filters were of size 21 x 21. Filters h1, h2a and h2b were designed with the window method, with band parameters determined by trial and error. The method gave the lowest mean-square reconstruction error on the standard set of Kodak test images compared to other published methods. Frequency-domain Bayer demosaicking

Mean square error comparison Frequency-domain Bayer demosaicking