1. EE 7700 Demosaicking Problem in Digital Cameras

2. Bahadir K. Gunturk2 Multi-Chip Digital Camera Lens Scene Spectral filters Beam- splitters Sensors To produce a color image, at least three spectral components are needed at each pixel. One approach is to use beam-splitters and multiple chips.

3. Bahadir K. Gunturk3 Single-Chip Digital Camera Multi-chip approach is expensive. Precise chip alignment is required. The alternative is to use a color filter array. Lens Scene Color filter array Sensors

4. Bahadir K. Gunturk4 Single-Chip Digital Camera The missing color samples must be estimated to produce the full color image. Since a mosaic of samples are available, this estimation (interpolation) process is called demosaicking.

5. Bahadir K. Gunturk5 Single-Chip Digital Camera Images suffer from color artifacts when the samples are not estimated correctly. Original imageBilinearly interpolated from CFA-filtered samples

6. Bahadir K. Gunturk6 Demosaicking Approaches Non-Adaptive Single-Channel Interpolation: Interpolate each color channel separately using a standard technique, such as nearest-neighbor interpolation, bilinear interpolation, etc. Edge-Directed Interpolation: Estimate potential edges, avoid interpolating across the edges. 1 2 3 4 x Edge-directed interpolation 1.Calculate horizontal gradient ΔH = |G1 – G2| 2.Calculate vertical gradient ΔV = |G3 – G4| 3.If ΔH > ΔV, Gx = (G3 + G4)/2 Else if ΔH < ΔV, Gx = (G1 + G2)/2 Else Gx = (G1 + G2 + G3 + G4)/4

7. Bahadir K. Gunturk7 Demosaicking Approaches Edge-Directed Interpolation: Based on the assumption that color channels have similar texture, various edge detectors can be used. Edge-directed interpolation 1.Calculate horizontal gradient ΔH = | (R3 + R7)/2 – R5 | 2.Calculate vertical gradient ΔV = | (R1 + R9)/2 – R5 | 3.If ΔH > ΔV, G5 = (G2 + G8)/2 Else if ΔH < ΔV, G5 = (G4 + G6)/2 Else G5 = (G2 + G8 + G4 + G6)/4 1 2 4 56 7 8 9 3

8. Bahadir K. Gunturk8 Demosaicking Approaches Constant-Hue-Based Interpolation: Hue does not change abruptly within a small neighborhood. Interpolate green channel first. Interpolate hue (defined as either color differences or color ratios). Estimate the missing (red/blue) from the interpolated hue. Red Interpolate d Red Interpolate Green Interpolate

9. Bahadir K. Gunturk9 Demosaicking Approaches Edge-Directed Interpolation of Hue: It is a combination of edge-directed interpolation and constant-hue-based interpolation. Hue is interpolated as in constant-hue-based interpolation approach, but this time, hue is interpolated based on the edge directions (as in the edge-directed interpolation algorithm).

10. Bahadir K. Gunturk10 Demosaicking Approaches Using Laplacian For Enhancement: Use the second-order gradients of red/blue channels to enhance green channel. 1 2 4 56 7 8 9 3 1.Calculate horizontal gradient ΔH = |G4 – G6| + |R5 – R3 + R5 – R7| 2.Calculate vertical gradient ΔV = |G2 – G8| + |R5 – R1 + R5 – R9| 3.If ΔH > ΔV, G5 = (G2 + G8)/2 + (R5 – R1 + R5 – R9)/4 Else if ΔH < ΔV, G5 = (G4 + G6)/2 + (R5 – R3 + R5 – R7)/4 Else G5 = (G2 + G8 + G4 + G6)/4 + (R5 – R1 + R5 – R9 + R5 – R3 + R5 – R7)/8

11. Bahadir K. Gunturk11 Aliasing Green channel Red/Blue channel Frequency spectrum of an image: After CFA sampling:

12. Bahadir K. Gunturk12 Demosaicking Approach Alias Cancelling: Based on the assumption that red, green, and blue channels have similar frequency components, the high-frequency components of red and blue channels are replaced by the high-frequency components of green channel. Red/Blue channel

13. Bahadir K. Gunturk13 Experiment Full Red/Green/Blue channels Subband decomposition CFA Sampling Subband decomposition Interpolate LL HL HH LH LL HL LH LL HL HH LH LL HL LH

14. Bahadir K. Gunturk14 Constraint Sets  Detail Constraint Set: Detail subbands of the red and blue channels must be similar to the detail subbands of the green channel. HL HH LH HL LH HH

15. Bahadir K. Gunturk15 Constraint Sets  Observation Constraint Set: Interpolated channels must be consistent with the observed data. CFA Sensors

16. Bahadir K. Gunturk16 HL LH HH Projection Operations  Projection onto the Detail Constraint Set:  Decompose the color channels.  Update the detail subbands of red and blue channels.  Apply synthesis filters to reconstruct back the channels.

17. Bahadir K. Gunturk17 Projection Operations  Projection onto the Observation Constraint Set:  Insert the observed data to their corresponding positions. CFA Sensors

18. Bahadir K. Gunturk18 Alternating Projections Algorithm Samples of color channels Initial interpolation Update Insert the observed data Projection onto the detail constraint set Projection onto the observation constraint set Iteration

19. Bahadir K. Gunturk19 Results OriginalHibbard 1995Laroche and Prescott 1994 Hamilton and Adams 1997 Kimmel 1999Gunturk 2002

20. Bahadir K. Gunturk20 Results Original Hibbard 1995 Laroche and Prescott 1994 Hamilton and Adams 1997 Kimmel 1999 Gunturk 2002

21. Bahadir K. Gunturk21 Previous Methods Gunturk et al, “Demosaicking: Color Filter Array Interpolation in Single-Chip Digital Cameras,” to appear in IEEE Signal Processing Magazine. [Gunturk02]

22. Bahadir K. Gunturk22 References  [Gunturk02] Gunturk et al, “Color Plane Interpolation Using Alternating Projections,” IEEE Trans. Image Processing, 2002.  [Hibbard 1995] R. H. Hibbard, “Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients,” U.S. Patent 5,382,976, January, 1995.  [Laroche and Prescott 1994] C. A. Laroche and M. A. Prescott, “Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients,” U.S. Patent 5,373,322, December, 1994.  [Hamilton and Adams 1997] J. F. Hamilton Jr. and J. E. Adams, “Adaptive color plane interpolation in single sensor color electronic camera,” U.S. Patent 5,629,734, May, 1997.  [Kimmel 1999] R. Kimmel, “Demosaicing: Image reconstruction from CCD samples,” IEEE Trans. Image Processing, vol. 8, pp. 1221-1228, 1999.