Acquisition and display of a still color image A-Z CEG4311 Case Study Acquisition and display of a still color image A-Z

Objective Acquire still color images with a 4:3 single-sensor 2400 X 1800 (4.3 MPixel) Bayer digital camera. Store the images in JPEG format in sRGB color space, and display on a 1024 X 768 plasma display. The objective is to maintain maximum image quality and color fidelity.

Issues in the system design Characterization of the acquisition process Interpolation of the color planes Conversion to sRGB Conversion to 1024 X 768 (resampling) JPEG compression Display process

Acquisition: Bayer CFA X x X y

Acquisition model Let the red, green and blue filters have spectral transmission curves The output can be modelled by Irradiance on the sensor:

Acquisition model (2)

RGB sensitivities of a typical Bayer CFA camera

Spatial multiplexing model subsampling multiplexing

Frequency-domain multiplexing model Re-arranging the spatial multiplexing expression

Frequency-domain multiplexing model David Alleysson, EPFL

Luma and chrominance components

Frequency-domain demosaicking algorithm Extract modulated C1 using a band-pass filter at (0.5,0.5) and demodulate to baseband Extract modulated C2 using band-pass filters at (0.5,0.0) and (0.0, 0.5), demodulate to baseband, and combine in some suitable fashion (the key) Subtract modulated C1 and remodulated C2 from the CFA to get the estimated luma component L. Matrix the L, C1 and C2 components to get the RGB representation.

Using C2a only Using C2b only

Demosaicking using C2a only or C2b only -- details Original From C2a only From C2b only

Demosaicking Block Diagram h2a h2b + - fCFA  (-1)n1+n2 -(-1)n2 (-1)n1 h1 combine (-1)n1-(-1)n2 matrix fR fG fB fC2am fC2bm fC1m fC1 fC2a fC2b fC2 fL

Spectrum of CFA signal b a

Design Issues How to choose the filters h1, h2a and h2b Frequency domain design methods Least-squares design methods Size of the filters How to combine the two estimates and Choice of features to guide weighting The two above issues may be inter-related.

Filter design Gaussian filters (Alleysson) Window design or minimax design Define ideal response, with pass, stop and transition bands Approximate using the window design method Refine using minimax or least pth optimization Can design low-pass filters and modulate to the center frequency

21 x 21 filters in SPL published algorithm h2a h2b h1 u v

Adaptive weighting of C2a and C2b We want to form the estimate of C2 by choosing the best between C2a and C2b, or perhaps by a weighted average. We have used should be near 1 when C2a is the best choice, and near 0 when C2b is the best choice

Typical scenarios for local spectrum C2b C1 C2b C1 C1 C1 L L C2a C2a C2a C2a u u C1 C1 C1 C2b C1 C2b v v A: C2a is better estimate B: C2b is better estimate

Scenario A Scenario B

Typical scenarios for local spectrum C2b C1 C2b C1 C1 C1 C2a L L C2a C2a C2a u u C1 C1 C1 C2b C1 C2b v v A: C2a is better estimate B: C2b is better estimate

Weight selection strategy Scenario A: average local energy near (fm, 0) is smaller than near (0, fm ). Scenario B: average local energy near (0, fm ) is smaller than near (fm, 0). Let be a measure of the average local energy near (fm, 0), and be a measure of the average local energy near (0, fm ).

Gaussian filters for local energy measurement fm = 0.375 v

Convert to sRGB Now we have a 2400 X 1800 RGB image defined on the square lattice with spacing X = 1/1800 ph. These are not tristimulus values! What is sRGB anyway? Link Rec 709 RGB and a gamma correction similar to Rec. 709

Convert to sRGB This is an approximate conversion. We can find a matrix M so that if we image the display, we get the correct tristimulus values!

Convert image size Original image is 2400 X 1800 Display is 1024 X 768

Sampling structure conversion 900 c/ph

Sampling structure conversion 900 c/ph 384 c/ph 384 c/ph

What is the display color space?

Plasma TV

What is the display color space? Close to sRGB! Note that output R, G, B sampling structures are offset.

What about JPEG compression? Convert to Luma/Chrominance YCrCb Downsample CrCb Do JPEG compression as we have studied in detail Can be done on 2400 X 1800 image or on 1024 X 768 image according to application.