1. Modelling, calibration and rendition of colour logarithmic CMOS image sensors Dileepan Joseph and Steve Collins Department of Engineering Science University of Oxford

2. IMTC 20022 May 21-23 Outline Logarithmic CMOS image sensors Modelling sensor response Image sensor calibration  Fixed pattern noise  Sensation of colour Rendition of images  CIE Lab (perceptual error)  IEC sRGB (standard display) Summary and future work

3. IMTC 20023 May 21-23 Logarithmic CMOS image sensors CMOS displacing CCD because of integration of signal processing and economies of scale Logarithmic sensors offer high dynamic range and high frame rate Linear sensors offer low fixed pattern noise and good colour rendition Example images taken from IMS Chips website Linear CCD sensor Logarithmic sensor Logarithmic sensor

4. IMTC 20024 May 21-23 Modelling sensor response Since I k = ∫ f k (λ) s(λ) dλ  For photocurrent I k, spectral response f k (λ) and light stimulus s(λ) at a pixel, where k = R, G or B And f k (λ) = g L (λ) g k (λ) g P (λ)  For spectral responses of lens g L (λ), colour filter g k (λ) and photodiode g P (λ)  Approximating a linear combination of three CIE XYZ basis functions Then I k = d k x  For mask coefficients d k and tricolour vector x, i.e. s(λ) in CIE XYZ space Ideally, y = a + b ln (c + I k ) + ε  For digital response y of pixel with offset a, gain b, bias c and error ε  Pixel-to-pixel variation of a, b or c causes fixed pattern noise (FPN)

5. IMTC 20025 May 21-23 FPN calibration Three types of FPN of interest:  Offset variation  Offset and gain variation  Offset, gain and bias variation Partition pixels by colour filter to permit FPN calibration of three monochromatic sensors Take images of uniform stimuli under different illuminances Calibrate each pixel’s response to average response of all pixels by least squares estimation of varying model parameters Fuga 15RGB sensor exhibits offset, gain and bias variation

6. IMTC 20026 May 21-23 Colour calibration Take and segment images of a standard chart, having patches of known CIE XYZ colour, under different illuminances Calibrate pixel responses to colour by estimating non-varying model parameters (e.g. mask d k ), using estimates of varying parameters Ideal model fails for Fuga 15RGB because absolute relationship between y and I k invalid (strong inversion component?) Empirical model y = a + b ln (c + (α + d k x) β ) worked well, with no change to relative responses of pixels or FPN calibration

7. IMTC 20027 May 21-23 Image rendition (CIE Lab) Images of a Macbeth Colour Chart, taken by the Fuga 15RGB, were rendered into CIE Lab space with the calibrated empirical model The perceptual error increases in dim lighting as the bias term c dominates the photocurrent I k Excluding the dimmest image (i.e. 5 lux), the error equals 12 over a 60 dB dynamic range for offset, gain and bias variation Images in Digital Photographer show that conventional (linear) digital cameras have an error of 15 over a 30 dB dynamic range

8. IMTC 20028 May 21-23 Image rendition (IEC sRGB) A Fuga 15RGB image of the Macbeth Chart, taken in 11 lux of illuminance, was rendered into IEC sRGB space with the calibrated empirical model Results for offset variation (top- left), offset and gain variation (top- right), offset, gain and bias variation (bottom-left) and true colours (bottom-right) are shown Two types of residual deviation for the rendered patches are visible:  Fixed pattern noise  Colour desaturation

9. IMTC 20029 May 21-23 Summary and future work Logarithmic image sensors offer high dynamic range and frame rate Combine theories of colour linear sensors and monochromatic logarithmic sensors to model colour logarithmic sensors Calibrate FPN, using images of uniform stimuli, by relative estimation of model parameters that vary from pixel to pixel Calibrate colour, using images of a colour chart, by absolute estimation of model parameters that do not vary Fuga 15RGB results expose limitations of ideal model in absolute estimation but reveal empirical model that works well Macbeth Chart results show colour rendition with calibrated Fuga 15RGB competes with conventional digital cameras Seek to minimise bias variation, so simple FPN models suffice, and bias magnitude, to improve colour rendition in dim lighting

10. IMTC 200210 May 21-23 Acknowledgements The authors gratefully acknowledge the support of the Natural Sciences and Engineering Research Council (Canada) and the Engineering and Physical Sciences Research Council (UK)