1. Modelling, calibration and correction of nonlinear illumination-dependent fixed pattern noise in logarithmic CMOS image sensors
Dileepan Joseph and Steve Collins
Department of Engineering Science
University of Oxford, England
Dileepan Joseph is a PhD student at Keble College and Dr Steve Collins is a University Lecturer based in University College.

2. Outline Logarithmic image sensors Pixel modelling
Fixed pattern noise (FPN)
Sensor calibration
Image correction
Summary and conclusions
This presentation was given to the IEEE Instrumentation and Measurement Technology Conference in Budapest on May 22, 2001.
May 21-23, 2001
IMTC, Budapest (IEEE)

3. Logarithmic image sensors
CMOS versus CCD image sensors
Electronics placed on same die as pixels
Cost, power consumption, size, weight
Quantum efficiency, yield, price pressure
Logarithmic versus linear pixels
Continuous sensing, random access
High dynamic range, low bit rate
Resolution, sensitivity, frame rate
The electronics industry expects CMOS image sensors to gradually replace CCD image sensors, which have been dominant for nearly three decades. Fundamentally, CMOS pixels scale well with shrinking process geometries because more electronics may be placed in each pixel to improve the output without affecting sensitivity or resolution.
A defect in one pixel of a CCD sensor may wipe out a column, making the chip unusable, whereas a similar defect in a CMOS sensor affects only the pixel and may be corrected. CCD sensors need specialised processes offered by a few foundries whereas CMOS sensors may be fabricated by many foundries, including those that manufacture microprocessors.
Logarithmic sensors are non-integrating so pixels may be accessed randomly in space and time, which permits a trade off between frame size and speed. Over five decades of illuminance, ten bits of resolution are sufficient to sense illuminance with one percent accuracy, with a logarithmic sensor, whereas 23 bits are necessary to accomplish the same task, with a linear sensor.
A 10 bit linear sensor could adapt over a 23 bit dynamic range by aperture or integration-time control but would still saturate for high dynamic range scenes. Most other methods to extend the dynamic range of image sensors result in decreased resolution, sensitivity or frame rate.
May 21-23, 2001
IMTC, Budapest (IEEE)

4. Pixel modelling Physical model Abstract model May 21-23, 2001
Before light reaches the photodiode in a pixel, it is attenuated due to absorption and reflection by the aperture and lens of the camera. Photons absorbed by the photodiode form electron-hole pairs that are swept out by the electric field across the device to produce a current.
The photodiode is reverse biased to prevent any current flowing to ground through it except for the photocurrent. However, a small leakage current also flows through the diode. The total current sets the gate voltage of transistor T2 via the diode-connected load transistor T1. Designed to operate in the subthreshold region, T1 has a logarithmic current-to-voltage relationship.
Transistors T2 and T4 form a source follower, when switch T3 is on, permitting the indicated pixel to drive the column line and set the gate voltage of transistor T5. Likewise, transistors T5 and T7 form a source follower, when switch T6 is on, permitting the indicated column to drive the output line and analogue-to-digital converter (ADC).
The complexity of the physical model, including the ADC, may be summarised by a logarithmic relationship with three abstract parameters, named the offset, gain and bias, and an error term.
May 21-23, 2001
IMTC, Budapest (IEEE)

5. Fixed pattern noise (FPN)
Offset (aj) variation (1  j  N pixels)
Offset and gain (bj) variation
Offset, gain and bias (cj) variation
The biggest problem in a logarithmic image sensor is fixed pattern noise (FPN), which is a distortion of the image due to variations in device parameters across the sensor. Work reported in the literature generally assumes the FPN is independent of illumination.
A variation from pixel to pixel of device parameters leads to a variation of the offset, gain, bias or a combination thereof. Therefore, by determining the abstract parameters that describe the response of each pixel, FPN may be corrected with calibration. Furthermore, the statistics of the error between the actual and modelled response (denoted with a caret) must be determined.
Since the complexity of calibration and correction depends on the number of parameters that vary spatially, no more variables should be introduced than are necessary. Offset variation takes FPN to be independent of illuminance. Offset and gain variation makes FPN a linear function of the logarithm of illuminance. Finally, offset, gain and bias variation models a nonlinear dependence of FPN on illuminance.
May 21-23, 2001
IMTC, Budapest (IEEE)

6. Sensor calibration
Calibrate the sensor with images yij of M uniform illuminances xi (e.g. white paper)
Extract parameters by minimising the mean square error
Experiments were done using a 512 by 512 pixel logarithmic image sensor, which is available commercially. Images of white paper under uniform illumination provided calibration data. The iris ring was rotated, thereby changing the aperture, to vary the illumination reaching the focal plane.
Parameters of the offset model, offset and gain model and offset, gain and bias model were extracted by minimising the mean square error (MSE) between the actual response and the modelled response over the variables. The figure compares the actual and modelled responses for two pixels.
While the offset model fits the top pixel’s response well, it does not fit the bottom pixel’s response except for an intersection in the middle. The offset and gain model matches the response slopes of both pixels but intersects each response twice. The offset, gain and bias model has no problem following the curved responses of both pixels.
May 21-23, 2001
IMTC, Budapest (IEEE)

7. Sensor calibration cont’d
Calibration error is 3.9, 1.9 and 0.9 for one, two and three parameter models
Calibration error versus illuminance differs markedly
After calibration, the standard deviation of the error is a useful measure by which to compare the three FPN models. The square of the standard deviation is like the MSE except that the denominator equals the degrees of freedom (DOF), providing an unbiased estimate. The DOF is the number of constraints minus the number of variables fitted to the same constraints.
When the standard deviation of the error is shown versus illuminance, indicated by the average response of all pixels, the models differ markedly. The one parameter model has a minimum error in the middle. The two parameter model has a maximum flanked by two minima. These features correspond to the intersections shown on the previous slide.
In contrast, the three parameter model has a flat error of less than one count (or LSB). The small but sharp rises at the sides may be due to parameter overfitting, which would favour the midrange of data. These results suggest that the nonlinear model fits the data very well.
May 21-23, 2001
IMTC, Budapest (IEEE)

8. Image correction
Left to right: original plus one, two and three parameter FPN corrected images
Top to bottom: apertures of 1.8, 4, 8 and 16 f-stops
Inter-scene plus intra-scene dynamic range equals 67 dB
The montage shows four scenes without and with FPN correction, using the one, two and three parameter models. The four scenes are really one scene made darker by changing the aperture from wide open to nearly closed. The inter-scene and intra-scene dynamic range is 38 dB and 29 dB respectively.
As the scenes are the same going from top to bottom except getting darker, an ideal logarithmic image sensor would give the same response with a progressively more negative offset. Since the histogram of each image has been stretched linearly to fit the 8-bit range of the display, images in a column would look identical for an ideal sensor.
As seen in the montage, three parameter correction gives good results and is better than one or two parameter correction (consider the tone of the door, for example). Similarly, two parameter correction gives better results than one. Note that image quality is poor without FPN correction.
May 21-23, 2001
IMTC, Budapest (IEEE)

9. Image correction cont’d
A close-up of the two and three parameter corrected images, for an aperture of 16 f-stops, shows the improvement in image quality with nonlinear FPN correction. Illuminance in this scene corresponds to the crossover between colour and night vision in the human eye. Because of a low signal level in dim lighting, temporal noise in the sensor is visible after FPN correction.
May 21-23, 2001
IMTC, Budapest (IEEE)

10. Summary and conclusions
Physical and abstract pixel model
Offset, gain, bias and error
Parameter variation causes FPN
Calibration with uniform illuminance
Results indicate FPN is nonlinear
FPN correction is necessary
Digital correction of images
More robust analogue circuits
The response of a logarithmic CMOS pixel to illuminance may be modelled with numerous physical parameters but can be abstracted by a logarithmic function with only three parameters---an offset, gain and bias. A spatial variation of these parameters leads to fixed pattern noise (FPN).
Offset variation, offset and gain variation and offset, gain and bias variation were considered as possible causes of FPN. Bias variation makes the FPN a nonlinear function of illuminance. Using mean square error minimisation, the model parameters may be extracted from images of uniform illuminance.
Experimental results of calibration and correction validate the nonlinear model. The error versus illuminance differs markedly between one, two and three parameter calibration. The one and two parameter models have v-shaped and w-shaped error curves, proving they are poor models, whereas the three parameter model has an error that is independent of illuminance.
Whether digital correction proves to be a practical approach to correct FPN remains to be seen. Nonetheless, while existing analogue methods to correct offset variation, such as correlated double sampling and delta difference sampling, are useful, they are inadequate. The nonlinear effect of offset, gain and bias variation on FPN requires more robust circuits or digital correction.
May 21-23, 2001
IMTC, Budapest (IEEE)

11. Acknowledgements
Many thanks to the Natural Sciences and Engineering Research Council of Canada and the Engineering and Physical Sciences Research Council of Britain for their generous support.
More details on this subject may be found in the following paper:
D Joseph and S Collins, “Modelling, calibration and correction of nonlinear illumination-dependent fixed pattern noise in logarithmic CMOS image sensors”, Proceedings of the IEEE Instrumentation and Measurement Technology Conference, May, 2001.
May 21-23, 2001
IMTC, Budapest (IEEE)