Noise Estimation from a Single Image Ce Liu William T. FreemanRichard Szeliski Sing Bing Kang

Parameter Tweaking in Computer Vision  Computer vision algorithms suffer from hand tuning parameters for particular images or image sequences  We want vision algorithms that behave properly under varying lighting conditions, blur levels and noise levels  Our work is one step in that direction  Given an image, estimate the noise level  Modify vision algorithms to be independent of noise

Image Noise Is Important in Vision  In image denoising the noise is assumed to be known as Additive Gaussian White Noise (AWGN)  However, in real applications the noise is unknown and non-additive  Many other computer vision algorithms also explicitly or implicitly assume the type and level of image noise  Hard to make vision algorithms fully automatic without knowing noise

Noise Level Function (NLF)  The standard deviation of noise  is a function of image brightness I  Measurable by fixing the camera and taking multiple shots of a static scene  For each pixel:  Mean: I  Standard deviation:   NLF depends on camera, ISO, shutter speed, aperture  Our goal is to estimate NLF from a single image  How to estimate noise without separating noise and signal? I 

An Example Image

Piecewise Smooth Image Prior Patch =+ SignalResidual For each RGB channel: Brightness mean I Standard deviation  RedGreenBlue I  I  I  Brightness Standard deviation Affine model

Piecewise Smooth Image Prior Patch =+ SignalResidual RedGreenBlue I  I  I  Brightness Standard deviation

Piecewise Smooth Image Prior Patch =+ SignalResidual RedGreenBlue I  I  I  Brightness Standard deviation

Segmentation-based Approach Observed image

Segmentation-based Approach Over- segmentation

Segmentation-based Approach Signal

Segmentation-based Approach Residual= noise + unmodelled image variation

Estimate NLFs  Assume brightness mean I is accurate estimate  Standard deviation  is an over-estimate: (may contain signal)  The lower envelope is the upper bound of NLF  III Brightness Residual std. dev.

Issues  Should the curve be strictly and tightly below the points?  III Brightness Residual std. dev.

Issues  Should the curve be strictly and tightly below the points?  How to handle the missing data?  III Brightness Residual std. dev.

Issues  Should the curve be strictly and tightly below the points?  How to handle the missing data?  Correlation between RGB channels?  III Brightness Residual std. dev.

Solutions  Formulate the inference problem in a probabilistic framework  Learn the prior of noise level functions  III Brightness Residual std. dev.

Outline  Over-segmentation and per-segment variance analysis  Learning the priors of noise level functions (NLF)  Synthesize CCD noise  Sample noise level functions  Learn the prior of noise level functions  Inference: estimate the upper bound of NLF  Bayesian MAP to estimate NLFs for RGB channels  Applications  Adaptive bilateral filtering  Canny edge detection

 Noise model  Camera response function (CRF) f: download from Columbia camera response function database (used 196 typical CRFs) Tsin et. al. Statistical calibration of CCD image process. ICCV, 2001 Camera Noise Atmospheric Attenuation Lens/ geometric Distortion CCD Imaging/ Bayer Pattern Fixed Pattern Noise Shot Noise Thermal Noise Interpolation/ Demosaic White Balancing Gamma Correction A/D Converter Dark Current Noise t Quantization Noise Scene Radiance Digital Image Camera Irradiance I L Dependent noise: Independent noise:

Synthesize CCD Noise Estimate NLF I Camera response function: f Dependent noise: Independent noise:

Sample NLFs by Varying the Parameters Camera response function (CRF) f Dependent noise: Independent noise:

The Prior of NLFs

Likelihood Function  The estimated standard deviation should be probabilistically bigger than and close to the true value  Bayesian MAP inference I

Validation (1): Synthetic Noise  Add synthetic CCD noise, estimate, compare to the ground truth — ground truth estimated ——————

Validation (2): Measure NLF of a Real Camera  29 images were taken under the same settings (the camera is not in the database for training)  The real NLF is obtained by computing mean and variance per pixel

Validation (3): Robustness Test  Verify that different images from the same camera give the same estimated NLF (camera not in the database for training)

Application (1): Adaptive Bilateral Filtering  Bilateral filter is an edge-preserving low-pass filter  Spatial sigma and range sigma  Adaptive bilateral filter  Down-weigh RGB values by signal and noise covariance matrices  The range sigma is set to be a function of the estimated standard deviation of the noise From Durand and Dorsey, SIGGRAPH 02 Input noisy imageSmoothing kernelDenoised image

Test on Low and High Noise low noise high noise RedGreenBlueRedGreenBlue

Results—Adaptive Bilateral Filtering Standard bilateral filtering Adaptive bilateral filtering low noise high noise

Results—Adaptive Bilateral Filtering Zoom in high noise Standard bilateral filtering Adaptive bilateral filtering

Application (2): Canny Edge Detection low noise high noise RedGreenBlueRedGreenBlue

Results—Canny Edge Detection low noise high noise Parameters adapted in MATLAB Parameters adapted by estimated noise

Conclusion  Piecewise-smooth image prior model to estimate the upper bound of noise level function (NLF)  Estimate the space of NLF by simulating CCD camera on the existing CRF database  Upper bounds are verified by both synthetic and real experiments  An important step to automate vision algorithms independent of noise

Thank you! Ce Liu William T. Freeman CSAIL MIT Rick Szeliski Sing Bing Kang Microsoft Research Noise Estimation from a Single Image