Technion - Israel Institute of Technology 1 On Interpolation Methods using Statistical Models RONEN SHER Supervisor: MOSHE PORAT

Technion - Israel Institute of Technology2 Outline Black & White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

Technion - Israel Institute of Technology3 The Interpolation Problem Factor of 2 Input Output

Technion - Israel Institute of Technology4 Image Interpolation Methods Nearest Neighbor Bilinear Bi-Cubic Spline

Technion - Israel Institute of Technology5 Motivations 1: Pixels Correlation Normalized histograms of Lena (gray Levels) 256x256-dashed ; 512x512-solid

Technion - Israel Institute of Technology6 Motivations 2: Image Compression Results Compression rates in bits/sample “Necessary Data”:

Technion - Israel Institute of Technology7 Proposed Approach

Technion - Israel Institute of Technology8 Approaching the problem Near Lossless Compression Scheme Inverse Scheme

Technion - Israel Institute of Technology9 Lossless Compression predictors

Technion - Israel Institute of Technology10 Lossless Compression - Context modeling The error value is subtracted from the average error in a given context Vertical edge Horizontal edge

Technion - Israel Institute of Technology11 Outline Black & White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overview  Components correlation  Statistical extension  Results Summary

Technion - Israel Institute of Technology12 Image Regions In regions of edges, averaging will result in a smoothing effect. The edge must be preserved. The edges exist in the input image and the same distribution is assumed in the larger interpolated image.

Technion - Israel Institute of Technology13 Image Regions In case of a horizontal edge: In case of a vertical edge: Depending on the four surrounding neighbors, there will be at most 4!=24 permutations:

Technion - Israel Institute of Technology14 Pixels fitting From Lena 256x256

Technion - Israel Institute of Technology15 Image Regions In each region a different weighted sum is valid for the prediction The coefficients are learned from the input image

Technion - Israel Institute of Technology16 Outline Black & White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

Technion - Israel Institute of Technology17 Step 1: Coefficients calculation Scanning the Input Image for the ‘x type’ pixel we determine its permutation from its four neighbors and save its value and its neighbors’ values in VM x Modeling only the regions with significant changes in gray levels Same treatment for the ‘+type’ pixels

Technion - Israel Institute of Technology18 Step 1: Coefficients calculation For each permutation we find the four coefficients using the Least Square solution: Same technique for the + coefficients

Technion - Israel Institute of Technology19 Step 2a: ‘x type’ Reconstruction Scanning the sparse Image, for each pixel we determine its matching permutation (coefficients) from its four neighbors and predict its value using

Technion - Israel Institute of Technology20 Step 2b: ‘+ type’ Reconstruction The Input is I x, for each “+” pixel we find its matching permutation (coefficients) and calculate its prediction by

Technion - Israel Institute of Technology21 Experiments - Lena The 4 coefficients in 24 cases of x-type  Lena size 512x512 o Lena size 256x256 Errors α1α1 α4α4 α3α3 α2α2

Technion - Israel Institute of Technology22 Example 1 - B&W images (128x128->256x256) Original Bilinear Bi-Cubic Spline Proposed Bi-Cubic Nearest neighbor (Input)

Technion - Israel Institute of Technology23 Example 2 - B&W images (128x128->256x256) Original Bilinear Bi-Cubic SplineBi-Cubic Nearest neighbor (Input) Proposed

Technion - Israel Institute of Technology24 Outline Black and White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

Technion - Israel Institute of Technology25 One-Dimensional Interpolation Interpolating y d, using NR. Its adjacent samples serve as the four neighbors for the coefficients’ calculation.

Technion - Israel Institute of Technology26 Synthetic Test Signal y1=sin(r.*(5+3.*sin(2.*(r+0.7)))).*sin(7.*(r+0.9)) t1=1,2..N1 r=(t1+OS1)/100 N1=2400 f1=1 Ts=2 OS1=3000 L=2

Technion - Israel Institute of Technology27 1D Interpolation result 1

Technion - Israel Institute of Technology28 1D Interpolation result 2 Voice signal: the word “Diskette”

Technion - Israel Institute of Technology29 Outline Black and White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

Technion - Israel Institute of Technology30 CCD structure

Technion - Israel Institute of Technology31 CCD Demosaicing Methods Bilinear Kimmel - gradient based function and hues R/G,B/G. Gunturk – data consistency and similarity between the high-frequency components. Muresan - interpolates R-G,B-G.  Not Linear  Changing the Input

Technion - Israel Institute of Technology32 Basic Method Treating each color component as an individual B&W image OriginalBilinearProposed

Technion - Israel Institute of Technology33 Basic Method – Aliasing Effect OriginalBilinear Basic Method

Technion - Israel Institute of Technology34 Components method Using all colors neighbors for the green reconstruction. Reconstructing the difference of the colors components – Hues (R-G, B-G, R-B). Processing smoother signals.

Technion - Israel Institute of Technology35 Statistical generalization Separating each case to sub-regions for better characterization. Using the mean and the standard deviation of each neighbors’ set for the division (size invariant). Each Sub-region will have its own coefficients – better representation of the region.

Technion - Israel Institute of Technology36 Case Study Maximal Size Region: From Light-House

Technion - Israel Institute of Technology37 Case Study 2  1 Region  14 Sub-Regions  98 Sub-Regions  140 Sub-Regions  196 Sub-Regions

Technion - Israel Institute of Technology38 Results 1 (384x256) OriginalBi-LinearGunturk Optimal recovery KimmelNeighbors Rule Optimal Numeric Values: σ – 2 divisions E – 7 divisions

Technion - Israel Institute of Technology39 Results 2 (384x256) OriginalBi-LinearGunturk Optimal recovery KimmelNeighbors Rule

Technion - Israel Institute of Technology40 Summary A new interpolation method has been presented for 1D signals, B&W images and CCD color demosaicing based on the correlation between low and high resolution versions. A non linear localized method was developed to overcome the artificial effects caused from under sampling. The proposed method outperforms the traditional scheme in terms of MSE. Good results has been achieved in 2D interpolation and CCD demosaicing.

Technion - Israel Institute of Technology41 Appendix

Technion - Israel Institute of Technology42 Comparison: Basic vs. Components

Technion - Israel Institute of Technology43 Mean and STD histograms MeanSTD Green x x256 From Light-House