Technion - Israel Institute of Technology 1 Interpolation Method using Statistical Models RONEN SHER Supervisor: MOSHE PORAT

Technion - Israel Institute of Technology2 Outline Black and White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

Technion - Israel Institute of Technology3 Outline Black and White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

Technion - Israel Institute of Technology4 The Problem Enlargement of an Image by 2x2 Input Output

Technion - Israel Institute of Technology5 Image Interpolation Methods Nearest Neighbor Bilinear Bi-Cubic Spline

Technion - Israel Institute of Technology6 Motivations 1: Pixels Correlation Normalized histograms of Lena gray Levels 256x256 -solid and 512x512-dashed

Technion - Israel Institute of Technology7 Motivations 2: Image Compression Results Compression rates in bits/sample “Necessary Data”:

Technion - Israel Institute of Technology8 Approaching the problem

Technion - Israel Institute of Technology9 Approaching the problem Near Lossless Compression Scheme Inverse Scheme

Technion - Israel Institute of Technology10 Lossless Compression predictors

Technion - Israel Institute of Technology11 Lossless Compression Context modeling The error value is being subtracted from the average error in a given context Vertical edge Horizontal edge

Technion - Israel Institute of Technology12 Outline Black and White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

Technion - Israel Institute of Technology13 Image Regions In edges regions an average prediction will result in a smoothness effect. The edge must be preserved. The edges exist in the input image and the same distribution is assumed in the large image.

Technion - Israel Institute of Technology14 Image Regions In case of a horizontal edge: In case of a vertical edge: Depending on the four surrounding neighbors, there will be at most 4!=24 permutations:

Technion - Israel Institute of Technology15 Pixels fitting From Lena 256x256

Technion - Israel Institute of Technology16 Image Regions In each region a different weighted sum is valid for the prediction The coefficients are learned from the input image

Technion - Israel Institute of Technology17 Outline Black and White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

Technion - Israel Institute of Technology18 Step 1: Coefficients calculation Scanning the Input Image for the ‘x type’ pixel we determine its permutation from its four neighbors and save its value and its neighbors’ values in VM x Modeling only the regions with significant changes in gray levels Same treatment for the ‘+type’ pixels

Technion - Israel Institute of Technology19 Step 1: Coefficients calculation For each permutation we find the four coefficients using the Least Square solution: Same technique for the + coefficients

Technion - Israel Institute of Technology20 Step 2: ‘x type’ Reconstruction Scanning the sparse Image, for each pixel we determine its matching permutation (coefficients) from its four neighbors and predict its value using

Technion - Israel Institute of Technology21 Step 2: ‘+ type’ Reconstruction The Input is I x, for each “+” pixel we find its matching permutation (coefficients) and calculate its prediction by

Technion - Israel Institute of Technology22 Experiments - Lena The 4 coefficients in 24 cases of x-type  Lena size 512x512 o Lena size 256x256 Errors α1α1 α4α4 α3α3 α2α2

Technion - Israel Institute of Technology23 Experiments 1 - B&W images (128x128->256x256) Original Bilinear Bi-Cubic Spline Proposed Bi-Cubic Nearest neighbor (Input)

Technion - Israel Institute of Technology24 Experiments 2 - B&W images (128x128->256x256) Original Bilinear Bi-Cubic SplineBi-Cubic Nearest neighbor (Input)

Technion - Israel Institute of Technology25 Outline Black and White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

Technion - Israel Institute of Technology26 One Dimension Interpolation Interpolating y d, by using NR. Its adjacent samples serve as the four neighbors for the coefficients calculation.

Technion - Israel Institute of Technology27 Synthetic Test Signal y1=sin(r.*(5+3.*sin(2.*(r+0.7)))).*sin(7.*(r+0.9)) t1=1,2..N1 r=(t1+OS1)/100 N1=2400 f1=1 Ts=2 OS1=3000 L=2

Technion - Israel Institute of Technology28 1D Interpolation results 1

Technion - Israel Institute of Technology29 1D Interpolation results 2 Voice signal: the word “Diskette”

Technion - Israel Institute of Technology30 Outline Black and White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

Technion - Israel Institute of Technology31 CCD structure

Technion - Israel Institute of Technology32 CCD Demosaicing Methods Bilinear Kimmel - gradient based function and hues R/G,B/G. Gunturk – data consistency and similarity between the high-frequency components. Muresan - interpolates R-G,B-G.  Not Linear  Changing the Input

Technion - Israel Institute of Technology33 Simple Method Treating each color component as individual B&W image OriginalBilinearProposed

Technion - Israel Institute of Technology34 Simple Method 2 – Aliasing Effect OriginalBilinear Simple Method

Technion - Israel Institute of Technology35 Components method Using all colors neighbors for the green reconstruction Reconstruct the difference of the colors components – Hues (R-G, B-G, R-B). Processing smoother signals.

Technion - Israel Institute of Technology36 Statistical extension Separating each case to sub-regions for better characterization. Using the mean and the standard deviation of each neighbors’ set for the division (size invariant). Each Sub-region will have its own coefficients – better representative of the region

Technion - Israel Institute of Technology37 Case Study Maximal Size Region:

Technion - Israel Institute of Technology38 Case Study 2  1 Region  14 Sub-Regions  98 Sub-Regions  140 Sub-Regions  196 Sub-Regions

Technion - Israel Institute of Technology39 Results 1 (384x256) OriginalBi-LinearGunturk Optimal recovery KimmelNeighbors Rule Optimal Numeric Values: σ – 2 divisions E – 7 divisions

Technion - Israel Institute of Technology40 Results 2 (384x256) OriginalBi-LinearGunturk Optimal recovery KimmelNeighbors Rule

Technion - Israel Institute of Technology41 Outline Black and White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

Technion - Israel Institute of Technology42 Summary A new reconstruction method was presented for 1D signals, B&W images and CCD demosaicing using the correlation between low and high resolution versions. A non linear Localize scheme was developed to overcome the artificial effects caused from under sampling. The new method showed better performance over the traditional scheme in terms of MSE in 1D interpolation. Satisfying results achieved in B&W interpolation and CCD demosaicing, compared to other known techniques.

Technion - Israel Institute of Technology43 Back Up

Technion - Israel Institute of Technology44 Comparison: Simple vs. Components

Technion - Israel Institute of Technology45 Mean and STD histograms MeanSTD Green