1. Technion - Israel Institute of Technology 1 On Interpolation Methods using Statistical Models RONEN SHER Supervisor: MOSHE PORAT

2. 2 Outline Black & White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overview  Components correlation  Statistical extension  Results Summary

3. 3 The Interpolation Problem Factor of 2 Input Output

4. 4 Image Interpolation Methods Nearest Neighbor Bilinear Bi-Cubic Spline

5. 5 Motivations 1: Pixels Correlation Normalized histograms of Lena (gray Levels) 256x256-dashed ; 512x512-solid

6. 6 Motivations 2: Image Compression Results Compression rates in bits/sample “Necessary Data”:

7. 7 Proposed Approach

8. 8 Near Lossless Compression Scheme Inverse Scheme

9. 9 Lossless Compression predictors

10. 10 Lossless Compression - Context modeling The error value is subtracted from the average error in a given context Vertical edge Horizontal edge

11. 11 Outline Black & White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overview  Components correlation  Statistical extension  Results Summary

12. 12 Image Regions In regions of edges, averaging will result in a smoothing effect. The edge must be preserved. The edges exist in the input image and the same distribution is assumed in the larger interpolated image.

13. 13 Image Regions In case of a horizontal edge: In case of a vertical edge: Depending on the four surrounding neighbors, there will be at most 4!=24 permutations:

14. 14 Pixels fitting From Lena 256x256

15. 15 Image Regions In each region a different weighted sum is valid for the prediction The coefficients are learned from the input image

16. 16 Outline Black & White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

17. 17 Step 1: Coefficients calculation Scanning the Input Image for the ‘x type’ pixel we determine its permutation from its four neighbors and save its value and its neighbors’ values in VM x Modeling only the regions with significant changes in gray levels Similar technique for the ‘+type’ pixels

18. 18 Step 1: Coefficients calculation For each permutation we find the four coefficients using the Least Square solution: Similar technique for the + coefficients

19. 19 Step 2a: ‘x type’ Reconstruction Scanning the sparse Image, for each pixel we determine its matching permutation (coefficients) from its four neighbors and predict its value using

20. 20 Step 2b: ‘+ type’ Reconstruction The Input is I x, for each “+” pixel we find its matching permutation (coefficients) and calculate its prediction by

21. 21 Experiments - Lena The 4 coefficients in 24 cases of x-type  Lena size 512x512 o Lena size 256x256 Errors α1α1 α4α4 α3α3 α2α2

22. 22 Example 1 - B&W images (128x128->256x256) Original Bilinear Bi-Cubic Spline Proposed Bi-Cubic Nearest neighbor (Input)

23. 23 Example 2 - B&W images (128x128->256x256) Original Bilinear Bi-Cubic SplineBi-Cubic Nearest neighbor (Input) Proposed

24. 24 Outline Black and White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

25. 25 One-Dimensional Interpolation Interpolating y d, using NR. Its adjacent samples serve as the four neighbors for the coefficients’ calculation.

26. 26 Synthetic Test Signal y1=sin(r.*(5+3.*sin(2.*(r+0.7)))).*sin(7.*(r+0.9)) t1=1,2..N1 r=(t1+OS1)/100 N1=2400 f1=1 Ts=2 OS1=3000 L=2

27. 27 1D Interpolation result 1

28. 28 1D Interpolation result 2 Voice signal: the word “Diskette”

29. 29 Outline Black and White image interpolation  Motivations  Concepts  Flow  Results 1D Signal interpolation CCD Demosaicing  Structure  Methods Overveiw  Components correlation  Statistical extension  Results Summary

30. 30 CCD structure

31. 31 CCD Demosaicing Methods Bilinear Kimmel - gradient based function and hues R/G,B/G. Gunturk – data consistency and similarity between the high-frequency components. Muresan - interpolates R-G,B-G.  Not Linear  Changing the Input

32. 32 Basic Method Treating each color component as an individual B&W image OriginalBilinearProposed

33. 33 Basic Method – Aliasing Effect OriginalBilinear Basic Method

34. 34 Components method Using all colors neighbors for the green reconstruction. Reconstructing the difference of the colors components – Hues (R-G, B-G, R-B). Processing smoother signals.

35. 35 Statistical generalization Separating each case to sub-regions for better characterization. Using the mean and the standard deviation of each neighbors’ set for the division (size invariant). Each Sub-region will have its own coefficients – better representation of the region.

36. 36 Case Study Maximal Size Region: From Light-House

37. 37 Case Study 2  1 Region  14 Sub-Regions  98 Sub-Regions  140 Sub-Regions  196 Sub-Regions

38. 38 Results 1 (384x256) OriginalBi-LinearGunturk Optimal recovery KimmelNeighbors Rule Optimal Numeric Values: σ – 2 divisions E – 7 divisions

39. 39 Results 2 (384x256) OriginalBi-LinearGunturk Optimal recovery KimmelNeighbors Rule

40. 40 Summary A new interpolation method has been introduced for 1D signals, B&W images and CCD color demosaicing based on the correlation between low and high resolution versions of a signal. A non linear localized method has been developed to overcome the artificial effects caused by under-sampling. The proposed method outperforms traditional methods in terms of MSE and visual perception. Good results have been achieved in 2D interpolation and CCD demosaicing.

41. 41 Appendix

42. 42 Comparison: Basic vs. Components

43. 43 Mean and STD histograms MeanSTD Green -- 192x128 -- 384x256 From Light-House