1. Iterative Techniques for Image Interpolation
Nickolaus Mueller and Prof. Truong Nguyen
Video Processing Group
University of California at San Diego

2. Outline Problem Overview Non-adaptive Image Interpolation
Iterative Image Interpolation
Conclusions

3. Problem Overview Still-image interpolation Applications
Increase given spatial resolution
Applications
Low-quality camera
Small transmission bandwidth, low storage capacity
Display format change—new high definition display
Wide-spread application areas: consumer, government, science, etc.

4. Outline Problem Overview Non-adaptive Image Interpolation:
Traditional methods (nearest-neighbor, bilinear, bicubic)
Edge directed interpolation
Wavelet-based
Iterative Image Interpolation
Conclusions

5. Non-Adaptive Interpolation Methods
Simple: Bilinear, Bicubic
More Complex: Splines, Fractals
Slow-varying image model
Do not account for sudden changes (e.g. object edges)
Images courtesy of Wikipedia ®

6. Interpolation Examples - Simple Methods
Nearest Neighbor
Bilinear
Bicubic
Complexity of Algorithm

7. Edge-Directed Interpolation: Quick Review
Canny Edge Based Expansion [Shi02]
Data-Dependent Triangulation [Su04]
Modify pixels on either side of edge
Use linear interpolation within triangles
New Edge-Directed Interpolation [Li01]
Edge-Guided Image Interpolation
[Zhang06]
Estimate high resolution covariances
from low resolution image.
Perform interpolation using two triangles
fuse with weighting scheme

8. Wavelet-Based Interpolation (1)
Assume low-resolution image output of wavelet decomposition
Improved anti-aliasing vs. block average
Goal: Predict lost coefficients

9. Wavelet-Based Interpolation (2)
Schemes attempt to explicitly predict wavelet coefficients in new sub-bands [Carey99]
Use dependency of wavelet coefficients across scales
Best linear scheme = lazy scheme [Li07]

10. Outline Problem Overview Non-adaptive Image Interpolation
Iterative Image Interpolation:
Contourlet interpolation
3D block matching
Combined approach (Laplacian blending)
Conclusions

11. Proposed Method: Interpolation by Iterating Constraints
Alternate two constraints on an upsampled image.
Know what downsampled version of high resolution image looks like: “observation” constraint
How did we obtain the low resolution image?
Know something about the image’s transform coefficients: “sparsity” constraint
Which transform should we use? What properties do we want it to exhibit?

12. Observation Constraint: Wavelet Decomposition
Low resolution image is obtained from a wavelet filter
High pass coefficients lost
Observation constraint on the low-pass wavelet coefficients
Keep current estimate of high-pass coefficients

13. Sparsity Constraint: Contourlet Decomposition
Directional multiresolution image representation
Effectively represent curves, edges, fine detail
Sparse signal representation de-noising via thresholding
Frequency Localization, Spatial Regularity

14. Iterative Contourlet-Based Image Interpolation
Initial estimate from linear wavelet interpolation
Decrease threshold [Guleryuz04]
Iterate for set number or until convergence

15. Results (1) Original Bilinear (26.19 dB) Wavelet (28.23 dB)
DDT (27.24 dB)
NEDI (28.50 dB)
Proposed (29.53 dB)

16. Results (2) Original Bilinear (25.01 dB) Wavelet (25.74 dB)
DDT (25.22 dB)
NEDI (25.04 dB)
Proposed (25.83 dB)

17. PSNR Gain vs. Number of Iterations
Algorithm can be run until convergence, set iterations
Diminishing returns on PSNR gain
Fewer iterations when computation time important

18. Progressive Iterations on Lena1 2 3 4 5 6 7 8

19. Improving de-noising near edges
Contourlet Interpolation:
Good texture preservation
Sharp edges, ringing artifacts near large intensity jumps
Result of long contourlet filters
Solution:
Modify de-noising technique for edge regions
Keep the same “observation” constraint

20. Image De-noising using Collaborative Filtering
Dabov et. al., 2007
Grouping - “collect similar d-dimensional fragments of a signal into a d+1 dimensional structure”
Enable higher dimensional filtering similar to super-resolution
Better idea of true underlying signal
Improves transform sparsity key for shrinkage algorithms

21. Fragment Grouping using Block Matching
“Find signal fragments similar to a reference one.”
Choose some distance function d( ) and
a threshold t.
Select all block Bi such that d(Br,Bi) < t
Stack the similar fragments into a 3-D array.
Can be computationally expensive!
(Want to find ways to reduce cost.)

22. Collaborative Filtering via Wavelet Shrinkage
3-D transform is able to exploit both inter- and intra- fragment correlation to produce a sparse signal representation
Applied to similar blocks to reduce required number of coefficients.

23. Algorithm Step 1: Initial Estimate
Group similar blocks into 3-D array
Apply separable 3-D wavelet transform
2-D wavelet (Bior1.5) each block
1-D wavelet (Haar) across blocks
Hard-Threshold coeffiecients
Inverse 3-D wavelet transform
Aggregate overlapping blocks

24. Algorithm Step 2: Final De-noised Image
Idea: Can get better grouping if we group based on denoised estimate
Block matching on de-noised image to form two 3-D groups (original, de-noised)
Apply separable 3-D wavelet transform to each group
Wiener filter noisy group using de-noised estimate as “true” energy spectrum
Inverse 3-D wavelet transform
Aggregate overlapping blocks

25. Iteration Scheme for 3DBM Image Interpolation
Initial High
Resolution Estimate
Block Matching/
3-D Wavelet
Inv 3-D Wavelet /
Block Aggregation
Wavelet
Shrinkage
High-pass
Inv. Wavelet Transform
Wavelet Transform
Low-pass
Low Res Image

26. Results of 3DBM Image Interpolation
Contourlet Interpolation
3DBM Interpolation

27. Results of 3DBM Image Interpolation
Contourlet Interpolation
3DBM Interpolation

28. Comparison of Contourlet and 3DBM Image Interpolation
Contourlet Interpolation
3DBM Interpolation

29. Comparison of Contourlet and 3DBM Image Interpolation
Contourlet Interpolation
3DBM Interpolation

30. Features Comparison Contourlet Interpolation: 3DBM Interpolation:
Reproduces fine textures well
Ringing near large discontinuities
3DBM Interpolation:
Edges sharp across, smooth along
Texture areas smoothed out
Idea: combine the best features of each

31. Blending Multiple Images into a Single Image
Goal: Blend two interpolation results into a single image with a seamless transition

32. First Step: Segment Image into Texture / Edge
Tool: Texture Spread Measure (Minoo, Nguyen)
Key Ideas:
Local DCT transform
Variance of AC Coefficients
Maps pixels in image to a Local Texture Spread on the real line
Threshold this map to create a binary image mask

33. Results: Mask Overlays
Texture: Contourlet Interpolation
Edges: 3DBM Interpolation

34. Results: Binary Mask vs. Threshold Level
Black = Texture Method White = Edge Method
Increasing Threshold
Heuristic choice of threshold = 1.5 works well for most images

35. Using an Image Mask to Blend Images
Burt and Adelson, 1983
Tool: Gaussian Pyramid
Create set of low-pass filtered and downsampled images of the image mask

36. Using an Image Mask to Blend Images
Burt and Adelson, 1983
Tool: Laplacian Pyramid
Create set of band-pass filtered and downsampled images for each interpolated image

37. Blending Laplacian Pyramid Levels
Keep sharp details by sharp blending of high-pass images
Smoother blending of lower frequencies
Texture Interpolation
Mask
Edge Interpolation
Create new Laplacian pyramid and reconstruct

38. Results of Proposed Method
Low-Res
3DBM
Contourlet
Proposed

39. Results of Proposed Method
Low-Res
Proposed
Original

40. Results of Proposed Method
Low-Res
3DBM
Contourlet
Proposed

41. Results of Proposed Method
Low-Res
Proposed
Original

42. Zoom-in Comparison Results
Bicubic
NEDI
Proposed

43. Conclusions Iterative approaches used to improve image quality
Contourlet method—best with fine texture
3D Block matching—best with edges
Combined approaches with seamless transition through Laplacian pyramid, get best of each method

44. Interpolation and Super-resolution Lab Part I: Introduction to Interpolation and Wavelet-based methods
1-D Interpolation - Use Matlab functions to interpolate a signal with a step discontinuity.
Image Interpolation - Use Matlab functions to explore basic interpolation methods (bilinear, bicubic, etc.)
Contourlet Interpolation - Explore a wavelet-based method for interpolation.
3-D Block Matching Interpolation - Explore a second wavelet-based method for interpolation
Image Blending - Blend two images together using the Laplacian pyramid
Interpolation using Classification and Stitching - Combine contourlet and 3-D Block Matching interpolation into a single image using the best parts from each.

45. References
1. X. Li and M. T. Orchard, “New edge-directed interpolation,” IEEE Trans. Image Proc. 10, pp. 1521–1527, October 2001.
2. W. K. Carey, D. B. Chang, and S. S. Hermami, “Regularity-preserving image interpolation,” IEEE Trans. Image Proc. 8, pp. 1293–1297, September 1999.
3. Y. Lu and M. N. Do, “A new contourlet transform with sharp frequency localization,” in Proc. IEEE Int. Conf. on Image Proc., (Atlanta, USA), October 2006.
4. O. G. Guleryuz, “Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising: Part I - theory,” IEEE Trans. Image Proc. 15, pp. 539–554, March 2006.
5. M. N. Do and M. Vetterli, “The contourlet transform: an efficient directional multiresolution image representation,” IEEE Trans. Image Proc. 14, December 2005.
6. O. G. Guleryuz, “Predicting wavelet coefficients over edges using estimates based on nonlinear approximants,” in Proc. IEEE Data Compression Conference, April 2004.
7. X. Li, “Image resolution enhancement via data-driven parametric models in the wavelet space.” under review, 2007.
8. J.-L. Starck, M. Elad, and D. Donoho, “Redundant multiscale transforms and their application for morphological component analysis,” Journal of Advances in Imaging and Electron Physics 132, pp. 287–348, 2004.