Iterative Techniques for Image Interpolation Nickolaus Mueller and Prof. Truong Nguyen Video Processing Group University of California at San Diego http://videoprocessing.ucsd.edu/

Outline Problem Overview Non-adaptive Image Interpolation Iterative Image Interpolation Conclusions

Problem Overview Still-image interpolation Applications Increase given spatial resolution Applications Low-quality camera Small transmission bandwidth, low storage capacity Display format change—new high definition display Wide-spread application areas: consumer, government, science, etc.

Outline Problem Overview Non-adaptive Image Interpolation: Traditional methods (nearest-neighbor, bilinear, bicubic) Edge directed interpolation Wavelet-based Iterative Image Interpolation Conclusions

Non-Adaptive Interpolation Methods Simple: Bilinear, Bicubic More Complex: Splines, Fractals Slow-varying image model Do not account for sudden changes (e.g. object edges) Images courtesy of Wikipedia ®

Interpolation Examples - Simple Methods Nearest Neighbor Bilinear Bicubic Complexity of Algorithm

Edge-Directed Interpolation: Quick Review Canny Edge Based Expansion [Shi02] Data-Dependent Triangulation [Su04] Modify pixels on either side of edge Use linear interpolation within triangles New Edge-Directed Interpolation [Li01] Edge-Guided Image Interpolation [Zhang06] Estimate high resolution covariances from low resolution image. Perform interpolation using two triangles fuse with weighting scheme

Wavelet-Based Interpolation (1) Assume low-resolution image output of wavelet decomposition Improved anti-aliasing vs. block average Goal: Predict lost coefficients

Wavelet-Based Interpolation (2) Schemes attempt to explicitly predict wavelet coefficients in new sub-bands [Carey99] Use dependency of wavelet coefficients across scales Best linear scheme = lazy scheme [Li07]

Outline Problem Overview Non-adaptive Image Interpolation Iterative Image Interpolation: Contourlet interpolation 3D block matching Combined approach (Laplacian blending) Conclusions

Proposed Method: Interpolation by Iterating Constraints Alternate two constraints on an upsampled image. Know what downsampled version of high resolution image looks like: “observation” constraint How did we obtain the low resolution image? Know something about the image’s transform coefficients: “sparsity” constraint Which transform should we use? What properties do we want it to exhibit?

Observation Constraint: Wavelet Decomposition Low resolution image is obtained from a wavelet filter High pass coefficients lost Observation constraint on the low-pass wavelet coefficients Keep current estimate of high-pass coefficients

Sparsity Constraint: Contourlet Decomposition Directional multiresolution image representation Effectively represent curves, edges, fine detail Sparse signal representation de-noising via thresholding Frequency Localization, Spatial Regularity

Iterative Contourlet-Based Image Interpolation Initial estimate from linear wavelet interpolation Decrease threshold [Guleryuz04] Iterate for set number or until convergence

Results (1) Original Bilinear (26.19 dB) Wavelet (28.23 dB) DDT (27.24 dB) NEDI (28.50 dB) Proposed (29.53 dB)

Results (2) Original Bilinear (25.01 dB) Wavelet (25.74 dB) DDT (25.22 dB) NEDI (25.04 dB) Proposed (25.83 dB)

PSNR Gain vs. Number of Iterations Algorithm can be run until convergence, set iterations Diminishing returns on PSNR gain Fewer iterations when computation time important

Progressive Iterations on Lena 1 2 3 4 5 6 7 8

Improving de-noising near edges Contourlet Interpolation: Good texture preservation Sharp edges, ringing artifacts near large intensity jumps Result of long contourlet filters Solution: Modify de-noising technique for edge regions Keep the same “observation” constraint

Image De-noising using Collaborative Filtering Dabov et. al., 2007 Grouping - “collect similar d-dimensional fragments of a signal into a d+1 dimensional structure” Enable higher dimensional filtering similar to super-resolution Better idea of true underlying signal Improves transform sparsity key for shrinkage algorithms

Fragment Grouping using Block Matching “Find signal fragments similar to a reference one.” Choose some distance function d( ) and a threshold t. Select all block Bi such that d(Br,Bi) < t Stack the similar fragments into a 3-D array. Can be computationally expensive! (Want to find ways to reduce cost.)

Collaborative Filtering via Wavelet Shrinkage 3-D transform is able to exploit both inter- and intra- fragment correlation to produce a sparse signal representation Applied to similar blocks to reduce required number of coefficients.

Algorithm Step 1: Initial Estimate Group similar blocks into 3-D array Apply separable 3-D wavelet transform 2-D wavelet (Bior1.5) each block 1-D wavelet (Haar) across blocks Hard-Threshold coeffiecients Inverse 3-D wavelet transform Aggregate overlapping blocks

Algorithm Step 2: Final De-noised Image Idea: Can get better grouping if we group based on denoised estimate Block matching on de-noised image to form two 3-D groups (original, de-noised) Apply separable 3-D wavelet transform to each group Wiener filter noisy group using de-noised estimate as “true” energy spectrum Inverse 3-D wavelet transform Aggregate overlapping blocks

Iteration Scheme for 3DBM Image Interpolation Initial High Resolution Estimate Block Matching/ 3-D Wavelet Inv 3-D Wavelet / Block Aggregation Wavelet Shrinkage High-pass Inv. Wavelet Transform Wavelet Transform Low-pass Low Res Image

Results of 3DBM Image Interpolation Contourlet Interpolation 3DBM Interpolation

Results of 3DBM Image Interpolation Contourlet Interpolation 3DBM Interpolation

Comparison of Contourlet and 3DBM Image Interpolation Contourlet Interpolation 3DBM Interpolation

Comparison of Contourlet and 3DBM Image Interpolation Contourlet Interpolation 3DBM Interpolation

Features Comparison Contourlet Interpolation: 3DBM Interpolation: Reproduces fine textures well Ringing near large discontinuities 3DBM Interpolation: Edges sharp across, smooth along Texture areas smoothed out Idea: combine the best features of each

Blending Multiple Images into a Single Image Goal: Blend two interpolation results into a single image with a seamless transition

First Step: Segment Image into Texture / Edge Tool: Texture Spread Measure (Minoo, Nguyen) Key Ideas: Local DCT transform Variance of AC Coefficients Maps pixels in image to a Local Texture Spread on the real line Threshold this map to create a binary image mask

Results: Mask Overlays Texture: Contourlet Interpolation Edges: 3DBM Interpolation

Results: Binary Mask vs. Threshold Level Black = Texture Method White = Edge Method Increasing Threshold Heuristic choice of threshold = 1.5 works well for most images

Using an Image Mask to Blend Images Burt and Adelson, 1983 Tool: Gaussian Pyramid Create set of low-pass filtered and downsampled images of the image mask

Using an Image Mask to Blend Images Burt and Adelson, 1983 Tool: Laplacian Pyramid Create set of band-pass filtered and downsampled images for each interpolated image

Blending Laplacian Pyramid Levels Keep sharp details by sharp blending of high-pass images Smoother blending of lower frequencies Texture Interpolation Mask Edge Interpolation Create new Laplacian pyramid and reconstruct

Results of Proposed Method Low-Res 3DBM Contourlet Proposed

Results of Proposed Method Low-Res Proposed Original

Results of Proposed Method Low-Res 3DBM Contourlet Proposed

Results of Proposed Method Low-Res Proposed Original

Zoom-in Comparison Results Bicubic NEDI Proposed

Conclusions Iterative approaches used to improve image quality Contourlet method—best with fine texture 3D Block matching—best with edges Combined approaches with seamless transition through Laplacian pyramid, get best of each method

Interpolation and Super-resolution Lab Part I: Introduction to Interpolation and Wavelet-based methods 1-D Interpolation - Use Matlab functions to interpolate a signal with a step discontinuity. Image Interpolation - Use Matlab functions to explore basic interpolation methods (bilinear, bicubic, etc.) Contourlet Interpolation - Explore a wavelet-based method for interpolation. 3-D Block Matching Interpolation - Explore a second wavelet-based method for interpolation Image Blending - Blend two images together using the Laplacian pyramid Interpolation using Classification and Stitching - Combine contourlet and 3-D Block Matching interpolation into a single image using the best parts from each.

References 1. X. Li and M. T. Orchard, “New edge-directed interpolation,” IEEE Trans. Image Proc. 10, pp. 1521–1527, October 2001. 2. W. K. Carey, D. B. Chang, and S. S. Hermami, “Regularity-preserving image interpolation,” IEEE Trans. Image Proc. 8, pp. 1293–1297, September 1999. 3. Y. Lu and M. N. Do, “A new contourlet transform with sharp frequency localization,” in Proc. IEEE Int. Conf. on Image Proc., (Atlanta, USA), October 2006. 4. O. G. Guleryuz, “Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising: Part I - theory,” IEEE Trans. Image Proc. 15, pp. 539–554, March 2006. 5. M. N. Do and M. Vetterli, “The contourlet transform: an efficient directional multiresolution image representation,” IEEE Trans. Image Proc. 14, December 2005. 6. O. G. Guleryuz, “Predicting wavelet coefficients over edges using estimates based on nonlinear approximants,” in Proc. IEEE Data Compression Conference, April 2004. 7. X. Li, “Image resolution enhancement via data-driven parametric models in the wavelet space.” under review, 2007. 8. J.-L. Starck, M. Elad, and D. Donoho, “Redundant multiscale transforms and their application for morphological component analysis,” Journal of Advances in Imaging and Electron Physics 132, pp. 287–348, 2004.