1. Patch-based Image Interpolation: Algorithms and Applications
Xin Li
Lane Dept. of CSEE
West Virginia University

2. Where Does Patch Come from?
Neuroscience: receptive fields of neighboring cells in human vision system have severe overlapping
Engineering: patch has been under the disguise of many different names such as windows in digital filters, blocks in JPEG and the support of wavelet bases,
Cited from D. Hubel, “Eye, Brain and
Vision”, 1988

3. Patch-based Image Models
Local models
Markov Random Field (MRF) and higher-order extensions (e.g., Field-of-Expert)
Transform-based: PCA, DCT, wavelets
Nonlocal models
Bilateral filtering (Tomasi et al. ICCV’1998)
Texture synthesis via Nonparametric resampling (Efros&Leung ICCV’1999)
Exemplar-based inpainting (Criminisi et al. TIP’2004)
Nonlocal mean denoising (Buades et al.’ CVPR’2005)
Total Least-Square denoising (Hirakawa&Parks TIP’2006)
Block-matching 3D denoising (Dabov et al. TIP’2007)

4. A Bayesian Formulation of Image Interpolation Problem
Likelihood
(our focus here)
Image prior
(e.g., sparsity-based)
Unobservable
data
Observable
data
Model class
(e.g., local vs. nonlocal)

5. A Simple Extension of BM3D
Hard thresholding
3D transform of similar patches
Basic idea: combine BM3D with progressive thresholding (Guleryuz TIP’2006)

6. Interpolation of LR Imagesx y
bicubic
NEDI1
this work
31.76dB dB dB
34.71dB dB dB
28.70dB dB dB
18.81dB dB dB
1X. Li and M. Orchard, “New edge directed interpolation”, IEEE TIP, 2001

7. Go Back to Biology rods cone
Spatially random distribution of rod/cone cells keeps
aliasing artifacts out of our vision

8. Interpolation of Nonuniformly-sampled Imagesx y DT KR
this work
29.06dB dB dB
DT- Delauney
Triangle-based
(griddata under
MATLAB)
KR- Kernal
Regression-based
(Takeda et al.
IEEE TIP 2007)
28.46dB dB dB
26.04dB dB dB
17.90dB dB dB

9. Modeling Spatial Randomness
Extensively studied in geostatistics and environmental statistics (e.g., spatial distribution of animals and plants)
Mathematically modeled by homogeneous Poisson process (density parameterλ)
Lack of positional differentiation
Lack of scale differentiation
Empirically there exist quadrant-based and distance-based randomness metrics

10. Monte-Carlo Based Optimization
The lower energy
the more random
Iterative procedure: randomly pick two locations (one black and the other white), if
swapping them decreases the energy, accept it; otherwise accept it with some probability

11. Importance of Locations
after optimization
In biological world:
evolution + development
before optimization
Identical reconstruction algorithm; only differ on sampling locations

12. Application into Compressive Imaging
Random
Sampling
Pattern S
quantization
channel
interpolation
sensor node
How is it different from conventional image coding system?
No bits are spent on coding the location information (random=no cost).

13. Coding Results R=0.21bpp original ours PSNR=27.85dB SSIM=0.8750 SPIHT

14. Error Resilience Results

15. Conclusions
A good image prior is useful to many processing tasks involving incomplete or noisy observation
As we move from local to nonlocal models, the location of sampling points becomes important – “location (address) and intensity (data) are the same thing” cited from T. Kohonen “Self-Organization and Associative Memory”
Image processing is at the intersection of science and engineering- will BM3D lead to a new class of SOM?