1. Why do we Need Image Model in the first place?
Any image processing algorithm has to work on a collection (class) of images instead of a single one
Mathematical model gives us the abstraction of common properties of the images within the same class
Model is to hypothesis what images are to observation data
In physics, can F=ma explain the relationship between force and acceleration?  In image processing, can this model fit this class of images?
EE565 Advanced Image Processing Copyright Xin Li

2. EE565 Advanced Image Processing Copyright Xin Li 2009-2012
Birdview
denoising
synthesis/inpainting
Encoding/decoding
… …
PDE
models
TV, MCD/PMD
Reaction-Diffusion/TV
TV, MCD/PMD
EZW
SPIHT
EBCOT …
transform
models
Wavelet/TI
thresholding
Pyramid-based
/DCT-based
Wavelet/TI
thresholding
patch
models
Image quilting/
BM3D-CS
NLM/BM3D
NLM/BM3D
EE565 Advanced Image Processing Copyright Xin Li

3. EE565 Advanced Image Processing Copyright Xin Li 2009-2012
PDE-based Denoising
Think of image as a 3D surface: a mapping from domain (x,y) to range u(x,y)
Geometry-driven ideas
How to measure “changes”: e.g., total-variation vs. surface area
Importance of direction: from isotropic diffusion to anisotropic diffusion
Discrete implementation: finite—difference method
EE565 Advanced Image Processing Copyright Xin Li

4. Wavelet-based Denoising
where
Laplacian
(signal, heavy-tail)
Gaussian
(noise, light-tail)
P: shape parameter
: variance parameter
EE565 Advanced Image Processing Copyright Xin Li

5. Bilateral Filtering (a Novel idea)
output
input
reproduced from [Durand 02]

6. Patch-based Denoising
Noisy patches
Denoised patches WD WD = T
Thresholding
T-1
EE565 Advanced Image Processing Copyright Xin Li

7. EE565 Advanced Image Processing Copyright Xin Li 2009-2012
Connections
Equivalence between TV and Haar-wavelet thresholding
Why BM3D works so much better?
Heuristics: Nonlocal similar patches around edges/textures
Theory: more accurate signal variance estimation
Lesson learned: Markov model is wrong
EE565 Advanced Image Processing Copyright Xin Li

8. TV-Inpainting Example
(Courtesy: Jackie Shen, UMN MATH)
EE565 Advanced Image Processing Copyright Xin Li

9. Wavelet-domain Histogram Matching
EE565 Advanced Image Processing Copyright Xin Li

10. Joint PDF of Wavelet CoefficientsY= X=
Joint pdf of two correlated
random variables X and Y
Neighborhood I(Q): {Left,Up,cousin and aunt}
Can you use this model to interpret why EZW works?
EE565 Advanced Image Processing Copyright Xin Li

11. Wavelet-Domain Parametric Texture Models
original
synthesized
EE565 Advanced Image Processing Copyright Xin Li

12. Locality Revisited: “relativity theory” for image processing
N past samples
Input image
The definition of local neighborhood has to be relative
EE565 Advanced Image Processing Copyright Xin Li

13. Nonparametric Texture Synthesis: Image Quilting
EE565 Advanced Image Processing Copyright Xin Li

14. EE565 Advanced Image Processing Copyright Xin Li 2009-2012
Connections
Local inpainting vs. global synthesis
Observation of scale: think of inpainting the boundary of a leaf vs. inpainting the occluded region of a bush
Selection of scanning order/window size
Deterministic vs. statistical
Regularization vs. prior (no fundamental difference, just different languages)
Variational (energy-based) vs. set theoretic (projection-based)
How do we minimize E(u)?
EE565 Advanced Image Processing Copyright Xin Li

15. Image Coding as Painting
What (intensity uncertainty) vs. Where (location uncertainty)
EE565 Advanced Image Processing Copyright Xin Li

16. Projection-based post-processing
● Quantization (observation data) set
Quantization WT
● Regularization (prior) constraint set C1
TV, MCD/PMD X1
Wavelet/TI
thresholding X∞ X0 X2
NLM/BM3D C2
EE5965 Advanced Image Processing Copyright Xin Li

17. EE565 Advanced Image Processing Copyright Xin Li 2009-2012
Summary on Models
Image models are at the foundation of any image processing algorithms
To gain a deeper understanding of why xxx works, try to distill the underlying model
“All models are wrong; but some are useful”
What really matters is how well your model matches with the observation data
“Nature is not economical of structures but of principles”
So we don’t need to work out zillions of problems/apps
EE565 Advanced Image Processing Copyright Xin Li

18. EE565 Advanced Image Processing Copyright Xin Li 2009-2012
Birdview
denoising
synthesis/inpainting
Encoding/decoding
… …
PDE
models
TV, MCD/PMD
Reaction-Diffusion/TV
TV, MCD/PMD
EZW
SPIHT
EBCOT …
transform
models
Wavelet/TI
thresholding
Pyramid-based
/DCT-based
Wavelet/TI
thresholding
patch
models
Image quilting/
BM3D-CS
NLM/BM3D
NLM/BM3D
EE565 Advanced Image Processing Copyright Xin Li

19. PDE-based Image Processing
We only discussed the application of PDE in image restoration in this class
More successful applications are segmentation-related
Active contour model (snake)
Active contour without edges (Chan&Vese’2001)
Mumford-Shah model
EE565 Advanced Image Processing Copyright Xin Li

20. EE565 Advanced Image Processing Copyright Xin Li 2009-2012
Active Contour
EE565 Advanced Image Processing Copyright Xin Li

21. Wavelet-based Image Processing
The most successful application is lossy image compression
Why is it less effective on other non-coding applications?
Segmentation/Classification: linearity
Detection/Recognition: invariance
Beyond wavelets vs. Beyond Hilbert space
EE565 Advanced Image Processing Copyright Xin Li

22. EE565 Advanced Image Processing Copyright Xin Li 2009-2012
Beyond Wavelets
Daubechies’ wavelet,1988
Do&Vetterli’s contourlet,2005
Bell&Sejnowski’ICA,1996
Elad&Aharon’K-SVD,2006
Dictionary construction
Dictionary learning
EE565 Advanced Image Processing Copyright Xin Li

23. Patch-based Image Processing
We have focused on the applications at the low-level vision (e.g., denoising, inpainting, post-processing)
Highly effective for high-level vision tasks (e.g., classification, recognition) too
SIFT/SURF/HOG/LBP/CARD/BRIEF…
Importance of clustering: kmeans vs. kNN
EE565 Advanced Image Processing Copyright Xin Li

24. EE565 Advanced Image Processing Copyright Xin Li 2009-2012
The Emerging Trend
global responses
Low-level vision:
Restoration etc.
High-level vision:
Recognition etc.
local responses
Bridge: SVM = Sparse coding
Biologically-plausible
EE565 Advanced Image Processing Copyright Xin Li

25. What are the Killer Applications? (Biased View)
Computational imaging
Gigapixel cameras, light field cameras
Human computer interaction
Kinect, touch-screen, siri
Intelligent transportation system
Mobileye, real-time traffic monitoring
Healthcare-related
The paygrade of radiologists is ridiculously high
Cyberlearning
“A picture is worth a thousand words”
EE565 Advanced Image Processing Copyright Xin Li

26. EE565 Advanced Image Processing Copyright Xin Li 2009-2012
Summary on Practice
MATLAB provides a user friendly platform for testing your ideas
You can see what you have done
C/C++ programming skills are a plus
Efficient implementation could make a difference (e.g., SPIHT vs. EZW)
Team work is becoming more valued than the past (e.g., Samsung story)
EE565 Advanced Image Processing Copyright Xin Li