Why do we Need Image Model in the first place? Any image processing algorithm has to work on a collection (class) of images instead of a single one Mathematical model gives us the abstraction of common properties of the images within the same class Model is to hypothesis what images are to observation data In physics, can F=ma explain the relationship between force and acceleration?  In image processing, can this model fit this class of images? EE565 Advanced Image Processing Copyright Xin Li 2009-2012

EE565 Advanced Image Processing Copyright Xin Li 2009-2012 Birdview denoising synthesis/inpainting Encoding/decoding … … PDE models TV, MCD/PMD Reaction-Diffusion/TV TV, MCD/PMD EZW SPIHT EBCOT … transform models Wavelet/TI thresholding Pyramid-based /DCT-based Wavelet/TI thresholding patch models Image quilting/ BM3D-CS NLM/BM3D NLM/BM3D EE565 Advanced Image Processing Copyright Xin Li 2009-2012

EE565 Advanced Image Processing Copyright Xin Li 2009-2012 PDE-based Denoising Think of image as a 3D surface: a mapping from domain (x,y) to range u(x,y) Geometry-driven ideas How to measure “changes”: e.g., total-variation vs. surface area Importance of direction: from isotropic diffusion to anisotropic diffusion Discrete implementation: finite—difference method EE565 Advanced Image Processing Copyright Xin Li 2009-2012

Wavelet-based Denoising where Laplacian (signal, heavy-tail) Gaussian (noise, light-tail) P: shape parameter : variance parameter EE565 Advanced Image Processing Copyright Xin Li 2009-2012

Bilateral Filtering (a Novel idea) output input reproduced from [Durand 02]

Patch-based Denoising Noisy patches Denoised patches WD WD = T Thresholding T-1 EE565 Advanced Image Processing Copyright Xin Li 2009-2012

EE565 Advanced Image Processing Copyright Xin Li 2009-2012 Connections Equivalence between TV and Haar-wavelet thresholding Why BM3D works so much better? Heuristics: Nonlocal similar patches around edges/textures Theory: more accurate signal variance estimation Lesson learned: Markov model is wrong EE565 Advanced Image Processing Copyright Xin Li 2009-2012

TV-Inpainting Example (Courtesy: Jackie Shen, UMN MATH) EE565 Advanced Image Processing Copyright Xin Li 2009-2012

Wavelet-domain Histogram Matching EE565 Advanced Image Processing Copyright Xin Li 2009-2012

Joint PDF of Wavelet Coefficients Y= X= Joint pdf of two correlated random variables X and Y Neighborhood I(Q): {Left,Up,cousin and aunt} Can you use this model to interpret why EZW works? EE565 Advanced Image Processing Copyright Xin Li 2009-2012

Wavelet-Domain Parametric Texture Models original synthesized EE565 Advanced Image Processing Copyright Xin Li 2009-2012

Locality Revisited: “relativity theory” for image processing N past samples Input image The definition of local neighborhood has to be relative EE565 Advanced Image Processing Copyright Xin Li 2009-2012

Nonparametric Texture Synthesis: Image Quilting EE565 Advanced Image Processing Copyright Xin Li 2009-2012

EE565 Advanced Image Processing Copyright Xin Li 2009-2012 Connections Local inpainting vs. global synthesis Observation of scale: think of inpainting the boundary of a leaf vs. inpainting the occluded region of a bush Selection of scanning order/window size Deterministic vs. statistical Regularization vs. prior (no fundamental difference, just different languages) Variational (energy-based) vs. set theoretic (projection-based) How do we minimize E(u)? EE565 Advanced Image Processing Copyright Xin Li 2009-2012

Image Coding as Painting What (intensity uncertainty) vs. Where (location uncertainty) EE565 Advanced Image Processing Copyright Xin Li 2009-2012

Projection-based post-processing ● Quantization (observation data) set Quantization WT ● Regularization (prior) constraint set C1 TV, MCD/PMD X1 Wavelet/TI thresholding X∞ X0 X2 NLM/BM3D C2 EE5965 Advanced Image Processing Copyright Xin Li 2009-2012

EE565 Advanced Image Processing Copyright Xin Li 2009-2012 Summary on Models Image models are at the foundation of any image processing algorithms To gain a deeper understanding of why xxx works, try to distill the underlying model “All models are wrong; but some are useful” What really matters is how well your model matches with the observation data “Nature is not economical of structures but of principles” So we don’t need to work out zillions of problems/apps EE565 Advanced Image Processing Copyright Xin Li 2009-2012

EE565 Advanced Image Processing Copyright Xin Li 2009-2012 Birdview denoising synthesis/inpainting Encoding/decoding … … PDE models TV, MCD/PMD Reaction-Diffusion/TV TV, MCD/PMD EZW SPIHT EBCOT … transform models Wavelet/TI thresholding Pyramid-based /DCT-based Wavelet/TI thresholding patch models Image quilting/ BM3D-CS NLM/BM3D NLM/BM3D EE565 Advanced Image Processing Copyright Xin Li 2009-2012

PDE-based Image Processing We only discussed the application of PDE in image restoration in this class More successful applications are segmentation-related Active contour model (snake) Active contour without edges (Chan&Vese’2001) Mumford-Shah model EE565 Advanced Image Processing Copyright Xin Li 2009-2012

EE565 Advanced Image Processing Copyright Xin Li 2009-2012 Active Contour EE565 Advanced Image Processing Copyright Xin Li 2009-2012

Wavelet-based Image Processing The most successful application is lossy image compression Why is it less effective on other non-coding applications? Segmentation/Classification: linearity Detection/Recognition: invariance Beyond wavelets vs. Beyond Hilbert space EE565 Advanced Image Processing Copyright Xin Li 2009-2012

EE565 Advanced Image Processing Copyright Xin Li 2009-2012 Beyond Wavelets Daubechies’ wavelet,1988 Do&Vetterli’s contourlet,2005 Bell&Sejnowski’ICA,1996 Elad&Aharon’K-SVD,2006 Dictionary construction Dictionary learning EE565 Advanced Image Processing Copyright Xin Li 2009-2012

Patch-based Image Processing We have focused on the applications at the low-level vision (e.g., denoising, inpainting, post-processing) Highly effective for high-level vision tasks (e.g., classification, recognition) too SIFT/SURF/HOG/LBP/CARD/BRIEF… Importance of clustering: kmeans vs. kNN EE565 Advanced Image Processing Copyright Xin Li 2009-2012

EE565 Advanced Image Processing Copyright Xin Li 2009-2012 The Emerging Trend global responses Low-level vision: Restoration etc. High-level vision: Recognition etc. local responses Bridge: SVM = Sparse coding Biologically-plausible EE565 Advanced Image Processing Copyright Xin Li 2009-2012

What are the Killer Applications? (Biased View) Computational imaging Gigapixel cameras, light field cameras Human computer interaction Kinect, touch-screen, siri Intelligent transportation system Mobileye, real-time traffic monitoring Healthcare-related The paygrade of radiologists is ridiculously high Cyberlearning “A picture is worth a thousand words” EE565 Advanced Image Processing Copyright Xin Li 2009-2012

EE565 Advanced Image Processing Copyright Xin Li 2009-2012 Summary on Practice MATLAB provides a user friendly platform for testing your ideas You can see what you have done C/C++ programming skills are a plus Efficient implementation could make a difference (e.g., SPIHT vs. EZW) Team work is becoming more valued than the past (e.g., Samsung story) EE565 Advanced Image Processing Copyright Xin Li 2009-2012