MMSE Estimation for Sparse Representation Modeling

Slides:



Advertisements
Similar presentations
Pattern Recognition and Machine Learning
Advertisements

Linear Regression.
Object Specific Compressed Sensing by minimizing a weighted L2-norm A. Mahalanobis.
Slide 1 Bayesian Model Fusion: Large-Scale Performance Modeling of Analog and Mixed- Signal Circuits by Reusing Early-Stage Data Fa Wang*, Wangyang Zhang*,
Joint work with Irad Yavneh
1 12. Principles of Parameter Estimation The purpose of this lecture is to illustrate the usefulness of the various concepts introduced and studied in.
Fast Bayesian Matching Pursuit Presenter: Changchun Zhang ECE / CMR Tennessee Technological University November 12, 2010 Reading Group (Authors: Philip.
K-SVD Dictionary-Learning for Analysis Sparse Models
Learning sparse representations to restore, classify, and sense images and videos Guillermo Sapiro University of Minnesota Supported by NSF, NGA, NIH,
Extensions of wavelets
* * Joint work with Michal Aharon Freddy Bruckstein Michael Elad
Visual Recognition Tutorial
Dictionary-Learning for the Analysis Sparse Model Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa 32000,
Image Super-Resolution Using Sparse Representation By: Michael Elad Single Image Super-Resolution Using Sparse Representation Michael Elad The Computer.
Exploiting Statistical Dependencies in Sparse Representations Michael Elad The Computer Science Department The Technion – Israel Institute of technology.
Sparse and Overcomplete Data Representation
SRINKAGE FOR REDUNDANT REPRESENTATIONS ? Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa 32000, Israel.
Image Denoising via Learned Dictionaries and Sparse Representations
An Introduction to Sparse Representation and the K-SVD Algorithm
ITERATED SRINKAGE ALGORITHM FOR BASIS PURSUIT MINIMIZATION Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa.
New Results in Image Processing based on Sparse and Redundant Representations Michael Elad The Computer Science Department The Technion – Israel Institute.
Optimized Projection Directions for Compressed Sensing Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa.
* Joint work with Michal Aharon Guillermo Sapiro
Review of Matrix Algebra
Recent Trends in Signal Representations and Their Role in Image Processing Michael Elad The CS Department The Technion – Israel Institute of technology.
Visual Recognition Tutorial
ECE 530 – Analysis Techniques for Large-Scale Electrical Systems
A Weighted Average of Sparse Several Representations is Better than the Sparsest One Alone Michael Elad The Computer Science Department The Technion –
A Sparse Solution of is Necessarily Unique !! Alfred M. Bruckstein, Michael Elad & Michael Zibulevsky The Computer Science Department The Technion – Israel.
Sparse and Redundant Representation Modeling for Image Processing Michael Elad The Computer Science Department The Technion – Israel Institute of technology.
Topics in MMSE Estimation for Sparse Approximation Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa 32000,
Retinex by Two Bilateral Filters Michael Elad The CS Department The Technion – Israel Institute of technology Haifa 32000, Israel Scale-Space 2005 The.
Normalised Least Mean-Square Adaptive Filtering
AMSC 6631 Sparse Solutions of Linear Systems of Equations and Sparse Modeling of Signals and Images: Midyear Report Alfredo Nava-Tudela John J. Benedetto,
Machine Learning CUNY Graduate Center Lecture 3: Linear Regression.
Introduction to variable selection I Qi Yu. 2 Problems due to poor variable selection: Input dimension is too large; the curse of dimensionality problem.
Cs: compressed sensing
“A fast method for Underdetermined Sparse Component Analysis (SCA) based on Iterative Detection- Estimation (IDE)” Arash Ali-Amini 1 Massoud BABAIE-ZADEH.
Bayesian Extension to the Language Model for Ad Hoc Information Retrieval Hugo Zaragoza, Djoerd Hiemstra, Michael Tipping Presented by Chen Yi-Ting.
ECE 8443 – Pattern Recognition ECE 8423 – Adaptive Signal Processing Objectives: Deterministic vs. Random Maximum A Posteriori Maximum Likelihood Minimum.
CSC321: 2011 Introduction to Neural Networks and Machine Learning Lecture 11: Bayesian learning continued Geoffrey Hinton.
SUPA Advanced Data Analysis Course, Jan 6th – 7th 2009 Advanced Data Analysis for the Physical Sciences Dr Martin Hendry Dept of Physics and Astronomy.
Fields of Experts: A Framework for Learning Image Priors (Mon) Young Ki Baik, Computer Vision Lab.
Image Denoising Using Wavelets
PROBABILITY AND STATISTICS FOR ENGINEERING Hossein Sameti Department of Computer Engineering Sharif University of Technology Principles of Parameter Estimation.
Sparse Signals Reconstruction Via Adaptive Iterative Greedy Algorithm Ahmed Aziz, Ahmed Salim, Walid Osamy Presenter : 張庭豪 International Journal of Computer.
Computer Vision Lecture 6. Probabilistic Methods in Segmentation.
Data Modeling Patrice Koehl Department of Biological Sciences National University of Singapore
Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter Sparse & Redundant Representation.
Image Decomposition, Inpainting, and Impulse Noise Removal by Sparse & Redundant Representations Michael Elad The Computer Science Department The Technion.
A Weighted Average of Sparse Representations is Better than the Sparsest One Alone Michael Elad and Irad Yavneh SIAM Conference on Imaging Science ’08.
Image Priors and the Sparse-Land Model
Stein Unbiased Risk Estimator Michael Elad. The Objective We have a denoising algorithm of some sort, and we want to set its parameters so as to extract.
Single Image Interpolation via Adaptive Non-Local Sparsity-Based Modeling The research leading to these results has received funding from the European.
ECE 530 – Analysis Techniques for Large-Scale Electrical Systems Prof. Hao Zhu Dept. of Electrical and Computer Engineering University of Illinois at Urbana-Champaign.
Jianchao Yang, John Wright, Thomas Huang, Yi Ma CVPR 2008 Image Super-Resolution as Sparse Representation of Raw Image Patches.
From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images Alfred M. Bruckstein (Technion), David L. Donoho (Stanford), Michael.
Data Modeling Patrice Koehl Department of Biological Sciences
Sparsity Based Poisson Denoising and Inpainting
Chapter 7. Classification and Prediction
12. Principles of Parameter Estimation
Sparse and Redundant Representations and Their Applications in
Where did we stop? The Bayes decision rule guarantees an optimal classification… … But it requires the knowledge of P(ci|x) (or p(x|ci) and P(ci)) We.
Sparse and Redundant Representations and Their Applications in
5.2 Least-Squares Fit to a Straight Line
Sparse and Redundant Representations and Their Applications in
Improving K-SVD Denoising by Post-Processing its Method-Noise
* * Joint work with Michal Aharon Freddy Bruckstein Michael Elad
12. Principles of Parameter Estimation
Presentation transcript:

MMSE Estimation for Sparse Representation Modeling * Joint work with Irad Yavneh & Matan Protter The CS Department, The Technion Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa 32000, Israel April 6th, 2009

? Noise Removal? In this talk we focus on signal/image denoising … Remove Additive Noise Important: (i) Practical application; (ii) A convenient platform for testing basic ideas in signal/image processing. Many Considered Directions: Partial differential equations, Statistical estimators, Adaptive filters, Inverse problems & regularization, Wavelets, Example-based techniques, Sparse representations, … Main Massage Today: Several sparse representations can be found and used for better denoising performance – we introduce, motivate, discuss, demonstrate, and explain this new idea. MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Agenda Background on Denoising with Sparse Representations Using More than One Representation: Intuition Using More than One Representation: Theory A Closer Look At the Unitary Case Summary and Conclusions MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Part I Background on Denoising with Sparse Representations MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Relation to measurements Denoising By Energy Minimization Many of the proposed signal denoising algorithms are related to the minimization of an energy function of the form Relation to measurements Prior or regularization y : Given measurements x : Unknown to be recovered Thomas Bayes 1702 - 1761 This is in-fact a Bayesian point of view, adopting the Maximum-A-posteriori Probability (MAP) estimation. Clearly, the wisdom in such an approach is within the choice of the prior – modeling the signals of interest. MMSE Estimation for Sparse Representation Modeling By: Michael Elad

M Sparse Representation Modeling K N A fixed Dictionary Every column in D (dictionary) is a prototype signal (atom). M The vector  is generated randomly with few (say L for now) non-zeros at random locations and with random values. A sparse & random vector N MMSE Estimation for Sparse Representation Modeling By: Michael Elad

D-y = - Back to Our MAP Energy Function The L0 “norm” is effectively counting the number of non-zeros in . The vector  is the representation (sparse/redundant). Bottom line: Denoising of y is done by minimizing D-y = - or MMSE Estimation for Sparse Representation Modeling By: Michael Elad

The Solver We Use: Greed Based The MP is one of the greedy algorithms that finds one atom at a time [Mallat & Zhang (’93)]. Step 1: find the one atom that best matches the signal. Next steps: given the previously found atoms, find the next one to best fit the residual. The algorithm stops when the error is below the destination threshold. The Orthogonal MP (OMP) is an improved version that re-evaluates the coefficients by Least-Squares after each round. MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Orthogonal Matching Pursuit OMP finds one atom at a time for approximating the solution of Main Iteration 1. 2. 3. 4. 5. Initialization No Yes Stop MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Part II Using More than One Representation: Intuition MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Back to the Beginning. What If … Basic Questions: What could we do with such a set of competing solutions in order to better denoise y? Why should this help? How shall we practically find such a set of solutions? Consider the denoising problem and suppose that we can find a group of J candidate solutions such that Relevant work: [Larsson & Selen (’07)] [Schintter et. al. (`08)] [Elad and Yavneh (’08)] MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Motivation – General D D Why bother with such a set? Because each representation conveys a different story about the desired signal. Because pursuit algorithms are often wrong in finding the sparsest representation, and then relying on their solution is too sensitive. … Maybe there are “deeper” reasons? D D MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Our Motivation D D An intriguing relationship between this idea and the common-practice in example-based techniques, where several examples are merged. Consider the Non-Local-Means [Buades, Coll, & Morel (‘05)]. It uses (i) a local dictionary (the neighborhood patches), (ii) it builds several sparse representations (of cardinality 1), and (iii) it merges them. Why not take it further, and use general sparse representations? MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Our Answer: Randomizing the OMP Generating Many Representations * Larsson and Schnitter propose a more complicated and deterministic tree pruning method * Our Answer: Randomizing the OMP Main Iteration 1. 2. 3. 4. 5. Initialization For now, lets set the parameter c manually for best performance. Later we shall define a way to set it automatically No Yes Stop MMSE Estimation for Sparse Representation Modeling By: Michael Elad

D Lets Try + = Proposed Experiment : 100 Form a random dictionary D. Multiply by a sparse vector α0 ( ). Add Gaussian iid noise v with σ=1 and obtain . Solve the problem using OMP, and obtain . Use Random-OMP and obtain . Lets look at the obtained representations … + = 100 200 MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Some Observations We see that The OMP gives the sparsest solution Nevertheless, it is not the most effective for denoising. The cardinality of a representation does not reveal its efficiency. MMSE Estimation for Sparse Representation Modeling By: Michael Elad

This representation IS NOT SPARSE AT ALL but it gives The Surprise (at least for us) … Lets propose the average as our representation This representation IS NOT SPARSE AT ALL but it gives MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Here are the results of 1000 trials with the same parameters … Is It Consistent? … Yes! Here are the results of 1000 trials with the same parameters … Cases of zero solution ? MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Part III Using More than One Representation: Theory MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Our Signal Model D is fixed and known. The vector α is built by: Choosing the support s with probability P(s) from all the 2K possibilities Ω. For simplicity, assume that |s|=k is fixed and known. Choosing the αs coefficients using iid Gaussian entries N(0,σx). The ideal signal is x=Dα=Dsαs. K N A fixed Dictionary The p.d.f. P(α) and P(x) are clear and known MMSE Estimation for Sparse Representation Modeling By: Michael Elad

+ Adding Noise Noise Assumed: K + Noise Assumed: The noise v is additive white Gaussian vector with probability Pv(v) N A fixed Dictionary The conditional p.d.f.’s P(y|s), P(s|y), and even also P(x|y) are all clear and well-defined (although they may appear nasty). MMSE Estimation for Sparse Representation Modeling By: Michael Elad

The Key – The Posterior P(x|y) We have access to Oracle known support s MAP MMSE The estimation of α and multiplication by D is equivalent to the above. These two estimators are impossible to compute, as we show next. MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Lets Start with The Oracle * When s is known * Comments: This estimate is both the MAP and MMSE. The oracle estimate of x is obtained by multiplication by Ds. MMSE Estimation for Sparse Representation Modeling By: Michael Elad

We have seen this as the oracle’s probability for the support s: The MAP Estimation We have seen this as the oracle’s probability for the support s: MMSE Estimation for Sparse Representation Modeling By: Michael Elad

The MAP Estimation Implications: The MAP estimator requires to test all the possible supports for the maximization. In typical problems, this is impossible as there is a combinatorial set of possibilities. This is why we rarely use exact MAP, and we typically replace it with approximation algorithms (e.g., OMP). MMSE Estimation for Sparse Representation Modeling By: Michael Elad

This is the oracle for s, as we have seen before The MMSE Estimation This is the oracle for s, as we have seen before MMSE Estimation for Sparse Representation Modeling By: Michael Elad

The MMSE Estimation Implications: The best estimator (in terms of L2 error) is a weighted average of many sparse representations!!! As in the MAP case, in typical problems one cannot compute this expression, as the summation is over a combinatorial set of possibilities. We should propose approximations here as well. MMSE Estimation for Sparse Representation Modeling By: Michael Elad

This is our c in the Random-OMP The Case of |s|=k=1 This is our c in the Random-OMP The k-th atom in D Based on this we can propose a greedy algorithm for both MAP and MMSE: MAP – choose the atom with the largest inner product (out of K), and do so one at a time, while freezing the previous ones (almost OMP). MMSE – draw at random an atom in a greedy algorithm, based on the above probability set, getting close to P(s|y) in the overall draw. MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Bottom Line The MMSE estimation we got requires a sweep through all supports (i.e. combinatorial search) – impractical. Similarly, an explicit expression for P(x/y) can be derived and maximized – this is the MAP estimation, and it also requires a sweep through all possible supports – impractical too. The OMP is a (good) approximation for the MAP estimate. The Random-OMP is a (good) approximation of the Minimum-Mean-Squared-Error (MMSE) estimate. It is close to the Gibbs sampler of the probability P(s|y) from which we should draw the weights. Back to the beginning: Why Use Several Representations? Because their average leads to a provable better noise suppression. MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Relative Mean-Squared-Error Comparative Results 0.5 1 1.5 s 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 Relative Mean-Squared-Error The following results correspond to a small dictionary (20×30), where the combinatorial formulas can be evaluated as well. 1. Emp. Oracle 2. Theor. Oracle 3. Emp. MMSE 4. Theor. MMSE 5. Emp. MAP 6. Theor. MAP 7. OMP Parameters: N=20, K=30 True support=3 σx=1 J=10 (RandOMP) Averaged over 1000 experiments 8. RandOMP Known support 2 MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Part IV A Closer Look At the Unitary Case MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Few Basic Observations Let us denote (The Oracle) MMSE Estimation for Sparse Representation Modeling By: Michael Elad

This part becomes a constant, and thus can be discarded Back to the MAP Estimation We assume |s|=k fixed with equal probabilities * This part becomes a constant, and thus can be discarded This means that MAP estimation can be easily evaluated by computing β, sorting its entries in descending order, and choosing the k leading ones. MMSE Estimation for Sparse Representation Modeling By: Michael Elad

What about the MMSE? Could it have a simple closed-form solution too ? Closed-Form Estimation It is well-known that MAP enjoys a closed form and simple solution in the case of a unitary dictionary D. This closed-form solution takes the structure of thresholding or shrinkage. The specific structure depends on the fine details of the model assumed. It is also known that OMP in this case becomes exact. What about the MMSE? Could it have a simple closed-form solution too ? MMSE Estimation for Sparse Representation Modeling By: Michael Elad

= + The MMSE … Again This is the formula we got: We combine linearly many sparse representations (with proper weights) + = The result is one effective representation (not sparse anymore) MMSE Estimation for Sparse Representation Modeling By: Michael Elad

This is the formula we got: The MMSE … Again This is the formula we got: We change the above summation to where there are K contributions (one per each atom) to be found and used. We have developed a closed-form recursive formula for computing the q coefficients. MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Indicator function stating if j is in s Towards a Recursive Formula We have seen that the governing probability for the weighted averaging is given by Indicator function stating if j is in s MMSE Estimation for Sparse Representation Modeling By: Michael Elad

The Recursive Formula where MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Relative Mean-Squared-Error An Example 0.1 This is a synthetic experiment resembling the previous one, but with few important changes: Theor. MAP OMP 0.09 D is unitary The representation’s cardinality is 5 (the higher it is, the weaker the Random-OMP becomes) Dimensions are different: N=K=64 J=20 (RandOMP runs) Recursive MMSE Theor. MMSE RandOMP Relative Mean-Squared-Error 0.08 Known support Oracle 0.07 0.5 1 1.5 2 s MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Part V Summary and Conclusions MMSE Estimation for Sparse Representation Modeling By: Michael Elad

Sparsity and Redundancy are used for denoising of signals/images Today We Have Seen that … Sparsity and Redundancy are used for denoising of signals/images By finding the sparsest representation and using it to recover the clean signal How ? Can we do better? Today we have shown that averaging several sparse representations for a signal lead to better denoising, as it approximates the MMSE estimator. More on these (including the slides and the relevant papers) can be found in http://www.cs.technion.ac.il/~elad MMSE Estimation for Sparse Representation Modeling By: Michael Elad