Download presentation
Presentation is loading. Please wait.
1
The K–SVD Design of Dictionaries for Redundant and Sparse Representation of Signals
* * Joint work with Michal Aharon Freddy Bruckstein Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa 32000, Israel SPIE – Wavelets XI San-Diego: August 2nd, 2005
2
Agenda A Visit to Sparseland
Motivating redundancy & Sparsity 2. The Quest for a Dictionary – Fundamentals Common Approaches? 3. The Quest for a Dictionary – Practice Introducing the K-SVD 4. Results Preliminary results and applications Welcome to Sparseland NOTE: This lecture concentrates on the fundamentals of this work, and thus does not fully overlap the accompanying paper. The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
3
M Generating Signals in Sparseland
K N A fixed Dictionary Every column in D (dictionary) is a prototype signal (Atom). M The vector is generated randomly with few non-zeros in random locations and random values. A sparse & random vector N The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
4
M Sparseland Signals Are Interesting Sparseland is here !?
Simple: Every generated signal is built as a linear combination of few atoms from our dictionary D Rich: A general model: the obtained signals are a special type mixture-of-Gaussians (or Laplacians). Popular: Recent work on signal and image processing adopt this model and successfully deploys it to applications. Sparseland is here !? Multiply by D M The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
5
? M Signal Processing in Sparseland noise Multiply by D
A sparse & random vector M Practical ways to get ? ? 4 Major Questions Is or even close? Recent results give optimistic answers to these questions How effective? How do we get D? OUR FOCUS TODAY!! The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
6
Agenda A Visit to Sparseland
Motivating redundancy & Sparsity 2. The Quest for a Dictionary – Fundamentals Common approaches? 3. The Quest for a Dictionary – Practice Introducing the K-SVD 4. Results Preliminary results and applications The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
7
M Problem Setting Multiply by D Given a set of data examples, we
assume the Sparseland model and ask The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
8
Choose D – Modeling Approach
Replace the model with another, well-defined simple mathematical, model (e.g. images as piece-wise C2 smooth regions with C2 smooth edges) and fit a dictionary accordingly, based on existing methods. Examples: Curvelet [Candes & Donoho] Contourlet [Do & Vetterli] Bandlet [Mallat & Le-Pennec] and others ... Pros: Build on existing methods, Fast transforms, Proven optimality for the model. Cons: Relation to Sparseland ? Linearity? How to adapt to other signals? Bad for “smaller” signal families. The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
9
OUR APPROACH FOR TODAY Choose D – Training Approach Pros: Cons:
Train the dictionary directly based on the given examples, optimizing w.r.t. sparsity and other desired properties (normalized atoms, etc.). Examples: ML (HVS) [Field & Olshausen] MAP [Lewicki & Sejnowski] MOD [Engan et. al.] ICA-like [Kreutz-Delgado et. al.] and others ... Pros: Fits the data, Design fits the true objectives, Can be adapted to small families. Cons: No fast version, Slow training, Scalability issues? The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
10
Agenda A Visit to Sparseland
Motivating redundancy & Sparsity 2. The Quest for a Dictionary – Fundamentals Common Approaches? 3. The Quest for a Dictionary – Practice Introducing the K-SVD 4. Results Preliminary results and applications The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
11
X A D Practical Approach – Objective
Each example is a linear combination of atoms from D Each example has a sparse representation with no more than L atoms Field & Olshausen (96’) Engan et. al. (99’) Lewicki & Sejnowski (00’) Cotter et. al. (03’) Gribonval et. al. (04’) (n,K,L are assumed known) The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
12
Column-by-Column by Mean computation over the relevant examples
K–Means For Clustering Clustering: An extreme sparse coding D Initialize D Sparse Coding Nearest Neighbor XT Dictionary Update Column-by-Column by Mean computation over the relevant examples The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
13
Column-by-Column by SVD computation
The K–SVD Algorithm – General Aharon, Elad, & Bruckstein (`04) D Initialize D Sparse Coding Use MP or BP XT Dictionary Update Column-by-Column by SVD computation The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
14
D is known! For the jth item we solve
K–SVD: Sparse Coding Stage D D is known! For the jth item we solve XT Pursuit Problem !!! The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
15
D K–SVD: Dictionary Update Stage
Gk : The examples in and that use the column dk. D The content of dk influences only the examples in Gk. Let us fix all A and D apart from the kth column and seek both dk and the kth column in A to better fit the residual! The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
16
D K–SVD: Dictionary Update Stage We should solve: ResidualE
dk is obtained by SVD on the examples’ residual in Gk. The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
17
Agenda A Visit to Sparseland
Motivating redundancy & Sparsity 2. The Quest for a Dictionary – Fundamentals Common Approaches? 3. The Quest for a Dictionary – Practice Introducing the K-SVD 4. Results Preliminary results and applications The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
18
D D K–SVD: A Synthetic Experiment Results
Create A 2030 random dictionary with normalized columns Generate 2000 signal examples with 3 atoms per each and add noise D Train a dictionary using the KSVD and MOD and compare Results The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
19
K–SVD on Images Overcomplete Haar K-SVD: 10,000 sample 8-by-8 images.
441 dictionary elements. Approximation method: OMP K-SVD: The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
20
Filling–In Missing Pixels
Given an image with missing values Get the recovered image Apply pursuit (per each block of size(88) using a decimated dictionary with rows removed Multiply the found representation by the complete dictionary 60% missing pixels Haar Results Average # coefficients 4.42 RMSE: 21.52 K-SVD Results Average # coefficients 4.08 RMSE: 11.68 The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
21
Summary Today we discussed: Open Questions:
1. A Visit to Sparseland Motivating redundancy & Sparsity 2. The Quest for a Dictionary – Fundamentals Common Approaches? 3. The Quest for a Dictionary – Practice Introducing the K-SVD 4. Results Preliminary results and applications Open Questions: Scalability – treatment of bigger blocks and large images. Uniqueness? Influence of noise? Equivalence? A guarantee to get the perfect dictionary? Choosing K? What forces govern the redundancy? Other applications? … The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
22
Supplement Slides The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
23
Is D unique in explaining the origin of the signals?
Uniqueness? Multiply by D M Is D unique in explaining the origin of the signals? (n,K,L are assumed known) The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
24
Uniqueness? YES !! M If is rich enough* and if Uniqueness
then D is unique. Uniqueness Aharon, Elad, & Bruckstein (`05) M “Rich Enough”: The signals from could be clustered to groups that share the same support. At least L+1 examples per each are needed. Comments: This result is proved constructively, but the number of examples needed to pull this off is huge – we will show a far better method next. A parallel result that takes into account noise could be constructed similarly. The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
25
Naïve Compression K-SVD dictionary Haar dictionary DCT dictionary
OMP with error bound The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
26
Naïve Compression DCT BPP = 1.01 RMSE = 8.14 K-SVD Haar BPP = 0.502
The K–SVD: Design of Dictionaries for Redundant and Sparse representation of Signals
Similar presentations
