Alfredo Nava-Tudela John J. Benedetto, advisor Sparse Solutions of Linear Systems of Equations and Sparse Modeling of Signals and Images Alfredo Nava-Tudela John J. Benedetto, advisor AMSC 663

Outline Problem: Find “sparse solutions” of Ax = b Why do we care? An application in signal processing: compression Working definition of “sparse solution” Why things eventually work, for some cases How do we find sparse solutions? OMP algorithm Case study: What happens when we combine the DCT and DWT? Conclusions/Recap Project timeline AMSC 663

Problem Let A be an n by m matrix with n < m, and rank(A)=n. We want to solve Ax = b, where b is a data or signal vector, and x is the solution with the fewest number of non-zero entries possible, that is, the “sparsest” one. Observations: - A is underdetermined and, since rank(A)=n, there is an infinite number of solutions. Good! - How do we find the “sparsest” solution? What does this mean in practice? Is there a unique sparsest solution? AMSC 663

231 kb, uncompressed, 320x240x3x8 bit But, why do we care? We live in a digital media era. Bandwidths are different and matter: cellphone vs DSL vs Gigabit, etc. 231 kb, uncompressed, 320x240x3x8 bit 74 kb, compressed 3.24:1 JPEG AMSC 663

But, why do we care? 512 x 512 Pixels, 75:1, 10.6 Kbyte 24-Bit RGB, Ever increasing size of files to transmit demand better compression algorithms. At the core of the JPEG and JPEG2000 standards, the DCT and the DWT. Transform encoding 512 x 512 Pixels, 24-Bit RGB, Size 786 Kbyte 75:1, 10.6 Kbyte JPEG2000 AMSC 663

“Sparsity” equals compression Both JPEG and JPEG2000 achieve their compression mainly because at their core one finds a linear transform (DCT and DWT, respectively) that reduces the number of non-zero entries required to represent the data, within an acceptable error. We can then think of signal compression in terms of our problem Ax = b, x is sparse, b is dense, store x! AMSC 663

Working definitions of “sparse” Convenient to introduce the l0 “norm”: ||x||0 = # {k : xk ≠ 0} (P0): minx ||x||0 subject to ||Ax - b||2 = 0 (P0e): minx ||x||0 subject to ||Ax - b||2 < e Observation: Solving (P0) is NP-hard, bummer. We can cast our problem as a minimization problem AMSC 663

Some theoretical results Definition: The spark of a matrix A is the minimum number of linearly dependent columns of A. We write spark(A) to represent this number. Theorem: If there is a solution x to Ax = b, and ||x||0 < spark(A) / 2, then x is the sparsest solution. That is, if y ≠ x also solves the equation, then ||x||0 < ||y||0. Observation: Computing spark(A) is combinatorial, therefore hard. Alternative? Address the question of uniqueness AMSC 663

More theoretical results Definition: The mutual coherence of a matrix A is the number Lemma: spark(A) ≥ 1+1/mu(A). Theorem: If x solves Ax = b, and ||x||0 < (1+mu(A)-1)/2, then x is the sparsest solution, as before. Observation: mu(A) is a lot easier and faster to compute, but 1+1/mu(A) far worse bound than spark(A), in general. Mutual coherence gives the worse case scenario criterion for uniqueness Mention that mutual coherence also plays a role in proving convergence of algorithms, but won’t see this here AMSC 663

Finding sparse solutions:OMP Orthogonal Matching Pursuit algorithm: OMP is one of a family of algorithms called “greedy algorithms”. Goal: implement, validate and test. Matlab. There are implementations available online against which to compare my implementation. OMP known to converge when ||x||0 < (1+1/mu(A))/2. AMSC 663

Revisiting compression Propose to study the compression properties of the matrix A = [DCT,DWT] and compare it with the compression properties of DCT or DWT alone. Study the behavior of OMP for this problem. Many wavelet options available to try, e.g., reversible 5/3 or floating-point 9/7 Daubechies, as in JPEG2000. Interested in compression vs error graph properties. Matrix A can typically be the concatenation of two basis AMSC 663

Conclusions/Recapitulation Finding sparse solutions to the linear system of equations Ax = b, when A is an n by m full rank matrix and n < m, is of interest to the signal processing community. There are simple criteria to assert the uniqueness of a given sparse solution. There are algorithms to find sparse solutions, e.g., OMP; and their convergence can be guaranteed when there are “sufficiently sparse” solutions. - Studies on the performance of OMP missing when A is the concatenation of unitary matrices. AMSC 663

Project timeline Oct. 15, 2010: Complete project proposal. Oct. 18 - Nov. 5: Implement OMP. Nov. 8 - Nov. 26: Validate OMP. Nov. 29 - Dec. 3: Write mid-year report. Dec. 6 - Dec. 10: Prepare mid-year oral presentation. Some time after that, give mid-year oral presentation. Jan. 24 - Feb. 11: Testing of OMP. Reproduce paper results. Jan. 14 - Apr. 8: Testing of OMP on A = [DCT,DWT]. Apr. 11 - Apr. 22: Write final report. Apr. 25 - Apr. 29: Prepare final oral presentation. Some time after that, give final oral presentation. AMSC 663

References A. M. Bruckstein, D. L. Donoho, and M. Elad, From sparse solutions of systems of equations to sparse modeling of signals and images, SIAM Review, 51 (2009), pp. 34–81. S. Mallat, A Wavelet Tour of Signal Processing, Academic Press, 1998. B. K. Natarajan, Sparse approximate solutions to linear systems, SIAM Journal on Computing, 24 (1995), pp. 227-234. G. W. Stewart, Introduction to Matrix Computations, Academic Press, 1973. D. S. Taubman and M. W. Mercellin, JPEG 2000: Image Compression Funda- mentals, Standards and Practice, Kluwer Academic Publishers, 2001. G. K. Wallace, The JPEG still picture compression standard, Communications of the ACM, 34 (1991), pp. 30-44. http://www.stanford.edu/class/ee398a/handouts/lectures/08-JPEG.pdf AMSC 663