1. M. Wu: ENEE631 Digital Image Processing (Spring'09) Optimal Bit Allocation and Unitary Transform in Image Compression Spring ’09 Instructor: Min Wu Electrical and Computer Engineering Department, University of Maryland, College Park   bb.eng.umd.edu (select ENEE631 S’09)   minwu@eng.umd.edu ENEE631 Spring’09 Lecture 14 (3/23/2009)

2. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[2] Overview and Logistics Last Time: Wavelet transform and coding –EZW example: exploit coefficient structures to remove redundancy –Haar basis and transform; Considerations for wavelet filters –JPEG 2000 Today –Optimal bit allocation: “equal slope” in R-D curves –Optimal unitary transform – KLT Midterm exam: Wednesday March 25 in class –Sample exam from previous year was posted online for reference –Emphasis: (1) Theoretical foundations: 2-D Signals & Systems, 2-D Fourier analysis, Quantization, Basis vectors/images and Unitary transform; (2) Principles and algorithmic techniques for: human visual system (color and monochorome vision); image enhancement and restoration; image coding. UMCP ENEE631 Slides (created by M.Wu © 2004)

3. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[3] Review: Non-Dyadic Decomposition – Wavelet Packets Figures are from slides at Gonzalez/ Woods DIP book 2/e website (Chapter 7)

4. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[4] Bit Allocation in Image Coding Bit Allocation in Image Coding Focus on MSE-based optimization; Can further adjust based on HVS UMCP ENEE631 Slides (created by M.Wu © 2004)

5. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[5] Here r k is # bits allocated for v(k);  k 2 : variance of k-th coeff. v(k); f(-): a func. relating bit rate to distortion based on a coeff’s p.d.f. Bit Allocation How many bits to be allocated for each coefficient? –Related to each coeff’s variance (and probability distribution) –More bits for high variance  k 2 to keep total MSE small “Reverse Water-filling” – –Try to keep same amount of error in each frequency band UMCP ENEE631 Slides (created by M.Wu © 2001)

6. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[6] Rate/Distortion Allocation via Reverse Water-filling UMCP ENEE631 Slides (created by M.Wu © 2001/2004) Optimal solution for Gaussian distributed coefficients Idea: try to keep same amount of error in each freq. band; no need to spend bits to represent coeff. w/ smaller variance than water level –Results based on R-D function and via Lagrange-multiplier optimization –Note the convex shape of R-D function for Gaussian => slope becomes milder; require more bits to further reduce distortion D RDRD 22 Given total rate R, determine  and then D; or vice versa given D, determine R Optimal Not Optimal

7. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[7] Details on Reverse Water-filling Solution Construct func. using Lagrange multiplier Necessary condition  Keep the same marginal gain Equal Slope Condition for bit allocation D RDRD 22 Resulting in equal distortions on Gaussian r.v’s as interpreted in “reverse water filling”

8. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[8] Key Result in Rate Allocation: Equal R-D Slope Keep the slope in R-D curve the same (not just for Gaussian r.v) –Otherwise some bits can be applied to other r.v. for better “return” in reducing the overall distortion If all r.v. are Gaussian, the same slope in R-D curves for these r.v. correspond to an identical amount of distortion. R(D) = ½ [ log  2 – log D ] => dR/dD = – 1 / (2D) Use operational/measured rate-distortion information for general r.v. UMCP ENEE631 Slides (created by M.Wu © 2004) Rate allocation results: => Equal slope (o.w. exist a better allocation strategy)

9. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[9] Optimal Transform UMCP ENEE631 Slides (created by M.Wu © 2004)

10. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[10] Optimal Transform Recall: Why use transform in coding/compression? –Decorrelate the correlated data –Pack energy into a small number of coefficients –Interested in unitary/orthogonal or approximate orthogonal transforms u Energy preservation s.t. quantization effects can be better understood and controlled Unitary transforms we’ve dealt so far are data independent –Transform basis/filters are not depending on the signals we are processing u Can hard-wire them in encoder and decoder What unitary transform gives the best energy compaction and decorrelation? –“Optimal” in a statistical sense to allow the codec works well with many images=> Signal statistics would play an important role UMCP ENEE631 Slides (created by M.Wu © 2004)

11. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[11] Review: Correlation After a Linear Transform Consider an Nx1 zero-mean random vector x – Covariance (or autocorrelation) matrix R x = E[ x x H ] u give ideas of correlation between elements u R x is a diagonal matrix for if all N r.v.’s are uncorrelated Apply a linear transform to x: y = A x What is the correlation matrix for y ? R y = E[ y y H ] = E[ (Ax) (Ax) H ] = E[ A x x H A H ] = A E[ x x H ] A H = A R x A H Decorrelation: try to search for A that can produce a decorrelated y (equiv. a diagonal correlation matrix R y ) UMCP ENEE631 Slides (created by M.Wu © 2004)

12. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[12] Review: Correlation After a Linear Transform Consider an Nx1 zero-mean random vector x – Covariance (or autocorrelation) matrix R x = E[ x x H ] u give ideas of correlation between elements u R x is a diagonal matrix for if all N r.v.’s are uncorrelated Apply a linear transform to x: y = A x What is the correlation matrix for y ? R y = E[ y y H ] = E[ (Ax) (Ax) H ] = E[ A x x H A H ] = A E[ x x H ] A H = A R x A H Decorrelation: try to search for A that can produce a decorrelated y (equiv. a diagonal correlation matrix R y ) UMCP ENEE631 Slides (created by M.Wu © 2004)

13. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[13] K-L Transform (Principal Component Analysis) Eigen decomposition of R x : R x u k = k u k –Recall the properties of R x u Hermitian (conjugate symmetric R H = R); u Nonnegative definite (real non-negative eigen values) Karhunen-Loeve Transform (KLT) y = U H x  x = U y with U = [ u 1, … u N ] –KLT is a unitary transform: to represent x with basis vectors that are the orthonormalized eigenvectors of R x –R x U = [ 1 u 1, … N u N ] = U diag{ 1, 2, …, N } => U H R x U = diag{ 1, 2, …, N } i.e. KLT performs decorrelation u we often order {u i } so that 1  2  …  N –Also known as Hotelling transform or Principle Component Analysis (PCA) UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

14. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[14] Review: Correlation Matrix

15. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[15] Properties of K-L Transform Decorrelation –E[ y y H ]= E[ (U H x) (U H x) H ]= U H E[ x x H ] U = diag{ 1, 2, …, N } –Note: Other matrices (unitary or nonunitary) may also decorrelate the transformed sequence [ Jain’s e.g.5.7 pp166] Minimizing MSE under basis restriction –If only allow to keep m coefficients for any 1  m  N, what’s the best way to minimize reconstruction error?  Retain the coefficients corresponding to the eigenvectors of the first m largest eigen values u Reference: Theorem 5.1 and Proof in Jain’s Book (pp166) UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

16. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[16] KLT Basis Restriction Basis restriction –Keep only a subset of m transform coefficients and then perform inverse transform (1  m  N) –Basis restriction error: MSE between original & new sequences Goal: to find the forward and backward transform matrices to minimize the restriction error for each and every m –The minimum is achieved by KLT arranged according to the decreasing order of the eigenvalues of R UMCP ENEE631 Slides (created by M.Wu © 2004)

17. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[17] K-L Transform for Images Work with 2-D autocorrelation function –R(m,n; m’,n’)= E[ x(m, n) x(m’, n’) ] for all 0  m, m’, n, n’  N-1 –K-L Basis images is the orthonormalized eigen function of 4-variable function R( ) Rewrite images into vector form (N 2 x1) –Need solve the eigen problem for N 2 xN 2 matrix! ~ O(N 6 ) Reduced computation for separable R –R(m,n; m’,n’)= r 1 (m,m’) r 2 (n,n’) –Only need solve the eigen problem for two NxN matrices ~ O(N 3 ) –KLT can now be performed separably on rows and columns u Reducing the transform complexity from O(N 4 ) to O(N 3 ) UMCP ENEE631 Slides (created by M.Wu © 2001)

18. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[18] Pros and Cons of K-L Transform Optimality –Decorrelation and MMSE for the same# of partial transform coeff. Applications –(non-universal) compression –Pattern recognition: e.g., eigen faces –Analyze the principal (“dominating”) components and reduce feature dimensions Data dependent: –Have to estimate the 2 nd -order statistics from a collection of images to determine the transform –Need to let decoder know the KL basis vectors used –Can we get data-independent transform with similar performance? u DCT for highly correlated data UMCP ENEE631 Slides (created by M.Wu © 2001)

19. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[19] Energy Compaction of DCT vs. KLT DCT has excellent energy compaction for highly correlated data DCT is a good replacement for K-L –Close to optimal for highly correlated data –Not depend on specific data like K-L does –Fast algorithm available [ref and statistics: Jain’s pp153, 168-175] UMCP ENEE631 Slides (created by M.Wu © 2004)

20. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[20] Energy Compaction of DCT vs. KLT (cont’d) Preliminaries –The matrices R, R -1, and  R -1 share the same set of eigen vectors –DCT basis vectors are eigenvectors of a symmetric tri-diagonal matrix Q c –Covariance matrix R of 1 st -order stationary Markov sequence (AR process) has an inverse in the form of symmetric tri-diagonal matrix DCT is close to KLT on 1 st -order stationary Markov –For highly correlated sequence, a scaled version of R -1 approx. Q c UMCP ENEE631 Slides (created by M.Wu © 2004)

21. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[21] SVD – Singular Value Decomposition u Jain’s book Section 5.13 UMCP ENEE631 Slides (created by M.Wu © 2004)

22. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[22] Hadamard Transform From Mani Thomas U.Del CISC489/689 2D FT http://vims.cis.udel.edu/~mani/TACourses/Spring06/cv/index.html

23. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[23] Summary and Review on Unitary Transform Representation with orthonormal basis  Unitary transform –Preserve energy Common unitary transforms –KLT; DFT, DCT, Haar … Which transform to choose? –Depend on need in particular task/application –DFT ~ reflect physical meaning of frequency or spatial frequency –KLT ~ optimal in energy compaction –DCT ~ real-to-real, and close to KLT’s energy compaction => A comparison table in Jain’s book Table 5.3 UMCP ENEE631 Slides (created by M.Wu © 2001)

24. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[24] Summary of Today’s Lecture Optimal bit allocation: “equal slope” in R-D curves –Lead to reverse water filling for Gaussian coefficients Optimal transform – KLT Next lecture –Compression of image sequence and video Readings: –Gonzalez’s 3/e book Section 8.2.8 –Optimal bit allocation: article by Ortega-Ramchandran in Nov. ’98 issue of IEEE Signal Processing Magazine u More reference: Jain’s book Section 2.13 & 11.4 –KLT: Jain’s book 5.11, 5.6, 5.14; 2.7-2.9 UMCP ENEE631 Slides (created by M.Wu © 2004)

25. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[25] More on Rate-Distortion Based Bit Allocation in Image Coding UMCP ENEE631 Slides (created by M.Wu © 2004)

26. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[26] Details on Reverse Water-filling Solution Construct func. using Lagrange multiplier Necessary condition  Keep the same marginal gain Necessary condition for choosing D RDRD 22

27. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[27] General Constrained Optimization Problem Define Necessary condition: “Kuhn-Tucker conditions”

28. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[28] Lagrangian Optimiz. for Indep. Budget Constraint Previous: fix distortion, minimize total rate Alternative: fix total rate (bit budget), minimize distortion (Discrete) Lagrangian optimization on general source –“Constant slope optimization” (e.g. in Box 6 Fig. 17 of Ortega’s tutorial) –Need to determine the quantizer q(i) for each coding unit i –Lagrangian cost for each coding unit u use a line with slope - to intersect with each operating point (Fig.14) –For a given operating quality, the minimum can be computed independently for each coding unit => Find operating quality satisfying the rate constraint

29. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[29] (from Box 6 Fig. 17 of Ortega’s tutorial) (from Fig. 14 of Ortega’s tutorial)

30. M. Wu: ENEE631 Digital Image Processing (Spring'09) Lec14 – Bit Allocation & KLT[30] (from Fig. 19 of Ortega’s tutorial)