1. ENEE631 Digital Image Processing (Spring'04) Transform Coding and JPEG Spring ’04 Instructor: Min Wu ECE Department, Univ. of Maryland, College Park   www.ajconline.umd.edu (select ENEE631 S’04)   minwu@eng.umd.edu Based on ENEE631 Spring’04 Section 10

2. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [2] Overview and Logistics Last Time: –Unitary transform and properties –DCT transform Today: –Continue on 2-D unitary transform and basis images –Transform coding –JPEG UMCP ENEE631 Slides (created by M.Wu © 2004)

3. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [3] Overview and Logistics Last Time: –JPEG compression standard: Baseline block-DCT based algorithm u lossy part: quantization with different step size for each coeff. Band u lossless part: differential coding, run-length coding, Huffman –Start Subband and Wavelet based compression Today –Continue on Wavelet-based image coding => Exploit the structures between coefficients for removing redundancy UMCP ENEE631 Slides (created by M.Wu © 2004)

4. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [4] Transform Coding UMCP ENEE631 Slides (created by M.Wu © 2004)

5. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [5] Transform Coding Use transform to pack energy to only a few coeff. How many bits to be allocated for each coeff.? –More bits for coeff. with high variance  k 2 to keep total MSE small –Also determined by perceptual importance From Jain’s Fig.11.15 UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

6. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [6] Zonal Coding and Threshold Coding Zonal coding –Only transmit a small predetermined zone of transformed coeff. Threshold coding –Transmit coeff. that are above certain thresholds Compare – Threshold coding is inherently adaptive u introduce smaller distortion for the same # of coded coeff. – Threshold coding need overhead in specifying index of coded coeff. u run-length coding helps to reduce overhead UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

7. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [7] Determining Block Size Why block based? –High transform computation complexity for large block u O( m log m  m ) per block in tranf. for (MN/m 2 ) blocks u complexity in bit allocation –Block transform captures local info. better than global transform Rate & complexity vs. block size –Commonly used block size ~ 8x8 From Jain’s Fig.11.16 complexity UMCP ENEE631 Slides (created by M.Wu © 2001)

8. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [8] Block-based Transform Coding Encoder –Step-1 Divide an image into m x m blocks and perfrom transform –Step-2 Determine bit-allocation for coefficients –Step-3 Design quantizer and quantize coefficients (lossy!) –Step-4 Encode quantized coefficients Decoder From Jain’s Fig.11.17 UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

9. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [9] How to Encode Quantized Coeff. in Each Block Basic tools –Entropy coding (Huffman, etc.) and run-length coding –Predictive coding ~ esp. for DC Ordering –zig-zag scan for block-DCT to better achieve run-length coding gain Horizontal frequency Vertical frequency DC AC 01 AC 07 AC 70 AC 77  low-frequency coefficients, then high frequency coefficients UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

10. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [10] Summary: List of Compression Tools Lossless encoding tools –Entropy coding: Huffman, Lemple-Ziv, and others (Arithmetic coding) –Run-length coding Lossy tools for reducing redundancy –Quantization: scalar quantizer vs. vector quantizer –Truncations: discard unimportant parts of data Facilitating compression via Prediction –Encode prediction parameters and residues with less bits Facilitating compression via Transforms –Transform into a domain with improved energy compaction UMCP ENEE631 Slides (created by M.Wu © 2004)

11. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [11] Put Basic Tools Together: JPEG Image Compression Standard UMCP ENEE631 Slides (created by M.Wu © 2004)

12. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [12] JPEG Compression Standard (early 1990s) JPEG - Joint Photographic Experts Group –Compression standard of generic continuous-tone still image –Became an international standard in 1992 Allow for lossy and lossless encoding of still images –Part-1 DCT-based lossy compression u average compression ratio 15:1 –Part-2 Predictive-based lossless compression Sequential, Progressive, Hierarchical modes –Sequential ~ encoded in a single left-to-right, top-to-bottom scan –Progressive ~ encoded in multiple scans to first produce a quick, rough decoded image when the transmission time is long –Hierarchical ~ encoded at multiple resolution to allow accessing low resolution without full decompression UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

13. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [13] Baseline JPEG Algorithm “Baseline” –Simple, lossy compression u Subset of other DCT-based modes of JPEG standard A few basics –8x8 block-DCT based coding –Shift to zero-mean by subtracting 128  [-128, 127] u Allows using signed integer to represent both DC and AC coeff. –Color ( YCbCr / YUV ) and downsample u Color components can have lower spatial resolution than luminance –Interleaving color components => Flash demo on Baseline JPEG algorithm by Dr. Ken Lam (HK PolyTech Univ.) (Based on Wang’s video book Chapt.1) UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

14. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [14] Block Diagram of JPEG Baseline From Wallace’s JPEG tutorial (1993)

15. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [15] More on Color Interleaving – Different components are ordered into Minimum Coding Unit (MCU) – MCUs define repeating interleaving patterns Minimum coding unit (MCU) MCU1 = {Y00, Y01, Y10, Y11, U00, V00} MCU2 = {Y02, Y03, Y12, Y13, U01, V01}    YUV UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

16. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [16] 475 x 330 x 3 = 157 KB luminance From Liu’s EE330 (Princeton)

17. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [17] RGB Components From Liu’s EE330 (Princeton)

18. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [18] Y U V (Y Cb Cr) Components Assign more bits to Y, less bits to Cb and Cr From Liu’s EE330 (Princeton)

19. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [19] JPEG Compression (Q=75%) 45 KB, compression ration ~ 4:1 From Liu’s EE330 (Princeton)

20. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [20] Lossless Coding Part in JPEG Differentially encode DC –(lossy part: DC differences are then quantized.) AC coefficients in one block – Zig-zag scan after quantization for better run-length u save bits in coding consecutive zeros – Represent each AC run-length using entropy coding u use shorter codes for more likely AC run-length symbols UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

21. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [21] Lossy Part in JPEG Quantization (adaptive bit allocation) –Different quantization step size for different coeff. bands –Use same quantization matrix for all blocks in one image –Choose quantization matrix to best suit the image –Different quantization matrices for luminance and color components Default quantization table –“Generic” over a variety of images Quality factor “Q” –Scale the quantization table –Medium quality Q = 50% ~ no scaling –High quality Q = 100% ~ unit quantization step size –Poor quality ~ small Q, larger quantization step u visible artifacts like ringing and blokiness UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

22. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [22] Uncompressed (100KB) JPEG 75% (18KB) JPEG 50% (12KB)JPEG 30% (9KB)JPEG 10% (5KB) UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

23. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [23] JPEG Compression (Q=75% & 30%) 45 KB 22 KB From Liu’s EE330 (Princeton)

24. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [24] Y Cb Cr After JPEG (Q=30%) From Liu’s EE330 (Princeton) JPEG Cb JPEG Cr

25. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [25] Lossless Coding Part in JPEG: Details Differentially encode DC –( SIZE, AMPLITUDE ), with amplitude range in [-2048, 2047] AC coefficients in one block – Zig-zag scan for better run-length – Represent each AC with a pair of symbols u Symbol-1: ( RUNLENGTH, SIZE )  Huffman coded u Symbol-2: AMPLITUDE  Variable length coded RUNLENGTH  [0,15] # of consecutive zero-valued AC coefficients preceding the nonzero AC coefficient  [0,15] SIZE  [0 to 10 in unit of bits] # of bits used to encode AMPLITUDE AMPLITUDE  in range of [-1023, 1024] UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)

26. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [26] UMCP ENEE631 Slides (created by M.Wu © 2004) Table is from slides at Gonzalez/ Woods DIP book website (Chapter 8)

27. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [27] Summary of Today’s Lecture JPEG compression standard –Baseline block-DCT based algorithm u lossy part: quantization with different step size for each coeff. Band u lossless part: differential coding, run-length coding, Huffman Next week: –Wrap up with JPEG compression –Subband coding and Wavelet based compression Readings –Jain’s book 11.5 Gonzalez’s book 8.5.2, 8.6.2 –Wallace’s tutorial paper on JPEG (see course webpage) UMCP ENEE631 Slides (created by M.Wu © 2004)

28. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [28] Reference Compandor design of Waveform coding book Transform coding article in Sig.Proc. Magzine Bovik’s book Chapt.5.1 (Lossless coding) – huffman, lempel-ziv, etc. Wallace’s JPEG paper Figure is from slides at Gonzalez/ Woods DIP book website (Chapter 3)

29. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [29] Bit Allocation in Image Coding UMCP ENEE631 Slides (created by M.Wu © 2004)

30. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [30] where  k 2 -- variance of k-th coeff. v(k); n k -- # bits allocated for v(k) f(-) – quantization distortion func. Bit Allocation How many bits to be allocated for each coeff.? –Determined by the variance of coeff. –More bits for high variance  k 2 to keep total MSE small “Inverse Water-filling” (from info. theory) – –Try to keep same amount of error in each freq. band  See Jain’s pp501 for detailed bit-allocation algorithm UMCP ENEE631 Slides (created by M.Wu © 2001)

31. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [31] Revisiting Quantizer Quantizer achieves compression in a lossy way –Lloyd-Max quantizer minimize MSE distortion with a given rate Need at least how many # bits for certain amount of error? –Rate-Distortion theory Rate distortion func. of a r.v. –Minimum average rate R D bits/sample required to represent this r.v. while allowing a fixed distortion D –For Gaussian r.v. and MSE u Convex shape (slope is becoming milder) (See info. theory course for details on R-D theory) D RDRD 22 UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

32. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [32] Bit Allocation Among Indep. r.v. of Different Var. Problem formulation –To encode a set of independent Gaussian r.v. {X1, …, Xn}, X k ~ N( 0,  k 2 ) –Allocate R k bits to represent each r.v. X k, incurring distortion D k –Total bit cost is R = R1+…+Rn –Total MSE distortion D = D1+…+Dn  What is the best bit allocation {R1, …, Rn} such that R is minimized and total distortion D  D (req) ?  What is the best bit allocation {R1, …, Rn} such that D is minimized and total rate R  R (req) ? –Recall R k = max( ½ * log(  k 2 / D k ), 0 ) –Solving the constrained optimization problem using Lagrange multiplier  0 UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

33. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [33] Rate/Distortion Allocation via Reverse Water-filling How many bits to be allocated for each coeff.? –Determined by the variance of the coefficients –More bits for high variance  k 2 to keep total MSE small Optimal solution for Gaussian: “Reverse Water-filling” – –Idea: Try to keep same amount of error in each freq. band and no need to spend bit resource to represent coeff. w/ small variance than water level – –Results based on R-D function & via Lagrange-multiplier optimization given D, determine  and then R D ; or vice versa for given R D => “Equal slope” idea can be extended to other convex (operational) R-D functions UMCP ENEE631 Slides (created by M.Wu © 2001/2004)

34. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [34] Key Result in Rate Allocation: Equal R-D Slope Keep the slope in R-D curve the same (not necessarily Gaussian r.v) –Otherwise the bits can be applied to other r.v. for better “return” in reducing the overall distortion If all r.v. are Gaussian, the slope at the same distortion are identical in R-D curves for the two r.v. UMCP ENEE631 Slides (created by M.Wu © 2004)

35. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [35] Summary of Today’s Lecture Wavelet filters Bit allocation techniques in image compression –Rate-distortion functions and reverse water-filling solution Next Lecture: –Optimal transform Readings –Wavelets and JPEG2000: reference list provided in project#1 u See also Gonzalez’s book Section 7.1, (7.2-7.3), 8.5.3, 8.6.2 –“Wavelets and the lifting scheme: A 5 minute tour” by Wim Sweldens [Zeitschrift für Angewandte Mathematik und Mechanik, 76 (2), pp. 41-44, 1996] –Results on rate-distortion theory: Jain’s book Section 2.13 u Further exploration – Tutorial on rate-distortion by Ortega-Ramchandran in IEEE Sig. Proc. Magazine, Nov. 98 UMCP ENEE631 Slides (created by M.Wu © 2004)

36. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [36] More on Rate-Distortion Based Bit Allocation in Image Coding UMCP ENEE631 Slides (created by M.Wu © 2004)

37. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [37] where  k 2 -- variance of k-th coeff. v(k); n k -- # bits allocated for v(k) f(-) – quantization distortion func. Bit Allocation How many bits to be allocated for each coeff.? –Determined by the variance of coeff. –More bits for high variance  k 2 to keep total MSE small “Inverse Water-filling” (from info. theory) – –Try to keep same amount of error in each freq. band  See Jain’s pp501 for detailed bit-allocation algorithm

38. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [38] Details on Reverse Water-filling Solution (cont’d) Construct func. using Lagrange multiplier Necessary condition  Keep the same marginal gain Necessary condition for choosing

39. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [39] General Constrained Optimization Problem Define Necessary condition: “Kuhn-Tucker conditions”

40. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [40] Lagrangian Opt. for Indep. Budget Constraint Previous: fix distortion, minimize total rate Alternative: fix total rate (bit budget), minimize distortion (Discrete) Lagrangian optimization on general source –“Constant slope optimization” (e.g. in Box 6 Fig. 17 of Ortega’s tutorial) –Need to determine the quantizer q(i) for each coding unit i –Lagrangian cost for each coding unit u use a line with slope - to intersect with each operating point (Fig.14) –For a given operating quality, the minimum can be computed independently for each coding unit => Find operating quality satisfying the rate constraint

41. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [41] (from Box 6 Fig. 17 of Ortega’s tutorial) (from Fig. 14 of Ortega’s tutorial)

42. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [42] Bridging the Theory and Ad-hoc Practice Operational R-D curves –Directly achievable with practical implementation Tradeoff! Tradeoff! Tradeoff! –Rate vs. Distortion u how close are the operational points to info. theory bounds? –Also tradeoff among practical considerations u Cost of memory, computation, delay, etc.  Often narrowing down to an optimization problem: ~ optimize an objective func. subj. to a set of constraints Model mismatch problem –How good are our assumptions on source modeling? –How well does a coding algorithm tolerate model mismatch?

43. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [43] Basic Steps in R-D Optimization Determine source’s statistics –Simplified models: Gaussian, Laplacian, Generalized Gaussian –Obtain from training samples Obtain operational R-D points/functions/curves –E.g., compute distortion for each candidate quantizer Determine objective func. and constraints Search for optimal operational R-D points –Lagrange multiplier approach u select only operating points on the convex hull of overall R-D characteristics (Fig.19 of Ortega’s tutorial) u may handle different coding unit independently –Dynamic programming approach u not constrained by convex hull but has larger search space (higher complexity)

44. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [44] Ref: convex hull Formally –Smallest convex set containing all points u For R-D problem, we concerned of the lower boundary of this set that close to the info. theoretical bound (the upper/right end are open) –Convex set u if X1,..,Xn are in the set, then the linear combinations of them (with non-negative combination factors) are still in the set Informally –Use a rubber band to wrap the outermost points so that the band encloses all points

45. ENEE631 Digital Image Processing (Spring'04) Lec13 – Transf. Coding & JPEG [45] (from Fig. 19 of Ortega’s tutorial)