2. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 1Lecture 7 ECEC 453 Image Processing Architecture Lecture 8, February 5, 2004 JPEG: A Standard for all Seasons Oleh Tretiak Drexel University

3. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 2Lecture 7 Course Overview Image Compression Fundamentals  Lossless  Lossy Image Compression Standards  Image  Video Image Compression Architectures Past Present Future

4. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 3Lecture 7 Review: Decorrelation and Compression Tradeoff between distortion and bits per picture Correlation allows for greater compression More complex prediction —> greater compression

5. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 4Lecture 7 Video as Sequence of Images Previous frame predicts current frame  I(x, y, t) = I(x, y, t-1) + e(x, y, t) x x time

6. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 5Lecture 7 Predictive Coding of Video E(x, y, t) = I(x, y, t) - P(x, y, t)  I ~ image, P ~ prediction, E ~ error P(x, y, t+1) = P(x, y, t) + Code(E(x, y, t)) At receiver, I e (x, y, t) = P(x, y, t+1)  I e (x, y, t) ~ estimate of image at time t Encoder Decoder

7. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 6Lecture 7 Generic Encoder - simplified

8. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 7Lecture 7 Generic Decoder

9. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 8Lecture 7 Predicting sequential images f(t-1)f(t)f(t) f(t)–f(t–1)

10. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 9Lecture 7 Motion Compensation Macroblock size  MxN Matching criterion  MAE (mean absolute error) Search window  ±p pixel locations Search algorithm  Full search  Logarithmic search  Parallel Hierarchical One-Dimensional Search  Pixel subsampling and projection  Hierarchical downsampling

11. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 10Lecture 7 Motion Estimation Methods No compensation Full search logarithmic search 3 level hierarchical

12. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 11Lecture 7 Comparison

13. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 12Lecture 7 This Lecture

14. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 13Lecture 7 JPEG standard, AKA ISO/IEC international standard 10918-1 ITU-T Recommendation T.81 JPEG: the Joint Photographic Experts Group Goals of JPEG  General method, independent of picture source  State-of-the art (as of 12 years ago)  Allow tradeoff between compression and quality  Low computational load, allow software implementation  Both sequential and progressive  Hierarchical

15. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 14Lecture 7 JPEG at a Glance Two decorrelation technologies:  DCT (lossy)  Predictive coding (lossless) Two coder structures  Huffman  Arithmetic Four modes:  Sequential DCT  Progressive DCT  Lossless  Hierarchical

16. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 15Lecture 7 JPEG in practice Lossy DCT coding used most often Some use of sequential DCT (Web images) Lossless? Hierarchical? JPEG DCT allows many varieties of parameters  Choice of quantizer tables  Choice of Huffman codes  Arithmetic coding (ever?)  Different encoding of color and luminance There are baseline (=default) combinations of quantizers, Huffman codes, and luminance/chrominance samples

17. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 16Lecture 7 Block Diagram — JPEG DCT For color images, each component is coded separately

18. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 17Lecture 7 Block Diagram — JPEG DCT For color images, each component is coded separately

19. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 18Lecture 7 JPEG DCT Decoder The JPEG DCT coder is (almost) symmetric

20. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 19Lecture 7 JPEG DCT Decoder The JPEG DCT coder is (almost) symmetric

21. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 20Lecture 7 Features of JPEG coder Standard specifies compressed bitstream structure only  File format not part of JPEG standard Many features of JPEG compression (e. g. coding tables) are optional Standard specifies input-output characteristics, not details of algorithms  Either Huffman or arithmetic coding may be used  Huffman coding almost universal — many implementations do not include arithmetic coding Standard specifies baseline implementation. This is optional, but often followed.

22. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 21Lecture 7 DCT JPEG Major Components Color Data interleaving Chrominance subsampling Quantization Entropy coding  DC coefficients  AC coefficients

23. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 22Lecture 7 Color and JPEG Coding of color images is an essential part of JPEG Details of color coding are not part of JPEG  Many color representation  RGB  YUV  YCrCb  CIELAB  Need to represent multispcectral images A JPEG image consists of up to 255 color components  Each color component can have different numbers of samples and different numbers of bits per sample Subsampling of color components is specified by the JPEG interchange standard, not the compression standard

24. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 23Lecture 7 Data Interleaving Image components can be stored (transmitted) with or without interleaving Without interleaving, each color component is stored as a unit  First send all luminance blocks in image  Then send all Cr blocks  Finally send all Cb blocks Interleaving terminology  Data unit (DU): smallest unit of data that can be processed, 8x8 pixels for DCT  Minimum coded unit (MCU): smallest group of interleaved data For example: if three color components have the same samping factor, an MCU consists of thee data units, one of each color.

25. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 24Lecture 7 Data Interleaving with Subsampling Example: a color image with Y (intensity), Cb, Cr, (color) components is subsampled so that one color block corresponds to four Y blocks MCU 1 = Y 00 Y 01 Y 10 Y 11 Cr 00 Cb 00, MCU 2 = Y 02 Y 03 Y 12 Y 13 Cr 01 Cb 01

26. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 25Lecture 7 The DCT 8x8 DCT only (no options) For 8 bit image data, subtract 128 before performing DCT  This makes control of over/underflow easier.  DCT DC coefficient in range of ±1024  AC coefficients in range ± 512 Standard allows up to 12 bit per pixel

27. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 26Lecture 7 Quantization and Tables Quantization equation y — DCT coefficient, q — quantization step, z — coded value All DCT blocks in a component have the same quantization table Different color components can have different (or same) quantization tables Typically, luminance and color components have different quantization tables q ij = 1 leads to (near) lossless coding Image quality can be varied wit an overall quality factor q ij = QF*b ij, b ij ~ baseline quantization steps, QF ~ quality factor  QF = 1, high quality, larger QF -> lower quality, higher compression

28. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 27Lecture 7 JPEG Baseline Quantization Tables LuminanceChrominance DC CCITT rec. T.81, Tables K.1 and K.2, p 143 [147]

29. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 28Lecture 7 Entropy Coding DCT coefficients are pre-processed before entropy coding  DC coefficient is coded differently from other coefficients  Non-zero coefficients are run-length coded Either Huffman coding or arithmetic coding may be used  Arithmetic coding produces better performance  Improvement is only a few percent over run-length coding  Arithmetic coding algorithms are patented — license fees! Coding tables must be included in bit stream  Typically, baseline coding tables are used  Restrictions:  No code may be longer than 16 bits  0xFF may not be a code word

30. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 29Lecture 7 Coding DC Coefficients DC coefficients are sent with differential coding  z i = DC i - DC i-1 (difference between quantized DC coefficients in two blocks) Baseline: same coding as for lossless predictive (CCITT rec. T.82, Table K.3, page 149 [153]) Example: Difference = 195, range = 8, value = 195-128 = 67 = 1000011 2. Assume that range is coded with 11110. The difference code is then 11110 1000011 For negative differences, value is represented in 1-s complement. Difference = -195, code = 111100111100

31. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 30Lecture 7

32. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 31Lecture 7

33. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 32Lecture 7 Coding AC Coefficients AC coefficients are coded in zig-zag (called ZZ in standard) order to maximize possible runs of zeros. Code unit consist of run length followed by coefficient size. Baseline coding of size category is the same as for DC differences (Table 2.9) Example: run of 6 zeros, size = -18. In the table, -18 is in category 5. Code is (6/5, 01101). If the Huffman code for 6/5 is 1101, codeword = 110101101

34. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 33Lecture 7 Coding AC: special conditions Maximum run size is 15. A longer run is represented by 5/0 followed by remainder of run.  Example a run of 24 zeros is followed by a 7. Symbols are 15/0 9/7 After all nonzero coefficients have been processed, the special pair 0/0 is coded (EOB = end of block).

35. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 34Lecture 7 Huffman Coding - Block Diagram

36. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 35Lecture 7 JPEG Coding Example AC coding Zig-zag scan  16 3 0 -2 -21 10 0 0 0 0 0 2 2 -15 0 -3 -1 0 0... Run length codes (length/value)  0/16, 0/3, 1/-2, 0/-21, 0/10, 5/2, 0/2, 0/-15, 1/-3, 0/-1,.... DC coding Assume DC(i-1) = 39 DC(i) - DC(i-1) = 42-40 = 3 Size = 2, value =3 Assume Huffman code for size 2 is 011 DC code is 01111

37. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 36Lecture 7 JPEG Entropy Coding - more Run length codes (length/value)  0/16, 0/3, 1/-2, 0/-21, 0/10, 5/2, 0/2, 0/-15, 1/-3, 0/-1,.... Run/size/amplitude (binary)  0/5/10000, 0/2/11, 1/2/01, 0/5/01010,.... Assume Huffman code for 0/5 is 11010 (JPEG baseline table)  First code word is 1101010000 Assume Huffman code for 0/2 is 01  Second code word is 0111 Assume Huffman code for 1/2 is 11010  What is the third codeword?

38. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 37Lecture 7 Remarkable DCT Entropy Coding DC and AC coefficients treated differently  DC is coded through inter-block differences  Size/Amplitude coding (like lossless JPEG Category/Residual coding, Table 2.9) Zig-zag scan of AC, followed by run-value encoding Value converted to Size/Amplitude Combination of run length and size is Huffman-coded Unusual coding system (not off the shelf!) Responsible for much of JPEG DCT compression

39. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 38Lecture 7 Example of JPEG compression Very high quality: compression = 2.33 Photoshop Image Very low quality: compression = 115 Produced by MATLAB with Quality = 0

40. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 39Lecture 7 More about JPEG JPEG Bible  William B. Pennbaker and Joan L. Mitchell, JPEG: Still Image Data Compression Standard, Van Nostrand Reinhold, 1993  Contains ISO DIS 10918-1 International Standards Organization Draft International Standard - not available on www - available on our web site  Includes block diagrams of coding algorithms JPEG read and write capability included in MATLAB, Photoshop JPEG programs (sources, in C) are available on the web. JPEG 2000 is a new system. Includes wavelet compression