1. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 1Lecture 8 ECEC 453 Image Processing Architecture Lecture 8, 2/13/2004 JPEG Modes of Operation, Implementation Issues, Interchange, Extensions Oleh Tretiak Drexel University

2. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 2Lecture 8 Review: Block Diagram — JPEG DCT DCTQuantizer Quantization Tables Coding Tables Coder Headers Tables Data For color images, each component is coded separately Encoding

3. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 3Lecture 8 Review: JPEG DCT Decoder IDCT Inverse Quantizer Decoder Headers Tables Data Quantization Tables Coding Tables The JPEG DCT coder is (almost) symmetric Decoding

4. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 4Lecture 8 Review: Major Components Color Data interleaving Chrominance subsampling Quantization Entropy coding  DC coefficients  AC coefficients

5. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 5Lecture 8 About Color In television (and Physics) a color image specified by giving 3 intensity images  Usual description are CIE (Commission Internationale de L’Eclairage) spectral primary R, G, B components  Other signals, e.g., to generate an image on a monitor, to print on an inkjet printer, are related to RGB components by simultaneous equations  There are several standard coordinate systems. Below are the equations to transform from CIE RGB to CIE XYZ. The signal Y has special importance: it gives the intensity of the image.

6. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 6Lecture 8 More Color Lore These equations must be applied to every pixel in the image (many multiplications and additions!) Some commonly used color coordinates (see Appendix A)  Y, Cr, Cb — most common system for still and video image compression  Y, I, Q — used in NTSC color television systems (USA)  Y, U, V — PAL color television system (Europe)  L*, a*, b* — CIELAB, standard for color fax

7. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 7Lecture 8 Review: Data Interleaving with Subsampling Example: a color image with Y (intensity), Cb, Cr, (color) components is subsampled so that one color block corresponds to four Y blocks MCU 1 = Y 00 Y 01 Y 10 Y 11 Cr 00 Cb 00, MCU 2 = Y 02 Y 03 Y 12 Y 13 Cr 01 Cb 01

8. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 8Lecture 8 Implications of Subsampling Original Color Image: 3 x 256 x 256 ~ 196 Kbytes Subsample chrominance: 256 x 256 + 2 x 128 x 128 ~ 98 Kbytes Luminance has 32 x 32 = 1024 DCT blocks Each chrominance component has 16 x 16 = 256 DCT blocks There are 4 luminance blocks for each pair of chrominance block Fewer blocks —> less computing! LuminanceChrominance

9. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 9Lecture 8 Review: Quantization Tables Quantization equation y — DCT coefficient, q — quantization step, z — coded value All DCT blocks in a component have the same quantization table Typically, luminance and color components have different quantization tables q ij = 1 leads to (near) lossless coding Image quality can be varied wit an overall quality factor q ij = QFxb ij, b ij ~ baseline quantization steps, QF ~ quality factor  QF = 1, high quality, larger QF -> lower quality, higher compression

10. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 10Lecture 8 Quantization and Compression Very high quality: compression = 2.33 PhotoShop Image Very low quality: compression = 115 Produced by MATLAB with Quality = 0

11. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 11Lecture 8 Review: Entropy Coding DCT coefficients are pre-processed before entropy coding  DC coefficient is coded differently from other coefficients  Non-zero coefficients are run-length coded DC coefficients are preprocessed: inter-block differences Huffman coding commonly used  Different tables for DC and AC  Different tables for Chrominance and Luminance Coding tables may be included in bit stream

12. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 12Lecture 8 Coding AC Coefficients AC coefficients are coded in zig-zag order to maximize possible runs of zeros. Code unit consist of run length followed by coefficient size. Baseline coding of size is the same as for DC differences (Table 2.9) Example: run of 6 zeros, size = -18. In the table, -18 is in category 5. Code is (6/5, 01101). If the Huffman code for 6/5 is 1101, codeword = 110101101

13. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 13Lecture 8 Huffman Coding - Block Diagram

14. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 14Lecture 8 This Lecture JPEG modes of operation Implementation Issues File Formats JPEG extensions JPEG2000

15. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 15Lecture 8 Four JPEG Modes of Operation

16. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 16Lecture 8 Progressive Modes SpectralSuccessive approximation

17. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 17Lecture 8 JPEG Hierarchical Coding

18. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 18Lecture 8 JPEG Implementation Issues Hardware vs. Software IDCT Complexity Arithmetic Precision Required JPEG coding Tables Color Conversion and Display Considerations JPEG compressed file format

19. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 19Lecture 8 Hardware vs. Software Implementation Example  640x480 DCT, color image -> 23 MOP (mega operations)  Color conversion ~ 40%  DCT ~ 45%  Coding ~ 15% Software  Less than a second on microcomputers  High-efficiency, scaled DCT & IDCT  Winograd & Feig method Hardware  Color conversion, subsampling, DCT are well adapted to special-purpose computers  Regular dataflow DCT (Chen, separable) better adapted to parallel hardware

20. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 20Lecture 8 IDCT, Precision IDCT has different details, but same general characteristics as DCT Many zero coefficients - can be used to reduce complexity  Smooth areas -> DC only  Low detail areas -> 4x4 IDCT Precision — DCT coefficients have fractional values!  Keep integer only - near lossless!  Dynamic range: 8x8 DCT -> x 8 expansion  Example: 8 bit [0-255] input range  Subtract 128 -> range [-128, 127]  After DCT -> range [-1024, 1023] worst case  Rule of thumb: 12 bit accuracy for DCT constants, 16 bits for intermediate calculations gives ‘exact’ answers

21. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 21Lecture 8 JPEG Coding and Q Tables Optimal performance is obtained with tables adapted to image Options  Baseline quantization & coding  Baseline quantization, custom coding  Custom quantization, baseline coding  Custom quantization & coding Big difference only for low rate (web?) images At low rate, coding has bigger effect than quantization

22. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 22Lecture 8 Color Conversion and Display Color can be represented in may ways  RGB, Y Cb Cr, YUV, CIELAB Color is usually coded at half resolution Encoding  Color convert, if necessary  Downsample chrominance  ~ 40% of computational load Decoding  Upsample color  Convert to display coordinates Simple algorithm structure (parallelizable)

23. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 23Lecture 8 JPEG Compressed File Format JPEG standard specifies bitstream  Compressed image = SOI, frame, EOI  SOI = FFD8, EOI = FFD9  frame = [tables/misc.], frame header, { scan, [DNL segment]}*  [] denotes optional component, {}* means that this can be repeated  frame header contains information about mode of operation, bits/pixel, picture dimensions, number of components, etc  scan = [tables/misc.], scan header, {ECS, [RST]}*, ECS  ECS ~ entropy coded segment, RST ~ restart marker for error recovery  ECS = {MCU}* Color coordinates? Image physical (print) dimensions? Ancillary information (comments) are not specified.

24. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 24Lecture 8 File Formats JFIF (JPEG file interchange format): De-facto JPEG file standard  Developed by Eric Hamilton at C-Cube Microsystems  ftp://ftp.uu.net/graphics/jpeg/ Courtesy of UUNET Inc.  jfif.pdf (our web site, notes) SPIFF: Official JPEG file interchange format (still picture interchange file format)  Includes following compression methods  None, JPEG, JBIG, Modified Huffman, MR, MMR  Backward compatible (mostly?) with JFIF  ITU standard (I can’t find it on the web) FlashPix: Potential industry standard. Hierarchical, using JPEG EXIF, DCF are standards for digitial cameras. They are based on JPEG and lossless TIFF

25. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 25Lecture 8 How MATLAB reads JPEG * RJPGC.MEX * * This is a mex interface to the Independent Jpeg Group's (IJG) * LIBJPEG libaray. This can read RGB and grayscale JPEG images. * * * The IJG code is available at: * http://www.imagefolio.com/users/docs/pro31/imagemagick.shtml

26. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 26Lecture 8 About JFIF  APP0 = FFE0 marker used to specify extensions  Specify Y, Cb, Cr for components  Includes ‘thumbnail’ picture for previewing  Thumbnail can be RGB, index color, or compressed  Image orientations: first scan line is on top  Spatial relation of components specified  Pixel aspect ratio (horizontal and vertical density) are specified

27. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 27Lecture 8 JFIF Sample File written by PhotoShop [cbis 210] % od -c -x Mallorca | more 0000000 377 330 377 340 \0 020 J F I F \0 001 002 001 \0 d ffd8 ffe0 0010 4a46 4946 0001 0201 0064 SOI, start of image 0000440 \0 \0 \0 031 h t t p : / / w w w. p 0000 0019 6874 7470 3a2f 2f77 7777 2e70 0000460 h o t o d i s c. c o m / \0 8 B URL of image copyright owner? 0001200 F i l e w r i t t e n b y 4669 6c65 2077 7269 7474 656e 2062 7920 0001220 A d o b e P h o t o s h o p 250 4164 6f62 6520 5068 6f74 6f73 686f 70a8 APP0 markerVersion

28. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 28Lecture 8 JPEG Extensions Part 3 of JPEG standard - ‘appendix’  Variable Quantization  Selective Refinement  Tiling  Color Facsimile Standard SPIFF is part of this extension

29. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 29Lecture 8 Variable Quantization JPEG does not allow changes of quantization within scan Variable quantization (depending on block content) is known to be useful  For example, graphics and image are optimized with different coding Extension allows per-block quality factor that multiplies quantization matrix

30. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 30Lecture 8 Selective Refinement Some parts of image can be coded in greater detail than others Three types  Hierarchical  Some parts have more hierarchy levels (higher resolution)  Progressive  Some parts have additional DCT coefficients  Component  Some parts have fewer components. Good for coding images that are part gray scale and part color

31. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 31Lecture 8 Tiling What can be done with huge images?  JPEG limits image to 65,535 pixels on side Solution: divide huge image into tiles  Allows random access into images Tiling options  Simple tiling: all tiles have the same size  Pyramidal tiling: multiple resolutions may be included  Composite tiling: tiles can be of different resolution and can overlap

32. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 32Lecture 8 Color Facsimile Standard Much awaited extension to original fax standard Input is color image scanned at 200 dpi Specifies color transformation and coding  CIELAB color coding  Chrominance components subsampled by a factor of 2  Image coded with JPEG (lossy or lossless) Optional features allow higher spatial resolution, no color subsampling, higher precision

33. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 33Lecture 8

34. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 34Lecture 8 From University of Patras (Greece). Avaialble from www.dsp.toronto.edu/~dsp/JPEG2000/

35. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 35Lecture 8

36. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 36Lecture 8 Compression = 64 JPEG JPEG2000

37. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 37Lecture 8 Compression = 32 JPEGJPEG2000

38. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 38Lecture 8 JPEG2000 Progress Kakadu software:  http://www.kakadusoftware.com/ http://www.kakadusoftware.com/  Apple's Quicktime v6 (for MAC only right now); Yahoo's latest Messanger [sic] application (uses JPEG2000 for live video MicroImages’ TNT products for Geospatial imagery (see http://www.microimages.com for more info) http://www.microimages.com JAVA sources  http://jj2000.epfl.ch (Ecole Polytechnique Federale Lausanne) http://jj2000.epfl.ch

39. Image Processing Architecture, © 2001-2004 Oleh TretiakPage 39Lecture 8 JPEG: Summary Standard came at the right time  Internet  Digital publishing  Digital cameras Complexity of JPEG has been overcome by computer power 90/10 rule  90 % of applications use 10 % of JPEG