1. EE465: Introduction to Digital Image Processing1 Lossy Image Compression: a Quick Tour of JPEG Coding Standard Why lossy for grayscale images?  Tradeoff between Rate and Distortion Transform basics  Unitary transform and properties Quantization basics  Uniform Quantization and 6dB/bit Rule JPEG=T+Q+C  T: DCT, Q: Uniform Quantization, C: Run-length and Huffman coding

2. EE465: Introduction to Digital Image Processing2 Why Lossy? In most applications related to consumer electronics, lossless compression is not necessary  What we care is the subjective quality of the decoded image, not those intensity values With the relaxation, it is possible to achieve a higher compression ratio (CR)  For photographic images, CR is usually below 2 for lossless, but can reach over 10 for lossy

3. EE465: Introduction to Digital Image Processing3 A Simple Experiment Bit-plane representation A=a 0 +a 1 2+a 2 2 2 + … … +a 7 2 7 Least Significant Bit (LSB) Most Significant Bit (MSB) Example A=129  a 0 a 1 a 2 …a 7 =10000001 a 0 a 1 a 2 …a 7 =00110011  A=4+8+64+128=204

4. EE465: Introduction to Digital Image Processing4 A Simple Experiment (Con ’ t) How will the reduction of gray-level resolution affect the image quality?  Test 1: make all pixels even numbers (i.e., knock down a 0 to be zero)  Test 2: make all pixels multiples of 4 (i.e., knock down a 0,a 1 to be zeros)  Test 3: make all pixels multiples of 4 (i.e., knock down a 0,a 1,a 2 to be zeros)

5. EE465: Introduction to Digital Image Processing5 Experiment Results original Test 1 Test 2 Test 3

6. EE465: Introduction to Digital Image Processing6 How to Measure Image Quality? Subjective  Evaluated by human observers  Do not require the original copy as a reference  Reliable, accurate yet impractical Objective  Easy to operate (automatic)  Often requires the original copy as the reference (measures fidelity rather than quality)  Works better if taking HVS model into account

7. EE465: Introduction to Digital Image Processing7 Mean Square Error (MSE) Peak Signal-to-Noise-Ratio (PSNR) Objective Quality Measures originaldecoded Question: Can you think of a counter-example to prove objective measure is not consistent with subjective evaluation?

8. EE465: Introduction to Digital Image Processing8 Answer Shifted (MSE=337.8)Original cameraman image By shifting the last row of the image to become the first row, we affect little on subjective quality but the measured MSE is large

9. EE465: Introduction to Digital Image Processing9 Overview of JPEG Lossy Compression Flow-chart diagram of DCT-based coding algorithm specified by Joint Photographic Expert Group (JPEG) TQ C

10. EE465: Introduction to Digital Image Processing10 Lossy Image Compression: a Quick Tour of JPEG Coding Standard Why lossy for grayscale images?  Tradeoff between Rate and Distortion Transform basics  Unitary transform and properties Quantization basics  Uniform Quantization and 6dB/bit Rule JPEG=T+Q+C  T: DCT, Q: Uniform Quantization, C: Run-length and Huffman coding

11. EE465: Introduction to Digital Image Processing11 An Example of 1D Transform with Two Variables x1x1 x2x2 y1y1 y2y2 Transform matrix (1,1) (1.414,0)

12. EE465: Introduction to Digital Image Processing12 forward transform Generalization into N Variables basis vectors (column vectors of transform matrix)

13. EE465: Introduction to Digital Image Processing13 Decorrelating Property of Transform x1x1 x2x2 y1y1 y2y2 x 1 and x 2 are highly correlated p(x 1 x 2 )  p(x 1 )p(x 2 ) y 1 and y 2 are less correlated p(y 1 y 2 )  p(y 1 )p(y 2 )

14. EE465: Introduction to Digital Image Processing14 Transform=Change of Coordinates Intuitively speaking, transform plays the role of facilitating the source modeling  Due to the decorrelating property of transform, it is easier to model transform coefficients (Y) instead of pixel values (X) An appropriate choice of transform (transform matrix A) depends on the source statistics P(X)  We will only consider the class of transforms corresponding to unitary matrices

15. EE465: Introduction to Digital Image Processing15 A matrix A is called unitary if A -1 =A* T Unitary Matrix and 1D Unitary Transform Definition conjugate transpose When the transform matrix A is unitary, the defined transform is called unitary transform Example For a real matrix A, it is unitary if A -1 =A T

16. EE465: Introduction to Digital Image Processing16 Inverse of Unitary Transform For a unitary transform, its inverse is defined by Inverse Transform basis vectors corresponding to inverse transform

17. EE465: Introduction to Digital Image Processing17 Properties of Unitary Transform Energy compaction: only few transform coefficients have large magnitude  Such property is related to the decorrelating role of unitary transform Energy conservation: unitary transform preserves the 2-norm of input vectors  Such property essentially comes from the fact that rotating coordinates does not affect Euclidean distance

18. EE465: Introduction to Digital Image Processing18 Energy Compaction Example Hadamard matrix insignificant significant

19. EE465: Introduction to Digital Image Processing19 Energy Conservation* A is unitary Proof

20. EE465: Introduction to Digital Image Processing20 Numerical Example Check:

21. EE465: Introduction to Digital Image Processing21 Implication of Energy Conservation A is unitary Q TT -1

22. EE465: Introduction to Digital Image Processing22 Summary of 1D Unitary Transform Unitary matrix: A -1 = A *T Unitary transform: A unitary Properties of 1D unitary transform  Energy compaction: most of transform coefficients y i are small  Energy conservation: quantization can be directly performed to transform coefficients

23. EE465: Introduction to Digital Image Processing23 Do N 1D transforms in parallel From 1D to 2D

24. EE465: Introduction to Digital Image Processing24 2D forward transform 1D column transform1D row transform Definition of 2D Transform

25. EE465: Introduction to Digital Image Processing25 2D Transform=Two Sequential 1D Transforms Conclusion:  2D separable transform can be decomposed into two sequential  The ordering of 1D transforms does not matter row transform column transform row transform column transform (left matrix multiplication first) (right matrix multiplication first)

26. EE465: Introduction to Digital Image Processing26 Energy Compaction Property of 2D Unitary Transform  Example insignificant coefficients (th=64) A coefficient is called significant if its magnitude is above a pre-selected threshold th

27. EE465: Introduction to Digital Image Processing27 Energy Conservation Property of 2D Unitary Transform Example: 2-norm of a matrix X A unitary You are asked to prove such property in your homework

28. EE465: Introduction to Digital Image Processing28 Implication of Energy Conservation Q TT -1 Similar to 1D case, quantization noise in the transform domain has the same energy as that in the spatial domain

29. EE465: Introduction to Digital Image Processing29 Important 2D Unitary Transforms Discrete Fourier Transform  Widely used in non-coding applications (frequency-domain approaches) Discrete Cosine Transform  Used in JPEG standard Hadamard Transform  All entries are  1  N=2: Haar Transform (simplest wavelet transform for multi-resolution analysis)

30. EE465: Introduction to Digital Image Processing30 Discrete Cosine Transform

31. EE465: Introduction to Digital Image Processing31 forward transforminverse transform 1D DCT  Real and orthogonal  fast implementation available: O(Nlog 2 N)  excellent energy compaction property  DCT is NOT the real part of DFT Fact The real and imaginary parts of DFT are generally not orthogonal matrices Properties

32. EE465: Introduction to Digital Image Processing32 2D DCT Notice the excellent energy compaction property of DCT Original cameraman image X Its DCT coefficients Y (2451 significant coefficients, th=64)

33. EE465: Introduction to Digital Image Processing33 Summary of 2D Unitary Transform Unitary matrix: A -1 =A *T Unitary transform: A unitary Properties of 2D unitary transform  Energy compaction: most of transform coefficients y i are small  Energy conservation: quantization can be directly performed to transform coefficients

34. EE465: Introduction to Digital Image Processing34 Lossy Image Compression: a Quick Tour of JPEG Coding Standard Why lossy for grayscale images?  Tradeoff between Rate and Distortion Transform basics  Unitary transform and properties Quantization basics  Uniform Quantization and 6dB/bit Rule JPEG=T+Q+C  T: DCT, Q: Uniform Quantization, C: Run-length and Huffman coding

35. EE465: Introduction to Digital Image Processing35 What is Quantization? In Physics  To limit the possible values of a magnitude or quantity to a discrete set of values by quantum mechanical rules In image compression  To limit the possible values of a pixel value or a transform coefficient to a discrete set of values by information theoretic rules

36. EE465: Introduction to Digital Image Processing36 Examples Unlike entropy, we encounter it everyday (so it is not a monster) Continuous to discrete  a quarter of milk, two gallons of gas, normal temperature is 98.6F, my height is 5 foot 9 inches Discrete to discrete  Round your tax return to integers  The mileage of my car is about 55K.

37. EE465: Introduction to Digital Image Processing37 A Game Played with Bits Precision is finite: the more precise, the more bits you need (to resolve the uncertainty)  Keep a card in secret and ask your partner to guess. He/she can only ask Yes/No questions: is it bigger than 7? Is it less than 4?... However, not every bit has the same impact  How much did you pay for your car? (two thousands vs. $2016.78)

38. EE465: Introduction to Digital Image Processing38 Scalar vs. Vector Quantization We only consider the scalar quantization (SQ) in this course  Even for a sequence of values, we will process (quantize) each sample independently Vector quantization (VQ) is the extension of SQ into high-dimensional space  Widely used in speech compression, but not for image compression

39. EE465: Introduction to Digital Image Processing39 Definition of (Scalar) Quantization x f f -1 x ^ sSsS original value quantization index quantized value Quantizer x Q x ^ f finds the closest (in terms of Euclidean distance) approximation of x from a codebook C (a collection of codewords) and assign its index to s; f -1 operates like a table look-up to return the corresponding codeword

40. EE465: Introduction to Digital Image Processing40 Numerical Example (I) 0 1 C={-1,0,1} Three codewords S={1,2,3} 123 red: codewords green: index in C x x ^ 0.5 -0.5 1 xRxR

41. EE465: Introduction to Digital Image Processing41 Numerical Example (II) 24 408 C={8,24,40,56,72,…,248} sixteen codewords S={1,2,3,…,16} 123 red: codewords green: index in C x x ^... 248 16 x  {0,1,…,255} 1632 24 40 (note that the scales are different) 8 floor operation

42. EE465: Introduction to Digital Image Processing42 Numerical Example (III) 225 222 235 228 220 206 209 44 49 56 64 42 128 106 94 27 Q x ^x 232 216 232 232 216 200 216 40 56 56 72 40 136 104 88 24  For scalar quantization, each sample is quantized independently Notes  Quantization is irreversible

43. EE465: Introduction to Digital Image Processing43 Uniform Quantization for Uniform Distribution Uniform Quantization A scalar quantization is called uniform quantization (UQ) if all its codewords are uniformly distributed (equally-distanced) 24 408... 248 Example (quantization stepsize  =16)  Uniform Distribution -A-AA 1/2A x f(x) denoted by U[-A,A]

44. EE465: Introduction to Digital Image Processing44 Quantization Noise of UQ  -  /2  /2 1/  e f(e) Quantization noise of UQ on uniform distribution is also uniformly distributed Recall Variance of U[-  /2,  /2] is -A-A A

45. EE465: Introduction to Digital Image Processing45 6dB/bit Rule Signal: X ~ U[-A,A] Noise: e ~ U[-  /2,  /2] Choose N=2 n (n-bit) codewords for X N=2 n (quantization stepsize)

46. EE465: Introduction to Digital Image Processing46 Lossy Image Compression: a Quick Tour of JPEG Coding Standard Why lossy for grayscale images?  Tradeoff between Rate and Distortion Transform basics  Unitary transform and properties Quantization basics  Uniform Quantization and 6dB/bit Rule JPEG=T+Q+C  T: DCT, Q: Uniform Quantization, C: Run-length and Huffman coding

47. EE465: Introduction to Digital Image Processing47 JPEG Coding Algorithm Overview

48. EE465: Introduction to Digital Image Processing48 Transform Coding of Images Why not transform the whole image together?  Require a large memory to store transform matrix  It is not a good idea for compression due to spatially varying statistics within an image Idea of partitioning an image into blocks  Each block is viewed as a smaller-image and processed independently  It is not a magic, but a compromise

49. EE465: Introduction to Digital Image Processing49 Block-based DCT Example J I note that white lines are artificially added to the border of each 8-by-8 block to denote that each block is processed independently

50. EE465: Introduction to Digital Image Processing50 Boundary Padding padded regions Example 12 13 14 15 16 17 18 19 12 13 14 15 16 17 18 19 12 13 14 15 16 17 18 19 12 13 14 15 16 17 18 19 When the width/height of an image is not the multiple of 8, the boundary is artificially padded with repeated columns/rows to make them multiple of 8

51. EE465: Introduction to Digital Image Processing51 Work with a Toy Example Any 8-by-8 block in an image is processed in a similar fashion

52. EE465: Introduction to Digital Image Processing52 Encoding Stage I: Transform Step 1: DC level shifting 128 (DC level) _

53. EE465: Introduction to Digital Image Processing53 Step 2: 8-by-8 DCT Encoding Step 1: Transform (Con’t) 8  8 DCT

54. EE465: Introduction to Digital Image Processing54 Encoding Stage II: Quantization Q-table : specifies quantization stepsize (see slide #28) Notes:  Q-table can be specified by customer  Q-table is scaled up/down by a chosen quality factor  Quantization stepsize Q ij is dependent on the coordinates ( i,j ) within the 8-by-8 block  Quantization stepsize Q ij increases from top-left to bottom-right

55. EE465: Introduction to Digital Image Processing55 Encoding Stage II: Quantization (Con’t) Example f x ij s ij

56. EE465: Introduction to Digital Image Processing56 Encoding Stage III: Entropy Coding Zigzag Scan (20,5,-3,-1,-2,-3,1,1,-1,-1, 0,0,1,2,3,-2,1,1,0,0,0,0,0, 0,1,1,0,1,EOB) zigzag scan End Of the Block: All following coefficients are zero

57. EE465: Introduction to Digital Image Processing57 Run-length Coding (20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB) DC coefficient AC coefficient - DC coefficient : DPCM coding - AC coefficient : run-length coding (run, level) (5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB) (0,5),(0,-3),(0,-1),(0,-2),(0,-3),(0,1),(0,1),(0,-1),(0,-1),(2,0),(0,1), (0,2),(0,3),(0,-2),(0,1),(0,1),(6,0),(0,1),(0,1),(1,0),(0,1),EOB Huffman coding encoded bit stream

58. EE465: Introduction to Digital Image Processing58 JPEG Decoding Stage I: Entropy Decoding (20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB) Huffman decoding encoded bit stream AC coefficients DC coefficient DPCM decoding (0,5),(0,-3),(0,-1),(0,-2),(0,-3),(0,1),(0,1),(0,-1),(0,-1),(2,0),(0,1), (0,2),(0,3),(0,-2),(0,1),(0,1),(6,0),(0,1),(0,1),(1,0),(0,1),EOB encoded bit stream

59. EE465: Introduction to Digital Image Processing59 JPEG Decoding Stage II: Inverse Quantization (20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB) zigzag f -1

60. EE465: Introduction to Digital Image Processing60 JPEG Decoding Stage III: Inverse Transform 8  8 IDCT 128 (DC level) +

61. EE465: Introduction to Digital Image Processing61 Quantization Noise X X ^ MSE=|| X-X || 2 ^ Distortion calculation: Rate calculation: Rate=length of encoded bit stream/number of pixels (bps)

62. EE465: Introduction to Digital Image Processing62 JPEG Examples 1000 90 (58k bytes)50 (21k bytes)10 (8k bytes) best quality, lowest compression worst quality, highest compression

63. EE465: Introduction to Digital Image Processing63 Wavelet vs. DCT JPEG (CR=64)JPEG2000 (CR=64) discrete cosine transform basedwavelet transform based

64. EE465: Introduction to Digital Image Processing64 Image Compression Summary JPEG=T+Q+C (so is JPEG2000 except T becomes wavelet transform instead of DCT) Key idea: sparsity of transform coefficients (most transform coefficients are quantized to zero) Main weakness: blocky artifacts at low bit rates (many deblocking algorithms have been proposed in the literature)

65. EE465: Introduction to Digital Image Processing65 One-Minute Survey How much do you think you have understood today’s lecture?  A) >90%, B)70-90%, C)50-70%, D) <50% How long have you spent on final project so far? What is the best score you have reached so far? (please indicate your status: G/UG)