1. EE591U Wavelets and Filter Banks Copyright Xin Li 20081 Roadmap to Lossy Image Compression JPEG standard: DCT-based image coding First-generation wavelet coding FBI WSQ standard Second-generation schemes Embedded Zerotree Wavelet (EZW) A unified where-and-what perspective A classification-based interpretation Beyond wavelet coding

2. EE591U Wavelets and Filter Banks Copyright Xin Li 20082 A Tour of JPEG Coding Algorithm Flow-chart diagram of DCT-based coding algorithm specified by Joint Photographic Expert Group (JPEG) TQ C

3. EE591U Wavelets and Filter Banks Copyright Xin Li 20083 Transform Coding of Images Why not transform the whole image together? Require a large memory to store transform matrix It is not a good idea for compression due to spatially varying statistics within an image Idea of partitioning an image into blocks Each block is viewed as a smaller-image and processed independently It is not a magic, but a compromise

4. EE591U Wavelets and Filter Banks Copyright Xin Li 20084 8-by-8 DCT Basis Images

5. EE591U Wavelets and Filter Banks Copyright Xin Li 20085 Block Processing under MATLAB Type “help blkproc” to learn the usage of this function B = BLKPROC(A,[M N],FUN) processes the image A by applying the function FUN to each distinct M-by-N block of A, padding A with zeros if necessary. Example I = imread('cameraman.tif'); fun = @dct2; J = blkproc(I,[8 8],fun);

6. EE591U Wavelets and Filter Banks Copyright Xin Li 20086 Block-based DCT Example J I note that white lines are artificially added to the border of each 8-by-8 block to denote that each block is processed independently

7. EE591U Wavelets and Filter Banks Copyright Xin Li 20087 Boundary Padding padded regions Example 12 13 14 15 16 17 18 19 12 13 14 15 16 17 18 19 12 13 14 15 16 17 18 19 12 13 14 15 16 17 18 19 When the width/height of an image is not the multiple of 8, the boundary is artificially padded with repeated columns/rows to make them multiple of 8

8. EE591U Wavelets and Filter Banks Copyright Xin Li 20088 Work with a Toy Example Any 8-by-8 block in an image is processed in a similar fashion

9. EE591U Wavelets and Filter Banks Copyright Xin Li 20089 Encoding Stage I: Transform Step 1: DC level shifting 128 (DC level) _

10. EE591U Wavelets and Filter Banks Copyright Xin Li 200810 Step 2: 8-by-8 DCT Encoding Step 1: Transform (Con’t) 8  8 DCT

11. EE591U Wavelets and Filter Banks Copyright Xin Li 200811 Encoding Stage II: Quantization Q-table : specifies quantization stepsize (see slide #28) Notes:  Q-table can be specified by customer  Q-table is scaled up/down by a chosen quality factor  Quantization stepsize Q ij is dependent on the coordinates ( i,j ) within the 8-by-8 block  Quantization stepsize Q ij increases from top-left to bottom-right

12. EE591U Wavelets and Filter Banks Copyright Xin Li 200812 Encoding Stage II: Quantization (Con’t) Example f x ij s ij

13. EE591U Wavelets and Filter Banks Copyright Xin Li 200813 Encoding Stage III: Entropy Coding Zigzag Scan (20,5,-3,-1,-2,-3,1,1,-1,-1, 0,0,1,2,3,-2,1,1,0,0,0,0,0, 0,1,1,0,1,EOB) zigzag scan End Of the Block: All following coefficients are zero

14. EE591U Wavelets and Filter Banks Copyright Xin Li 200814 Run-length Coding (20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB) DC coefficient AC coefficient - DC coefficient : DPCM coding - AC coefficient : run-length coding (run, level) (5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB) (0,5),(0,-3),(0,-1),(0,-2),(0,-3),(0,1),(0,1),(0,-1),(0,-1),(2,0),(0,1), (0,2),(0,3),(0,-2),(0,1),(0,1),(6,0),(0,1),(0,1),(1,0),(0,1),EOB Huffman coding encoded bit stream

15. EE591U Wavelets and Filter Banks Copyright Xin Li 200815 JPEG Decoding Stage I: Entropy Decoding (20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB) Huffman decoding encoded bit stream AC coefficients DC coefficient DPCM decoding (0,5),(0,-3),(0,-1),(0,-2),(0,-3),(0,1),(0,1),(0,-1),(0,-1),(2,0),(0,1), (0,2),(0,3),(0,-2),(0,1),(0,1),(6,0),(0,1),(0,1),(1,0),(0,1),EOB encoded bit stream

16. EE591U Wavelets and Filter Banks Copyright Xin Li 200816 JPEG Decoding Stage II: Inverse Quantization (20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB) zigzag f -1

17. EE591U Wavelets and Filter Banks Copyright Xin Li 200817 JPEG Decoding Stage III: Inverse Transform 8  8 IDCT 128 (DC level) +

18. EE591U Wavelets and Filter Banks Copyright Xin Li 200818 Quantization Noise X X ^ MSE=|| X-X || 2 ^ Distortion calculation: Rate calculation: Rate=length of encoded bit stream/number of pixels (bps)

19. EE591U Wavelets and Filter Banks Copyright Xin Li 200819 JPEG Examples 1000 90 (58k bytes)50 (21k bytes)10 (8k bytes) best quality, lowest compression worst quality, highest compression

20. EE591U Wavelets and Filter Banks Copyright Xin Li 200820 Roadmap to Lossy Image Compression Lifting scheme: unifying prediction and transform First-generation schemes FBI WSQ standard Second-generation schemes Probabilistic modeling of wavelet coefficients Embedded Zerotree Wavelet (EZW) SPIHT coder A unified where-and-what perspective JPEG2000

21. EE591U Wavelets and Filter Banks Copyright Xin Li 200821 Early Attempts Each band is modeled by a Guassian random variable with zero mean and unknown variance (e.g., WSQ) Only modest gain over JPEG (DCT- based) is achieved Question: is this an accurate model? and how can we test it?

22. EE591U Wavelets and Filter Banks Copyright Xin Li 200822 FBI Wavelet Scalar Quantization (WSQ) k: band index mk=mk= image size subband size Each band is approximately modeled by a Gaussian r.v. Given R, minimize

23. EE591U Wavelets and Filter Banks Copyright Xin Li 200823 Rate Allocation Problem* Solution: Lagrangian Multiplier technique (turn a constrained optimization Into an unconstrained optimization problem) LL LHHH HL Given a quota of bits R, how should we allocate them to each band to minimize the overall MSE distortion?

24. EE591U Wavelets and Filter Banks Copyright Xin Li 200824 Proof by Contradiction (I) Suppose each coefficient X in a high band does observe Gaussian distribution, i.e., X~N(0,σ 2 ), then flip the sign of X (i.e., replace X with –X) should not matter and generates another element in Ω (i.e., a different but meaningful image) Assumption: our modeling target Ω is the collection of natural images Let’s test it!

25. EE591U Wavelets and Filter Banks Copyright Xin Li 200825 Proof by Contradiction (II) DWT sign flip IWT

26. EE591U Wavelets and Filter Banks Copyright Xin Li 200826 What is wrong with that? Think of two coefficients: one in smooth region and the other around edge, do they observe the same probabilistic distribution? Think of all coefficients around the same edge, do they observe the same probabilistic distribution? Ignorance of topology and geometry

27. EE591U Wavelets and Filter Banks Copyright Xin Li 200827 The Importance of Modeling Singularity Location Uncertainty Singularities carry critical visual information: edges, lines, corners … The location of singularities is important Recall locality of wavelets in spatial- frequency domain Singularities in spatial domain → significant coefficients in wavelet domain

28. EE591U Wavelets and Filter Banks Copyright Xin Li 200828 Where-and-What Coding Communication context Where The location of significant coefficients What The sign and magnitude of significant coefficients AliceBob communication channel picture

29. EE591U Wavelets and Filter Banks Copyright Xin Li 200829 Roadmap to Lossy Image Compression Lifting scheme: unifying prediction and transform First-generation schemes FBI WSQ standard Second-generation schemes Embedded Zerotree Wavelet (EZW) A unified where-and-what perspective A classification-based interpretation Scalable and ROI coding in JPEG2000

30. EE591U Wavelets and Filter Banks Copyright Xin Li 200830 1993-2003 Embedded Zerotree Wavelet (EZW)’1993 Set Partition In Hierarchical Tree (SPIHT)’1995 Space-Frequency Quantization (SFQ)’ 1996 Estimation Quantization (EQ)’1997 Embedded Block Coding with Optimal Truncation (EBCOT)’2000 Least-Square Estimation Quantization (LSEQ)’2003

31. EE591U Wavelets and Filter Banks Copyright Xin Li 200831 A Simpler Two-Stage Coding Position coding stage (where) Generate a binary map indicating the location of significant coefficients (|X|>T) Use context-based adaptive binary arithmetic coding (e.g., JBIG) to code the binary map Intensity coding stage (what) Code the sign and magnitude of significant coefficients

32. EE591U Wavelets and Filter Banks Copyright Xin Li 200832 Classification-based Modeling Insignificant class Significant class Mixture

33. EE591U Wavelets and Filter Banks Copyright Xin Li 200833 Classification Gain Without classification With classification Classification gain

34. EE591U Wavelets and Filter Banks Copyright Xin Li 200834 Example

35. EE591U Wavelets and Filter Banks Copyright Xin Li 200835 Advanced Wavelet Coding SPIHT: a simpler yet more efficient implementation of EZW coder SFQ: Rate-Distortion optimized zerotree coder EQ: Rate-Distortion optimization via backward adaptive classification EBCOT (adopted by JPEG2000): a versatile embedded coder

36. EE591U Wavelets and Filter Banks Copyright Xin Li 200836 Beyond SPIHT JPEG-decoded at rate of 0.32bpp (PSNR=32.07dB) SFG-enhanced at rate of 0.32bpp (PSNR=33.22dB) SPIHT-decoded at rate of 0.20bpp (PSNR=26.18dB) SFG-enhanced at rate of 0.20bpp (PSNR=27.33dB) Maximum-Likelihood (ML) Decoding Maximum a Posterior (MAP) Decoding

37. EE591U Wavelets and Filter Banks Copyright Xin Li 200837 Open Problems Related to Image Coding Coding of specific class of images (e.g., Satellite, microarray, fingerprint) Coding of color-filter-array (CFA) images Error resilient coding of images Perceptual image coding Image coding for pattern recognition

38. EE591U Wavelets and Filter Banks Copyright Xin Li 200838 Coding of Specific Class of Images How to design specific coding algorithms for each class?

39. EE591U Wavelets and Filter Banks Copyright Xin Li 200839 CFA Image Coding Bayer Pattern CFA Interpolation (demosaicing) Color image compression CFA Interpolation (demosaicing) CFA data compression Approach I Approach II Which one is better and why?

40. EE591U Wavelets and Filter Banks Copyright Xin Li 200840 Error Resilient Image Coding source encoder channel source decoder sourcedestination super-channel channel encoder channel decoder How can we optimize the end-to-end performance in the presence of channel errors?

41. EE591U Wavelets and Filter Banks Copyright Xin Li 200841 Perceptual Image Coding Characterizing image distortion is difficult! How do we objectively define mage quality which has to be subject to individual opinions?

42. EE591U Wavelets and Filter Banks Copyright Xin Li 200842 Image Coding for PR image sensor Communication channel Pattern recognition How does coding distortion affect the recognition performance? We need to develop a new image representation which Can simultaneously support low-level (e.g., compression, denoising) and high-level (e.g., recognition and retrieval) vision tasks