1. EE465: Introduction to Digital Image Processing Binary Image Compression 4 The art of modeling image source –Image pixels are NOT independent events 4 Run length coding of binary and graphic images –Applications: BMP, TIF/TIFF 4 Lempel-Ziv coding* –How does the idea of building a dictionary can achieve data compression?

2. EE465: Introduction to Digital Image Processing From Theory to Practice 4 So far, we have discussed the problem of data compression under the assumption that the source distribution is known (i.e., the probabilities of all possible events are given). 4 In practice, the source distribution (probability models) is unknown and can only be approximately obtained by relative frequency counting method due to the inherent dependency among source data.

3. EE465: Introduction to Digital Image Processing An Image Example Binary image sized 100  100 (approximately 1000 dark pixels)

4. EE465: Introduction to Digital Image Processing A Bad Model 4 Each pixel in the image is observed as an independent event 4 All pixels can be characterized by a single discrete random variable – binary Bernoulli source 4 We can estimate the probabilities by relative frequency counting

5. EE465: Introduction to Digital Image Processing Binary Bernoulli distribution (P(X=black)=0.1) Synthesized Image by the Bad Model

6. EE465: Introduction to Digital Image Processing Why does It Fail? 4 Roughly speaking –Pixels in an image are not independent events (source is not memoryless but spatially correlated) –Pixels in an image do not observe the same probability model (source is not stationary but spatially varying) 4 Fundamentally speaking –Pixels are the projection of meaningful objects in the real world (e.g., characters, lady, flowers, cameraman, etc.) –Our sensory processing system has learned to understand meaningful patterns in sensory input through evolution

7. EE465: Introduction to Digital Image Processing Similar Examples in 1D 4 Scenario I: think of a paragraph of English texts. Each alphabet is NOT independent due to semantic structures. For example, the probability of seeing a “u” is typically small; however, if a “q” appears, the probability of the next alphabet being “u” is large. 4 Scenario II: think of a piece of human speech. It consists of silent and voiced segments. The silent segment can be modeled by white Gaussian noise; while the voiced segment can not (e.g., pitches)

8. EE465: Introduction to Digital Image Processing Data Compression in Practice source modeling entropy coding discrete source X binary bit stream probability estimation P(Y) Y The art of data compression is the art of source modeling Probabilities can be estimated by counting relative frequencies either online or offline

9. EE465: Introduction to Digital Image Processing Source Modeling Techniques 4 Transformation –transform the source into an equivalent yet more convenient representation 4 Prediction –Predict the future based on the causal past 4 Pattern matching –Identify and represent repeated patterns

10. EE465: Introduction to Digital Image Processing Non-image Examples 4 Transformation in audio coding (MP3) –Audio samples are transformed into frequency domain 4 Prediction in speech coding (CELP) –Human vocal tract can be approximated by an auto-regressive model 4 Pattern matching in text compression –Search repeated patterns in texts

11. EE465: Introduction to Digital Image Processing How to Build a Good Model 4 Study the source characteristics –The origin of data: computer-generated, recorded, scanned … –It is about linguistics, physics, material science and so on … 4 Choose or invent the appropriate tool (modeling technique) Q: why pattern matching is suitable for texts, but not for speech or audio?

12. EE465: Introduction to Digital Image Processing Image Modeling 4 Binary images –Scanned documents (e.g., FAX) or computer- generated (e.g., line drawing in WORD) 4 Graphic images –Windows icons, web banners, cartoons 4 Photographic images –Acquired by digital cameras 4 Others: fingerprints, CT images, astronomical images …

13. EE465: Introduction to Digital Image Processing Lossless Image Compression 4 No information loss – i.e., the decoded image is mathematically identical to the original image –For some sensitive data such as document or medical images, information loss is simply unbearable –For others such as photographic images, we only care about the subjective quality of decoded images (not the fidelity to the original)

14. EE465: Introduction to Digital Image Processing Binary Image Compression 4 The art of modeling image source –Image pixels are NOT independent events 4 Run length coding of binary and graphic images –Applications: BMP, TIF/TIFF 4 Lempel-Ziv coding* –How does the idea of building a dictionary can achieve data compression?

15. EE465: Introduction to Digital Image Processing Run Length Coding (RLC) What is run length? Run length is defined as the length of consecutively identical symbols HHHHH T HHHHHHH 571 Examples SSSS EEEE NNNN WWWW 4444 Coin-flip random walk

16. EE465: Introduction to Digital Image Processing Run Length Coding (Con’t) Transformation by run-length counting Entropy coding discrete source X binary bit stream probability estimation P(Y) Y Y is the sequence of run-lengths from which X can be recovered losslessly

17. EE465: Introduction to Digital Image Processing Properties -“0” run-length (red) and “1” run-length (green) alternates - run-lengths are positive integers RLC of 1D Binary Source 0000 111 000000 11111 00 463 X Y 52 (need extra 1 bit to denote what the starting symbol is) Huffman coding compressed data

18. EE465: Introduction to Digital Image Processing 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 … 57408run-length When P(x=0) is close to 1, we can record run-length of dominant symbol (“0”) only Example Properties - all coded symbols are “0” run-lengths - run-length is a nonnegative integer Variation of 1D Binary RLC

19. EE465: Introduction to Digital Image Processing Modeling Run-length geometric source: P(X=k)=(1/2) k, k=1,2,… Run-length kProbability 12345…12345… 1/2 1/4 1/8 1/16 1/32 …

20. EE465: Introduction to Digital Image Processing Golomb Codes k12345678…k12345678… codeword 0 10 110 1110 11110 111110 1111110 11111110 … Optimal VLC for geometric source: P(X=k)=(1/2) k, k=1,2,… 0 1 10 10 10 …

21. EE465: Introduction to Digital Image Processing From 1D to 2D white run-length black run-length Question: what distinguishes 2D from 1D coding? Answer: inter-line dependency of run-lengths

22. EE465: Introduction to Digital Image Processing Relative Address Coding (RAC)* 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 0 0 0 0 0 1 1 0 0 previous line current line 7 d 1 =1d 2 =-2NS,run=2 NS – New Start Its variation was adopted by CCITT for Fax transmission

23. EE465: Introduction to Digital Image Processing Image Example CCITT test image No. 1 Size: 1728  2376 513216 bytes Raw data (1bps) filesize of ccitt1.pbm: filesize of ccitt1.tif: 37588 bytes Compression Ratio=13.65

24. EE465: Introduction to Digital Image Processing Graphic Images (Cartoons) Observations: -dominant background color (e.g., white) -objects only contain a few other colors Total 12 different colors (black,white,red,green,blue,yellow …)

25. EE465: Introduction to Digital Image Processing Palette-based Image Representation index color 0 1 2 3 4 5 … white black red green blue yellow … Any (R,G,B) 24-bit color can be repre- sented by its index in the palette.

26. EE465: Introduction to Digital Image Processing 0 0 0 0 0 3 3 3 0 0 0 0 0 0 0 2 2 2 2 1 0 0 0 0 0 0 0 0 … 53748run-length Example 03020 color-index RLC of 1D M-ary Source Basic idea Record not only run-lengths but also color indexes 1 1 (color, run) representation:(0,5) (3,3) (0,7) (2,4) (1,1) (0,8) … WWWWGGGWWWWWWWRRRR… color sequence Note: run-length is a positive integer

27. EE465: Introduction to Digital Image Processing Variation of 1D M-ary RLC When P(x=0) is close to 1, we can record run-length of dominant symbol (“0”) only 0 0 0 0 0 1 0 0 0 0 0 0 0 4 0 0 0 0 3 2 0 0 0 0 0 0 0 0 1 … 57408run Example Properties - “0” run-lengths only - run-length is a nonnegative integer level14321 (run, level) representation: (5,1) (7,4) (4,3) (0,2) (8,1) …

28. EE465: Introduction to Digital Image Processing Image Example Raw data: party8.ppm, 526  286, 451308 bytes Compressed: party8.bmp, 17094 bytes Compression ratio=26.4

29. EE465: Introduction to Digital Image Processing Binary Image Compression 4 The art of modeling image source –Image pixels are NOT independent events 4 Run length coding of binary and graphic images –Applications: BMP, TIF/TIFF 4 Lempel-Ziv coding* –How does the idea of building a dictionary can achieve data compression?

30. EE465: Introduction to Digital Image Processing History of Lempel-Ziv Coding 4 Invented by Lempel-Ziv in 1977 4 Numerous variations and improvements since then 4 Widely used in different applications –Unix system: compress command –Winzip software (LZW algorithm) –TIF/TIFF image format –Dial-up modem (to speed up the transmission)

31. EE465: Introduction to Digital Image Processing Dictionary-based Coding 4 Use a dictionary –Think about the evolution of an English dictionary It is structured - if any random combination of alphabets formed a word, the dictionary would not exist It is dynamic - more and more words are put into the dictionary as time moves on –Data compression is similar in the sense that redundancy reveals as patterns, just like English words in a dictionary

32. EE465: Introduction to Digital Image Processing Toy Example I took a walk in town one day And met a cat along the way. What do you think that cat did say? Meow, Meow, Meow I took a walk in town one day And met a pig along the way. What do you think that pig did say? Oink, Oink, Oink I took a walk in town one day And met a cow along the way. What do you think that cow did say? Moo, Moo, Moo - from “Wee Sing for Baby” I took a walk in town one day 1 entrypattern 2 3 4 5 And met a What do you think that did say? along the way 6 7 cat meow ……

33. EE465: Introduction to Digital Image Processing Basic Ideas 4 Build up the dictionary on-the-fly (based on causal past such that decoder can duplicate the process) 4 Achieve the goal of compression by replacing a repeated string by a reference to an earlier occurrence 4 Unlike VLC (fixed-to-variable), LZ parsing goes the other way (variable-to-fixed)

34. EE465: Introduction to Digital Image Processing Lempel-Ziv Parsing 4 Initialization: –D={all single-length symbols} –L is the smallest integer such that all codewords whose length is smaller than L have appeared in D (L=1 at the beginning) 4 Iterations: w  next parsed block of L symbols –Rule A: If w  D, then represent w by its entry in D and update D by adding a new entry with w concatenated with its next input symbol –Rule B: If w  D, then represent the first L-1 symbols in w by its entry in D and update D by adding a new entry with w

35. EE465: Introduction to Digital Image Processing Example of Parsing a Binary Stream 0 1 1 0 0 1 1 1 0 1 1 … Dictionary entrypattern 1 2 3 4 5 6 0 1 01 11 10 00 7 011 1, 2, 2, 1, 3, 4, 7, … 8 110 (entries in D) fixed -length variable -length output: input: L=1 L=2 L=3 w: 0, 11, 10, 00, 01, 110, 011, … A AABBBB rule: Illustration: Step 1: w=0, Rule A, output 1, add 01 to D, L=2 Step 2: w=11, Rule B, output 2, add 11 to D Step 3: w=10, Rule B, output 2, add 10 to D Step 4: w=00, Rule B, output 1, add 00 to D Step 5: w=01, Rule A, output 3, add 011 to D, L=3 Step 6: w=110, Rule B, output 4, add 110 to D

36. EE465: Introduction to Digital Image Processing Binary Image Compression Summary 4 Theoretical aspect –Shannon’s source entropy formula tells us the lower bound for coding a memoryless discrete source –To achieve the source entropy, we need to use variable length codes (i.e., long codeword assigned to small-probability event and vice versa) –Huffman’s algorithm (generating the optimal prefix codes) offers a systematic solution 4 Practical aspect –Data in the real world contains memory (dependency, correlation …) –We don’t know the probability distribution function of the source –Tricky point: the goal of image compression (processing) is not to make your algorithm work for one image, but for a class of images

37. EE465: Introduction to Digital Image Processing The Art of Source Modeling 4 How do I know a model matches or not? Hypothesis Model: Bernoulli distribution (P(X=black)=0.1) Observation Data

38. EE465: Introduction to Digital Image Processing A Matched Example Synthesis Model

39. EE465: Introduction to Digital Image Processing Transformation by run-length counting Huffman coding discrete source X binary bit stream probability estimation P(Y) Y Y is the sequence of run-lengths from which X can be recovered losslessly Good Models for Binary Images Transformation by Lempel-Ziv parsing Y is the sequence of dictionary entries from which X can be recovered losslessly

40. EE465: Introduction to Digital Image Processing Remaining Questions 4 How to choose different compression techniques, say RLC vs. LZC? –Look at the source statistics –For instance, compare binary image (good for RLC) vs. English texts (good for LZC) 4 What is the best binary image compression technique? –It is called JBIG2 – an enhanced version of JBIG 4 What is the entropy of binary image source? –We don’t know and the question itself is questionable –A more relevant question would be: what is the entropy for a probabilistic model which we believe generates the source?