1. 1 Source Coding and Compression Dr.-Ing. Khaled Shawky Hassan Room: C3-222, ext: 1204, Email: khaled.shawky@guc.edu.eg Lecture 5

2. Static vs. Adaptive Coding Encoder 1. Initialize the data model based on a first pass over the data (i.e., perform the probabilities analysis) 2. Transmit the data model (encoder). 3. Send data and while there is more data to send: -- Encode the next symbol using the existing data model and send it.Decoder 1. Receive the data model (decoder). 2. Receive the data and while there is more data to receive -- Decode the next symbol using the data model and output it. Summary about the Two-Pass procedure: 1. Collect statistics, generate codewords(1 st pass round) 2. Perform actual encoding/compression(2 nd pass round) 3. Not practical in many situations (e.g., compressing network transmissions) Static (Two-Pass Model) 2

3. Static vs. Adaptive Coding Encoder 1. Initialize the data model as fixed probability and fixed code length. 2. Send data first and while there is more data to send a.Encode the next symbol using the data model (if we have) and send it. b.Modify the existing data model based on the last symbol.Decoder 1. Initialize the data model as per agreement. 2. While there is more data to receive a.Decode the next symbol using the data model and output it. b.Modify the data model based on the decoded symbol. Adaptive (One-Pass Model) 3 What Do We Find ? No Encoder map to send!

4. Huffman Coding (e.g.: Lossless JPEG) Properties: I.Huffman codes are built from the bottom up, starting with the leaves of the tree and working progressively closer to the root II.Huffman coding will always at least work more efficient than Shannon-Fano coding, so it has become the predominate entropy coding method III.It was shown that Huffman coding cannot be improved or with any other integral bit- width coding stream Sibling Property: Defined by Gallager [Gallager 1978]: “A binary code tree has the sibling property if each node (except the root) has a sibling and if the nodes can be listed in order of nonincreasing (decreasing) weight with each node adjacent to its sibling.” Thus: 1- If A is the parent node of B (left) and C (right) is a child of B, then W(A) > W(B) > W(C) Thus if A is the parent node of B (left) and C (right), then W(B) < W(C) 4

5. Huffman Coding Properties A binary tree is a Huffman tree if and only if it obeys the sibling property, i.e., W(#1) ≤ W(#3) ≤ W(#3) ≤ … ≤ W(#7) ≤ W(#8) ≤ W(#9) 5 Non-Decreasing Order

6. Huffman Tree Sibling Property 6 ✔ Adaptive Huffman tree is found by adjusting the Huffman tree on the fly, based on data previously seen and having no knowledge about future statistics ✔ Sibling property during the update assure that we have a Huffman tree with right weights W(#1) ≤ W(#3) ≤ W(#3) ≤ … ≤ W(#7) ≤ W(#8) ≤ W(#9)

7. Huffman Tree Sibling Property 7 ✔ Sibling property during the update shows that we don't have a right Huffman tree W(#1) ≤ W(#3) ≤ W(#3) ≤ … ≤ W(#7) ≤ W(#8) ≤ W(#9) Dismissed Order

8. Huffman Tree Sibling Property 8 Sibling property during the update assure that we have a Huffman tree with right weights

9. Huffman Tree Sibling Property 9 ✔ Adaptive Huffman tree is found by adjusting the Huffman tree on the fly, based on data previously seen and having no knowledge about future statistics ✔ Sibling property during the update assure that we have a Huffman tree with right weights Dismissed Order W(#1) ≤... ≤ W(#3) ≤ W(#4) > W(#5) ≤ W(#6) ≤ W(#7) > W(#8) ≤ W(#9) Now node W(#4) > W(#5) and W(#7) > W(#8), i.e., violate the sibling property!

10. Adaptive Huffman Coding Algorithm: o Given: alphabet S = {s 1, …, s n } (NO Probabilities !!!) block/quadratic code o Pick a fixed default binary codes for all symbols (block/quadratic code) Empty o Start with an empty “Huffman” tree (I said and I mean it – Empty ) o Read symbol s from source If NYT(s) % (//) Not Yet Transmitted Send NYT, default(s) (except for the first symbol) Update the tree (and keep it Huffman) Else codeword Send codeword for s Update tree o Repeat until done with all symbols in the source 10

11. Example (Adaptive Huffman) Assume we are encoding the message [a a r d v a r k] The total number of nodes in this tree will be (at most) 2*n – 1 + 2 = 2*26 -1 +2 = 53 where n is the number of usable alphabets and +2 is only for the “NYT” and its “node” The first letter to be transmitted is “a” As a does not yet exist in the tree, we send a binary code 00000 for a and then add a to the tree The NYT node gives birth to a new NYT node and a terminal node corresponding to “a” In this example, we will consider only 51 nodes and leaves (instead of 53!!). However, the correct is 53. The weight of the terminal node will be higher than the NYT node, so we assign the number 49 to the NYT node and 50 to the terminal node “a” The next symbol is a, and the transmitted code is 1 now (as a = 1 only now!) Lest see an example … (we first starts with a fixed code!) 11

12. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT0 a r d v k Output: 12 To keep the rest of the slides as is, we started as the book with 51; however, the correct thing is to start with 53!

13. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT0 a r d v k Output: 00000 13

14. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT0 a1 r d v k Output: 00000 14 1 1

15. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT0 a1 r d v k Output: 000001 15

16. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT0 a1 r01 d v k Output: 000001010001 16

17. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT000 a1 r01 d001 v k Output:0000010100010000011 17

18. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT0000 a1 r01 d001 v0001?? k Output:0000010100010000011000 18

19. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT0000 a1 r01 d001 v0001?? k Output:0000010100010000011000 19

20. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT0000 a1 r01 d001 v0101 ?? k Output:0000010100010000011000 20

21. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT000 a1 r01 d001 v?? k Output:0000010100010000011000 21

22. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT1100 a0 r10 d111 v1101 k Output:000001010001000001100010101 22

23. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT1100 a0 r10 d111 v1101 k Output:0000010100010000011000101010 23

24. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT1100 a0 r10 d111 v1101 k Output:000001010001000001100010101010 24

25. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT11000? a0 r10 d111 v1101 k11001?? Output:0000010100010000011000101010 10110001010 25

26. Example: Adaptive Huffman Coding Input: aardvark SymbolCode NYT11000? a0 r10 d111 v1101 k11001 ?? Output:0000010100010000011000101010 10110001010 26

27. Example: Adaptive Huffman Coding k Input: aardvark SymbolCode NYT11100 a0 r10 d110 v1111 k11101 Output:0000010100010000011000101010 10110001010 27

28. Adaptive Huffman Decoding Input: a SymbolCode NYT a r d v k Output:0000010100010000011000101010 10110001010 28 1 1

29. Input: aa SymbolCode NYT0 a1 r d v k Output:0000010100010000011000101010 10110001010 Adaptive Huffman Decoding 29

30. Input: aar SymbolCode NYT0 a1 r01 d v k Output:0000010100010000011000101010 10110001010 Adaptive Huffman Decoding 30

31. Input: aard SymbolCode NYT000 a1 r01 d001 v k Output:0000010100010000011000101010 10110001010 Adaptive Huffman Decoding 31

32. Input: aardv SymbolCode NYT0000 ? a1 r01 d001 v0001?? k Output:0000010100010000011000101010 10110001010 Adaptive Huffman Decoding 32

33. Input: aardv SymbolCode NYT0000 a1 r01 d001 v0001?? k Output:0000010100010000011000101010 10110001010 Adaptive Huffman Decoding 33

34. Input: aardv SymbolCode NYT0000 a1 r01 d001 v0101 ?? k Output:0000010100010000011000101010 10110001010 Adaptive Huffman Decoding 34

35. Input: aardv SymbolCode NYT000 a1 r01 d001 v?? k Output:0000010100010000011000101010 10110001010 Adaptive Huffman Decoding 35

36. Input: aardv SymbolCode NYT1100 a0 r10 d111 v1101 k Output:0000010100010000011000101010 10110001010 Adaptive Huffman Decoding 36

37. Input: aardva SymbolCode NYT1100 a0 r10 d111 v1101 k Output:0000010100010000011000101010 10110001010 Adaptive Huffman Decoding 37

38. Input: aardvar SymbolCode NYT1100 a0 r10 d111 v1101 k Output:0000010100010000011000101010 10110001010 Adaptive Huffman Decoding 38

39. Input: aardvar SymbolCode NYT1100 a0 r10 d111 v1101 k Output:0000010100010000011000101010 10110001010 Adaptive Huffman Decoding 39

40. Input: aardvark SymbolCode NYT11000 a0 r10 d111 v1101 k?? Output:0000010100010000011000101010 10110001010 Adaptive Huffman Decoding 40

41. k Input: aardvark SymbolCode NYT11100 a0 r10 d110 v1111 k11101 Output:0000010100010000011000101010 10110001010 ? Adaptive Huffman Decoding 41

42. Find the adaptive Huffman encoder (compressor) for the following text: raaaabcbaacvkl Assuming 26 alphabet set! Adaptive Huffman Exercise 42 Try to solve the following!

43. If the source has an alphabet {a 1,a 2, …, a m } of size m, then pick e and r such that m = 2 e +r and 0 ≤ r <2 e. The letter a k is encoded as the ﴾e+1﴿-bit corresponds to k−1, iff 1≤ k ≤2r; else, a k is encoded as (only) the e-bit binary representation of k−r−1. Example: suppose m = 26, then e = 4, and r=10. Then symbol a1 is encoded as 00000, (“a” in English) the symbol a2 is encoded as 00001, (“b” in English) and the symbol a22 is encoded as 1011 (“b” in English) Adaptive Huffman Notes 43 To Follow the Text Book example:

44. Adaptive Huffman Applications Lossless Image Compression Steps to have lossless image compression: 1.Generate a Huffman code for each uncompressed image (but already quantized and compressed with lossy methods) 2.Encode the image using the Huffman code 3.Save it in a file again !!! The original (uncompressed) image representation uses 8 bits/pixel. The image consists of 256 rows of 256 pixels, so the uncompressed representation uses 65,536 bytes. Compression ratio → number of bytes (uncompressed): number of bytes compressed 44

45. Adaptive Huffman Applications Lossless Image Compression 45

46. Adaptive Huffman Applications Lossless Image Compression Image NameBits/PixelTotal Size (B)Compression Ratio Sena7.0157,5041.14 Sensin7.4961,4301.07 Earth4.9440,5341.62 Omaha7.1258,3741.12 Huffman (Lossless JPEG) Compression Based on Pixel value 46

47. Adaptive Huffman Applications Lossless Image Compression Image NameBits/PixelTotal Size (B)Compression Ratio Sena4.0232,9681.99 Sensin4.7038,5411.70 Earth4.1333,8801.93 Omaha6.4252,6431.24 Huffman Compression Based on Pixel Difference value and Two-Pass Model 47

48. Adaptive Huffman Applications Lossless Image Compression Image NameBits/PixelTotal Size (B)Compression Ratio Sena3.9332,2612.03 Sensin4.6337,8961.73 Earth4.8239,5041.66 Omaha6.3952,3211.25 Huffman Compression Based on Pixel Difference Value and One-Pass Adaptive Model 48

49. Adaptive Huffman Applications Lossless Image Compression Image NameBits/PixelTotal Size (B)Compression Ratio Sena3.9332,2612.03 Sensin4.6337,8961.73 Earth4.8239,5041.66 Omaha6.3952,3211.25 Huffman Compression Based on Pixel Difference Value and One-Pass Adaptive Model 49

50. Optimality of Huffman Codes! The necessary conditions for an optimal variable-length binary code: 50 Condition 1: Condition 1: Given any two letters a j and a k, if P(a j ) ≥ P(a k ), then l j ≤ l k, where l j is the number of bits in the codeword for a j. Condition 2: Condition 2: The two least probable letters have codewords with the same maximum length lm. Condition 3: Condition 3: In the tree corresponding to the optimum code, there must be two branches stemming from each intermediate node. Condition 4: Condition 4: Suppose we change an intermediate node into a leaf node by combining all the leaves descending from it into a composite word of a reduced alphabet. Then, if the original tree was optimal for the original alphabet, the reduced tree is optimal for the reduced alphabet.

51. Minimum Variance Huffman Codes By performing the sorting procedure in a slightly different manner, we could have found a different Huffman code. 51

52. Huffman Coding: Self Study! 3.2.1 Minimum Variance Huffman Codes (pp. 46 – 47 {redo the examples}) 52 3.2.3 Length of Huffman Codes (pp. 49 ~ 51 and the example 3.2.2) 3.2.3 Huffman Codes optimality condition!!

53. Huffman Tree Sibling Property 53 W(#1) ≤... ≤ W(#3) ≤ W(#4) > W(#5) ≤ W(#6) ?? W(#7) > W(#8) ≤ W(#9) ➔ Now, sweep nodes #4 and #5

54. Huffman Tree Sibling Property 54 W(#1) ≤... ≤ W(#3) ≤ W(#4) ≤ W(#5) ≤ W(#6) > …. W(#8) ≤ W(#9) ➔ But W(#6) > W(#8)! Then sweep #6 and #8 #8 E(10) #6 A(13) #4 C(2) #5(15) #7(19) #9(29) Looks better?! #1 D(2) #2 B(2) #3(4)

55. Huffman Tree Sibling Property 55 W(#1) ≤ W(#3) ≤ W(#3) ≤ … ≤ W(#7) ≤ W(#8) ≤ W(#9) #8 E(10) #8 A(13) #4 C(2) #5(15) #7(19) #9(29) NOW it is ok! #1 D(2) #2 B(2) #3(4)