1 Source Coding and Compression Dr.-Ing. Khaled Shawky Hassan Room: C3-217, ext: 1204, Lecture 4 (Week 2)

2 Modeling & Coding Developing compression algorithms ● Phase I: Modeling – Develop the means to extract redundancy information – Redundancy → Predictability ● Phase II: Coding – Binary representation of the – The representation depends on the “Modeling”

3 Modeling Example 1 Let us consider this arbitrary sequence: S n = 9, 11, 11, 11, 14, 13, 15, 17, 16, 17, 20, 21 Binary encoding requires 5 – bits/sample; WHY ? Now, let us consider the model: Ŝ n = n + 8: Thus: Ŝ n = 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 Can be decoded as: e n = S n -Ŝ n : 0, 1, 0, -1, 1, -1, 0, 1, -1, -1, 1, 1 How many bits are required now? Coding: ‘00’ -1, ‘01’ 0, ‘10’ 1 2 bits/sample Therefore, the model also needs to be designed and encoded in the algorithm

4 Modeling: Yet another example Let us consider this arbitrary sequence: S n = Assume it correctly describes the probabilities generated by the source; then P(1) = P(6) = P(7) = p(10) = 1/16 P(2) = P(3) = P(4) = P(5) = P(8) = P(9) = 2/16 Assuming the sequence is independent and identically distributed (i.i.d.) Then: However, if we found somehow a sort of correlation, then: Thus, instead of coding the samples, code the difference only: i.e.,: Now, P(1) = 13/16, P(-1) = 3/16 Then: H = 0.70 bits (per symbol)

5 Stochastic sources  A source outputs symbols X 1, X 2,...  Each symbol take its value from an alphabet A = (a 1, a 2, …).  Model: P(X 1,…,X N ) assumed to be known for all combinations. Example 1: A text is a sequence of symbols each taking its value from the alphabet A = (a, …, z, A, …, Z, 1, 2, …9, !, ?, …). Example 2: A (digitized) grayscale image is a sequence of symbols each taking its value from the alphabet A = (0,1) or A = (0, …, 255). Source X 1, X 2, …

6 Two Special Cases 1.The Memoryless Source  Each symbol independent of the previous ones.  P(X 1, X 2, …, X n ) = P(X 1 ) ¢ P(X 2 ) ¢ … ¢ P(X n ) 2.The Markov Source  Each symbol depends on the previous one.  P(X 1, X 2, …, X n ) = P(X 1 ) ¢ P(X 2 |X 1 ) ¢ P(X 3 |X 2 ) ¢ … ¢ P(X n |X n-1 )

7 Markov Models ●Assume that each output symbol depends on previous k ones. Formally: ●Let { x n } be a sequence of observations ●We call { x n } a k th -order discrete Markov chain (DMC) if ●Usually, we use a first order DMC (the knowledge of 1 symbol in the past is enough) ●The State of the Process

8 Non-Linear Markov Models ●Consider a BW image as a string of black & white pixels (e.g. row-by-row) ●Define two states: S b & S w for the current pixel ●Define probabilities: ●P(S b ) = prob of being in S b ●P(S w ) = prob of being in S w ●Transition probabilities ●P(b|b), P(w|b) ●P(b|w), P(w|w)

9 Markov Models Example ●Assume P(Sw) = 30/31 P(Sb) = 1/31 P(w/w) = 0.99 P(b/w) = 0.01 P(b/b) = 0.7 P(w/b) = 0.3 ●For the Markov Model H(Sb) = -0.3log(0.3) – 0.7log(0.7) = H(Sw) = 0.01log(0.01) – 0.99log(0.99) = H Markov = 30/31* /31*0.881 = ●What is iid different in ??

10 Markov Models in Text Compression ●In written English, probability of next letter is heavily influenced by previous ones ➢ E.g. “u” after “q” ●Shannon's work: ➢ He used 2nd-order MM, 26 letters + space,, found H = 3.1 bits/letter ➢ Word-based model, H=2.4bits/letter ➢ Human prediction based on 100 previous letters, he found the limits 0.6 ≤H ≤1.3 bits/letter ●Longer context => better prediction

11 Composite Source Model ●In many applications, it is not easy to use a single model to describe the source. In such cases, we can define a composite source, which can be viewed as a combination or composition of several sources, with only one source being active at any given time. E.g.: an executable contains: For very complicated resources (text, images, …) ●Solution: composite model, different sources/each with different model work in sequence :

12 Let Us Decorate This: Knowing something about the source it self can help us to ‘reduce’ the entropy This is called; Entropy Encoding Note the we cannot actually reduce the entropy of the source, as long as our coding is lossless Strictly speaking, we are only reducing our estimate of the entropy

13 Examples: Modeling!

14 ● Proposed by Dr. David A. Huffman in 1952 ➢ “A Method for the Construction of Minimum Redundancy Codes” ● Applicable to many forms of data transmission ➢ Mostly used example: text files Huffman Coding

15 Huffman Coding Algorithm: – The procedure to build Huffman codes – Extended Huffman Codes (new) Adaptive Huffman Coding (new) – Update Procedure – Decoding Procedure Huffman Coding: What We Will Discuss

16 Shannon-Fano Coding ● The second code based on Shannon’s theory ✗ It is a suboptimal code (it took a graduate student (Huffman) to fix it!) ● Algorithm: ➢ Start with empty codes ➢ Compute frequency statistics for all symbols ➢ Order the symbols in the set by frequency ➢ Split the set (almost half-half!) to minimize*difference ➢ Add ‘0’ to the codes in the first set and ‘1’ to the rest ➢ Recursively assign the rest of the code bits for the two subsets, until sets cannot be split.

17 Shannon-Fano Coding ● Example: ● Assume a sequence: A={a,b,c,d,e,f} with the following occurrence weights, {9, 8, 6, 5, 4, 2}, respectively ● Apply Shannon-Fano Coding and discuss the sub- optimality

18 Shannon-Fano Coding

19 Shannon-Fano Coding

20 Shannon-Fano Coding

21 Shannon-Fano Coding

22 Shannon-Fano Coding

23 Shannon-Fano Coding ef

24 Shannon-Fano Coding ● Shannon-Fano does not always produce optimal prefix codes; the ordering is performed only once at the beginning!! ● Huffman coding is almost as computationally simple and produces prefix codes that always achieve the lowest expected code word length, under the constraints that each symbol is represented by a code formed of an integral number of bits ● Sometimes prefix-free codes are called, Huffman codes ● Symbol-by-symbol Huffman coding (the easiest one) is only optimal if the probabilities of these symbols are independent and are some power of a half, i.e. (½) n

25 ● Proposed by Dr. David A. Huffman in 1952 ➢ “A Method for the Construction of Minimum Redundancy Codes” ● Applicable to many forms of data transmission ➢ Our example: text files ● In general, Huffman coding is a form of statistical coding as not all characters occur with the same frequency! Huffman Coding

26 ● Why Huffman coding (likewise all entropy coding): – Code word lengths are no longer fixed like ASCII. – Code word lengths vary and will be shorter for the more frequently used characters, i.e., overall shorter average code length! Huffman Coding: The Same Idea

27 1. Scan the text to be compressed and compute the occurrence of all characters. 2. Sort or prioritize characters based on number of occurrences in text (from low-to-high). 3. Build Huffman code tree based on prioritized list. 4. Perform a traversal of tree to determine all code words. 5. Scan text again and create (encode) the characters in a new coded file using the Huffman codes. Huffman Coding: The Algorithm

28 ● Consider the following short text: – Eerie eyes seen near lake. ● Count up the occurrences of all characters in the text – E e r i e e y e s s e e n n e a r l a k e. Huffman Coding: Building The Tree E e r i space y s n a r l k.Example

29 Eerie eyes seen near lake. What is the frequency of each character in the text? Huffman Coding: Building The Tree Example Char Freq. Char Freq. Char Freq. E 1 y1 k1 e 8 s 2.1 r 2 n 2 i 1 a2 space 4 l1

30 Huffman Coding: Building The Tree 1. Create binary tree nodes with character and frequency of each character 2. Place nodes in a priority queue “??” The lower the occurrence, the higher the priority in the queue

31 Huffman Coding: Building The Tree second bonus assignment: Construct the Huffman tree as follows!! Uses binary tree nodes (OOP-like View; second bonus assignment: Construct the Huffman tree as follows!!) public class HuffNode { public char myChar; public int myFrequency; public HuffNode, myLeft, myRight; } priorityQueue myQueue;

32 Huffman Coding: Building The Tree  The queue after inserting all nodes  Null Pointers are not shown E1E1 i1i1 y1y1 l1l1 k1k1.1.1 r2r2 s2s2 n2n2 a2a2 sp 4 e8e8

33 Huffman Coding: Building The Tree  While priority queue contains two or more nodes – Create new node – Dequeue node and make it left sub-tree – Dequeue next node and make it right sub-tree – Frequency of new node equals sum of frequency of left and right children – Enqueue new node back into queue in the right order!!

34 Huffman Coding: Building The Tree E1E1 i1i1 y1y1 l1l1 k1k1.1.1 r2r2 s2s2 n2n2 a2a2 sp 4 e8e8

35 Huffman Coding: Building The Tree y1y1 l1l1 k1k1.1.1 r2r2 s2s2 n2n2 a2a2 sp 4 e8e8 2 E1E1 i1i1

36 Huffman Coding: Building The Tree k1k1.1.1 r2r2 s2s2 n2n2 a2a2 sp 4 e8e8 E1E1 i1i1 2 y1y1 l1l

CS 307 E1E1 i1i1 k1k1.1.1 r2r2 s2s2 n2n2 a2a2 sp 4 e8e8 2 y1y1 l1l1 2 Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 r2r2 s2s2 n2n2 a2a2 sp 4 e8e8 2 y1y1 l1l1 2 k1k Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 r2r2 s2s2 n2n2 a2a2 sp 4 e8e8 2 y1y1 l1l1 2 k1k Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 n2n2 a2a2 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 n2n2 a2a2 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a2 4 Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a2 4 Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a2 4 4 Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a2 4 4 Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a What is happening to the characters with a low number of occurrences? Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a Huffman Coding: Building The Tree

CS 307 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a Huffman Coding: Building The Tree After enqueueing this node there is only one node left in priority queue.

CS 307 Dequeue the single node left in the queue. This tree contains the new code words for each character. Frequency of root node should equal number of characters in text. E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a Huffman Coding: Building The Tree Eerie eyes seen near lake. ----> 26 characters

CS 307  Perform a traversal of the tree to obtain new code words  Going left is a 0 going right is a 1  code word is only completed when a leaf node is reached E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a Encoding the File Traverse Tree

Huffman Coding (contd.) CharCode E0000 i0001 y0010 l0011 k space 011 e10 r1100 s1101 n1110 a1111 E1E1 i1i1 sp 4 e8e8 2 y1y1 l1l1 2 k1k r2r2 s2s2 4 n2n2 a2a

Huffman Coding: Example (2) Symbols: {a,b,c,d,e,} Weights: {0.2, 0.4, 0.2,0.1, 0.1} Required: Maximum Variance Tree! b, 0.4 c, 0.2 a, 0.2 de, b, 0.4 c, dea, 0.4 b, deac, 0.6 deacb, 1.0 b, 0.4 c, 0.2 a, 0.2 e, 0.1 d,

Example: Maximum Variance Tree Average code length l = 0.4x x x x x4 = 2.2 bits/symbol 60

Huffman Coding: Example (3) Symbols: {a,b,c,d,e,} Weights: {0.2, 0.4, 0.2,0.1, 0.1} Required: Minimum Variance Tree! b, 0.4 c, 0.2 a, 0.2 de, b, 0.4 de, ac, deb, 0.6 deacb, 1.0 b, 0.4 c, 0.2 a, 0.2 e, 0.1 d,

Example: Minimum Variance Tree Average code length l = 0.4x2 + ( )x3+ ( )*2 = 2.2 bits/symbol 62 b ed c a

Example: Yet Another Tree Average code length l = 0.4x1 + ( )x3 = 2.2 bits/symbol 63

Example Summary Average code length  l = 0.4x x x x x4 = 2.2 bits/symbol Entropy  H = Σ s=a..e P(s) log 2 P(s) = bits/symbol Redundancy  l – H = bits/symbol 64

Min Variance Huffman Trees Huffman codes are not unique All versions yield the same average length Which one should we choose? The one with the minimum variance in codeword lengths i.e., with the minimum height tree Why? It will ensure the least amount of variability in the encoded stream 65

Another Example! Consider the source: A = {a, b, c}, P(a) = 0.8, P(b) = 0.02, P(c) = 0.18 H = bits/symbol Huffman code: a → 0 b → 11 c → 10 l = 1.2 bits/symbol Redundancy = bits/symbol (on average)(47%!) Q: Could we do better? 66

Extended Huffman Codes Example 1: Consider encoding sequences of two letters Redundancy = bits/symbol l = ( 0.64x x x x x x x x x8)/2 /2 = /2 bits/symbol 67

Extended Huffman Codes (Remarks) The idea can be extended further Consider all possible n m sequences (we did 3 2 ) In theory, by considering more sequences we can improve the coding !! (is it applicable ? ) In reality, the exponential growth of the alphabet makes this impractical E.g., for length 3 ASCII seq.: = 2 24 = 16M Practical consideration: most sequences would have zero frequency → Other methods are needed (Adaptive Huffman Coding) 68