Applied Algorithmics - week7 Huffman Coding David A. Huffman (1951) Huffman coding uses frequencies of symbols in a string to build a variable rate prefix code Each symbol is mapped to a binary string More frequent symbols have shorter codes No code is a prefix of another Example: D C B A 1 A 0 B 100 C 101 D 11 05/07/2018 Applied Algorithmics - week7

Applied Algorithmics - week7 Variable Rate Codes Example: 1) A  00; B  01; C  10; D  11; 2) A  0; B  100; C  101; D  11; Two different encodings of AABDDCAA 0000011111100000 (16 bits) 00100111110100 (14 bits) 05/07/2018 Applied Algorithmics - week7

Applied Algorithmics - week7 Cost of Huffman Trees Let A={a1, a2, .., am} be the alphabet in which each symbol ai has probability pi We can define the cost of the Huffman tree HT as C(HT)= pi·ri, where ri is the length of the path from the root to ai The cost C(HT) is the expected length (in bits) of a code word represented by the tree HT. The value of C(HT) is called the bit rate of the code. m i=1 05/07/2018 Applied Algorithmics - week7

Cost of Huffman Trees - example Let a1=A, p1=1/2; a2=B, p2=1/8; a3=C, p3=1/8; a4=D, p4=1/4 where r1=1, r2=3, r3=3, and r4=2 HT D C B A 1 C(HT) =1·1/2 +3·1/8 +3·1/8 +2·1/4=1.75 05/07/2018 Applied Algorithmics - week7

Applied Algorithmics - week7 Huffman Tree Property Input: Given probabilities p1, p2, .., pm for symbols a1, a2, .., am from alphabet A Output: A tree that minimizes the average number of bits (bit rate) to code a symbol from A I.e., the goal is to minimize function: C(HT)= pi·ri, where ri is the length of the path from the root to leaf ai. This is called a Huffman tree or Huffman code for alphabet A 05/07/2018 Applied Algorithmics - week7

Applied Algorithmics - week7 Huffman Tree Property Input: Given probabilities p1, p2, .., pm for symbols a1, a2, .., am from alphabet A Output: A tree that minimizes the average number of bits (bit rate) to code a symbol from A I.e., the goal is to minimize function: C(HT)= pi·ri, where ri is the length of the path from the root to leaf ai. This is called a Huffman tree or Huffman code for alphabet A 05/07/2018 Applied Algorithmics - week7

Construction of Huffman Trees Form a (tree) node for each symbol ai with weight pi Insert all nodes to a priority queue PQ (e.g., a heap) ordered by nodes probabilities while (the priority queue has more than two nodes) min1  remove-min(PQ); min2  remove-min(PQ); create a new (tree) node T; T.weight  min1.weight + min2.weight; T.left  min1; T.right  min2; insert(PQ, T) return (last node in PQ) 05/07/2018 Applied Algorithmics - week7

Construction of Huffman Trees P(A)= 0.4, P(B)= 0.1, P(C)= 0.3, P(D)= 0.1, P(E)= 0.1 0.1 0.1 0.1 C 0.3 0.4 D E B A 0.2 0.1 0.3 0.4 B C A 1 D E 05/07/2018 Applied Algorithmics - week7

Construction of Huffman Trees 0.1 0.2 0.3 0.4 B C A 1 D E 0.3 0.3 0.4 C A 1 B 1 D E 05/07/2018 Applied Algorithmics - week7

Construction of Huffman Trees 0.3 0.3 0.4 0.6 0.4 C A A 1 1 B C 1 1 D E B 1 D E 05/07/2018 Applied Algorithmics - week7

Construction of Huffman Trees 0.4 0.6 A 1 1 A C 1 1 B C 1 1 B D E 1 D E 05/07/2018 Applied Algorithmics - week7

Construction of Huffman Trees 1 A = 0 A B = 100 1 C = 11 C D = 1010 1 E = 1011 B 1 D E 05/07/2018 Applied Algorithmics - week7

Applied Algorithmics - week7 Huffman Codes Theorem: For any source S the Huffman code can be computed efficiently in time O(n·log n) , where n is the size of the source S. Proof: The time complexity of Huffman coding algorithm is dominated by the use of priority queues One can also prove that Huffman coding creates the most efficient set of prefix codes for a given text It is also one of the most efficient entropy coder 05/07/2018 Applied Algorithmics - week7

Basics of Information Theory The entropy of an information source (string) S built over alphabet A={a1, a2, .., am}is defined as: H(S) = ∑ i pi·log2(1/pi) where pi is the probability that symbol ai in S will occur log2(1/pi) indicates the amount of information contained in ai, i.e., the number of bits needed to code ai. For example, in an image with uniform distribution of gray-level intensity, i.e. all pi = 1/256, then the number of bits needed to encode each gray level is 8 bits. The entropy of this image is 8. 05/07/2018 Applied Algorithmics - week7

Huffman Code vs. Entropy P(A)= 0.4, P(B)= 0.1, P(C)= 0.3, P(D)= 0.1, P(E)= 0.1 Entropy: 0.4 · log2(10/4) + 0.1 · log2(10) + 0.3 · log2(10/3) + 0.1 · log2(10) + 0.1 · log2(10) = 2.05 bits per symbol Huffman Code: 0.4 · 1 + 0.1 · 3 + 0.3 · 2 + 0.1 · 4 + 0.1 · 4 = 2.10 Not bad, not bad at all. 05/07/2018 Applied Algorithmics - week7

Lempel-Ziv-Welch Compression The Lempel-Ziv-Welch (LZW) compression algorithm is an example of dictionary based methods, in which longer fragments of the input text are replaced by much shorter references to code words stored in the special set called dictionary LZW is an implementation of a lossless data compression algorithm developed by Abraham Lempel and Jacob Ziv. It was published by Terry Welch in 1984 as an improved version of the LZ78 dictionary coding algorithm developed by Lempel and Ziv. 05/07/2018 Applied Algorithmics - week7

Applied Algorithmics - week7 LZW Compression The key insight of the method is that it is possible to automatically build a dictionary of previously seen strings in the text being compressed. The dictionary starts off with 256 entries, one for each possible character (single byte string). Every time a string not already in the dictionary is seen, a longer string consisting of that string appended with the single character following it in the text, is stored in the dictionary. 05/07/2018 Applied Algorithmics - week7

Applied Algorithmics - week7 LZW Compression The output consists of integer indices into the dictionary. These initially are 9 bits each, and as the dictionary grows, can increase to up to 16 bits. A special symbol is reserved for "flush the dictionary" which takes the dictionary back to the original 256 entries, and 9 bit indices. This is useful if compressing a text which has variable characteristics, since a dictionary of early material is not of much use later in the text. This use of variably increasing index sizes is one of Welch's contributions. Another was to specify an efficient data structure to store the dictionary. 05/07/2018 Applied Algorithmics - week7

LZW Compression - example Fibonacci language: w0 =a, w1=b, wi = wi-1·wi-2 for i>1 For example, w6 = babbababbabba We show how LZW compresses babbababbabba CW0 CW1 CW2 CW3 CW4 CW5 b a 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Virtual part In general: CWi = CWj o First(CWi+1) and j<i And in particular: CW4 = CW3 o First(CW5) 05/07/2018 Applied Algorithmics - week7

LZW Compression - example cw-2 = b cw-1 = a cw0 = ba cw1 = ab cw2 = bb cw3 = bab cw4 = babb cw5 = babba a b cw-1 cw-2 a b b cw1 cw2 cw0 b cw3 b cw4 a cw5 05/07/2018 Applied Algorithmics - week7

Applied Algorithmics - week7 LZW Compression - compression stage 05/07/2018 Applied Algorithmics - week7

Applied Algorithmics - week7 LZW Compression - compression stage cw  ; while ( read next symbol s from IN ) if cw·s exists in the dictionary then cw  cw·s; else add cw·s to the dictionary; save cw in OUT; cw  s; 05/07/2018 Applied Algorithmics - week7

Applied Algorithmics - week7 Decompression stage Input IN – Compressed file of integers. Output OUT – Decompressed file of characters. |IN| = Z – Size of the compressed file. Copy all numbers from file IN to vector V [256………..Z+255] Create vector F [256…Z+255] containing first characters of each code word Create vector CW [256…Z+255] of all code words for i=256 to Z+255 do if V[i] < 256 then CW[i]  Concatenate(char(V[i]), F[i+1]) else CW[i]  Concatenate(CW(V[i]), F[i+1]) Write to the output file OUT all code words without their last symbols 05/07/2018 Applied Algorithmics - week7

Applied Algorithmics - week7 Theorem: For any input string S LZW algorithm computes its compressed counterpart in time O(n), where n is the length of S. Proof: The most complex and expensive operations are performed on dictionary. However with a help of hash tables all operations can be performed in linear time. Also decompression process is linear. 05/07/2018 Applied Algorithmics - week7