1. 1 Source Coding and Compression Dr.-Ing. Khaled Shawky Hassan Room: C3-222, ext: 1204, Email: khaled.shawky@guc.edu.eg Lecture 7 (W5)

2. 2 Arithmatic Coding Def.: Arithmetic coding is similar to Huffman coding; they both achieve their compression by reducing the average number of bits required to represent a symbol Unlike Huffman coding, arithmetic coding provides the ability to represent symbols with fractional values (floating point or rather fixed point representation) !!!

3. 3 Arithmatic Coding Huffman Coding ● Replacing an input symbol with a codeword ● Hard to adapt to changing statistics ● Need to store the codeword table ● Minimum codeword length is 1 bit Arithmetic Coding ● Replace the entire input with a single floating-point number ● Adaptive coding is very easy ● No need to keep and send codeword table ● Fractional codeword length

4. 4 Arithmatic Coding Recall the extended Huffman code ● N: alphabet size ● M: Max codeword length (order of extension) ● N^M: the total number of entries need to be encoded! Also the length of the new code words :( ● Encode sequences even if they will not appear in the context! Arithmetic Coding Features ● Normalizes the range [0, N^M] to only values in [0, 1]. ● Only consider the sequences that appears in the file. tag ● Map each input sequence to a unique tag (code, but floating point) in [0, 1).

5. Notes About Ranges 5 ● square brackets '[' and ']' mean the adjacent number is included ● parenthesis '(' and ')' mean the adjacent number is excluded ● Ranges assigned can then be use for encoding and decoding strings of symbols in the alphabet range coders ● Algorithms using ranges for coding are often referred to as range coders !!!!! ● The floating point tag has to be encoded further into binary format !!?? (HOW?)

6. Fixed-Point Arithmetic (Signed) 6 ● Let us assume a 8-bit binary sign-magnitude fixed-point representation comprising a sign bit, three integer bits, and four fractional bits ● The sign bit is used only to represent the sign of the value (0 = positive, 1 = negative). [0 is only considered in Arithmetic coding] ● Let us give an example; assume: – three integer bits that can be used to represent an integer in the range 0 to 7. [Not relevant to Arithmetic coding] – 4 bits after the fraction bit, i.e., from 0.0 to 0.934, which is divided as follows:

7. Fixed-Point Arithmetic (Signed) 7 8-bit binary sign-magnitude fixed-point representation

8. Fixed-Point Arithmetic (Signed) 8 Fixed-point representation: Integer part + Sign

9. Fixed-Point Arithmetic (Signed) 9 Fixed-point representation: Fraction Part

10. Encoding Strings 10 ● It is possible to encode a single symbol by its probability range ● Interval size is proportional to symbol probability ● More symbols can be encoded by partition its probability range by the new symbols, e.g., [0, 1)-> [0.2, 0.6) -> [0. 5, 0.60) -> [0.55, 0.58) ● The first symbol restricts the tag position to be in one of the intervals. ● Once the tag falls into an interval, it never gets out of it ● The reduced interval is partitioned recursively as more symbols are processed.

11. Example: 11 ● Map to real line range [0, 1) ● Order does not matter; however, decoder needs to use the same order ● partition: ● 1:[0, 0.8) - >0to0.799999…9 ● 2:[0.8, 0.82) - >0.8to0.819999…9 ● 3:[0.82, 1) - >0.82to0.999999…9 Symbol Prob. 10.8 20.02 30.18

12. Example: 12 Now, we need to encode the input sequence: “1321” Range 14.4% Range 80% Range 100% Range 0.23 % Tag (float) = (High + Low)/2 = (0.773504 + 0.7712)/2 = 0.77235 !!!

13. Example: (contd.) 13 The range chart is nothing more than a CDF, where Fx(1) = 0.8, Fx(2) = 0.82, and Fx(3) = 1.0. Tag (float) = (High + Low)/2 = (0.773504 + 0.7712)/2 = 0.77235 Range = (0.773504 – 0.7712) = 0.0023040 Accuracy in bits > ceil(log2(1/Range)) = ceil(8.7) = 9 bits!! To find the binary word, after the sign, we compute: 2^-1*1+2^-2*1+2^-3*0+2^-4*0+2^-5*0+2^-6*1+2^-7*0+2^-8*1+2^-9*1 Final word: 0|110001011 … You may take out the sign at the end!

14. Example: (contd.) 14 Tag (float) = (High + Low)/2 = 0.77235 and accuracy in bits >9 bits!! Straightforward method=> Tag*2 = (0.77235)*2 = 1.5447> 1 → 1 st bit = 1 (Ans-1)*2 = (0.5447)*2 = 1.0894> 1 → 2 nd bit = 1 (Ans-1)*2 = (0.0894)*2 = 0.1788< 1 → 3 rd bit = 0 (Ans-0)*2 = (0.1788)*2 = 0.3576< 1 → 4 th bit = 0 (Ans-0)*2 = (0.3576)*2 = 0.7152< 1 → 5 th bit = 0 (Ans-0)*2 = (0.7152)*2 = 1.4304> 1 → 6 th bit = 1 (Ans-1)*2 = (0.4304)*2 = 0.8608< 1 → 7 th bit = 0 (Ans-0)*2 = (0.8608)*2 = 1.7728> 1 → 8 th bit = 1 (Ans-1)*2 = (0.7728)*2 = 1.5456> 1 → 9 th bit = 1 –-- stop at the precision only → 9 bits Final word: 0|110001011 … You may take out the sign at the end! This value is exactly = 0.77148

15. Encoding Pseudo-Code 15 Arithmetic Coding Pseduo-Code: Input symbols to read = 1321 => get from ReadSymbol() CDF = {0; 0.8; 0.82; 1} %0 for the first CDF(n-1) LOW=0.0, HIGH=1.0; while (not EOF) { %EOF --> no more input to the word n = ReadSymbol()+1; %this time symbols are integer RANGE = HIGH – LOW; %calculate the range HIGH = LOW + RANGE * CDF(n); %update HIGH LOW = LOW + RANGE * CDF(n-1);%update LOW } ● Keep track of three values: LOW, HIGH, RANGE ● Any two are sufficient, e.g., only LOW and RANGE. Any value between lower and upper probability can encode the input string

16. Decoding Procedure: Concept Decoder 16 Decode:0.7712 Drawback: need to recalculate all thresholds each time

17. Decoding Procedure: Simplified Decoder 17 decodes the lower end: x = 0.7712

18. Simplified Decoder Pseduo Code 18 CDF = {0,0.8,0.82,1} Low = 0; high = 1; x = GetEncodedNumber(); While (x != low) { n = DecodeOneSymbol(x); output symbol n; x = (x - CDF(n-1)) / (CDF(n) - CDF(n-1)); }; But it still needs high-precision floating point operations

19. Uniqueness and Efficiency 19 How to represent the final tag uniquely and efficiently? Answer: 1- Take the binary a unique value of the tag T(X) 2- Truncate T(X) to (X m is the sequence {x 1 …x m }) to this number of bits 1 bit longer than Shannon code due to selecting the TAG at the middle!! where I(X) is the bits to code a sequence {x1, …, xm} Symbol Prob. 10.8 20.02 30.18 = ceil(log2(1/(0.8*0.02*0.18))) + 1

20. Efficiency of Arithmetic Code 20 Assuming i.i.d. Sequence, then L/m → H(X) for large m.

21. Another Example with a Sequence 21 A= {a 1, a 2, a 3 }, P= {0.7, 0.1, 0.2) Input seq. for encoding:{a 1 a 2 a 3,... } Output Tag for decoding: 0.50391 cdf: F X (1) = 0.7, F X (2) = 0.8, F X (3) = 1.0

22. Another Example with a Sequence “iou” 22 To send and encode “iou”: Send any # C such that: 0.37630 ≤ C < 0.37819 Using Binary Fraction of 0.011000001 (9 bits) i.e., 9 bits for Arithmetic

23. Another Example with a Sequence “ CAE$ ” 23 A source output symbols {A, B, C, D, E, F, $}. $ is the termination symbol. Their probabilities are as follows. P(A) = 0.2 P(B) = 0.1 P(C) = 0.2 P(D) = 0.05 P(E) = 0.3 P(F) = 0.05 P($) = 0.1 1 0 0.2 0.3 0.5 0.55 0.85 0.9

24. Another Example with a Sequence “ CAE$ ” 24 Assume we have an input sequence= C A E $ 1 0 0.2 0.3 0.5 0.55 0.85 0.9 A B C D E F $ 0.5 0.3 0.34 0.334 0.322 0.3328 0.333 = 0.0101010101 Code: 0101010101 0.34 0.3 0.322 0.334

25. Conclusion 25 For IID sequence of length m: Do you see that Arithmetic coding is worse than Huffman??!! So why consider AC: Remember, this is for coding the entire length of m symbols… You’d need N^m codewords in Huffman… which is too much! For Huffman must be kept small but for AC it can be VERY large!