1 Source Coding and Compression Dr.-Ing. Khaled Shawky Hassan Room: C3-222, ext: 1204, Lecture 7 (W5)

2 Arithmatic Coding Def.: Arithmetic coding is similar to Huffman coding; they both achieve their compression by reducing the average number of bits required to represent a symbol Unlike Huffman coding, arithmetic coding provides the ability to represent symbols with fractional values (floating point or rather fixed point representation) !!!

3 Arithmatic Coding Huffman Coding ● Replacing an input symbol with a codeword ● Hard to adapt to changing statistics ● Need to store the codeword table ● Minimum codeword length is 1 bit Arithmetic Coding ● Replace the entire input with a single floating-point number ● Adaptive coding is very easy ● No need to keep and send codeword table ● Fractional codeword length

4 Arithmatic Coding Recall the extended Huffman code ● N: alphabet size ● M: Max codeword length (order of extension) ● N^M: the total number of entries need to be encoded! Also the length of the new code words :( ● Encode sequences even if they will not appear in the context! Arithmetic Coding Features ● Normalizes the range [0, N^M] to only values in [0, 1]. ● Only consider the sequences that appears in the file. tag ● Map each input sequence to a unique tag (code, but floating point) in [0, 1).

Notes About Ranges 5 ● square brackets '[' and ']' mean the adjacent number is included ● parenthesis '(' and ')' mean the adjacent number is excluded ● Ranges assigned can then be use for encoding and decoding strings of symbols in the alphabet range coders ● Algorithms using ranges for coding are often referred to as range coders !!!!! ● The floating point tag has to be encoded further into binary format !!?? (HOW?)

Fixed-Point Arithmetic (Signed) 6 ● Let us assume a 8-bit binary sign-magnitude fixed-point representation comprising a sign bit, three integer bits, and four fractional bits ● The sign bit is used only to represent the sign of the value (0 = positive, 1 = negative). [0 is only considered in Arithmetic coding] ● Let us give an example; assume: – three integer bits that can be used to represent an integer in the range 0 to 7. [Not relevant to Arithmetic coding] – 4 bits after the fraction bit, i.e., from 0.0 to 0.934, which is divided as follows:

Fixed-Point Arithmetic (Signed) 7 8-bit binary sign-magnitude fixed-point representation

Fixed-Point Arithmetic (Signed) 8 Fixed-point representation: Integer part + Sign

Fixed-Point Arithmetic (Signed) 9 Fixed-point representation: Fraction Part

Encoding Strings 10 ● It is possible to encode a single symbol by its probability range ● Interval size is proportional to symbol probability ● More symbols can be encoded by partition its probability range by the new symbols, e.g., [0, 1)-> [0.2, 0.6) -> [0. 5, 0.60) -> [0.55, 0.58) ● The first symbol restricts the tag position to be in one of the intervals. ● Once the tag falls into an interval, it never gets out of it ● The reduced interval is partitioned recursively as more symbols are processed.

Example: 11 ● Map to real line range [0, 1) ● Order does not matter; however, decoder needs to use the same order ● partition: ● 1:[0, 0.8) - >0to …9 ● 2:[0.8, 0.82) - >0.8to …9 ● 3:[0.82, 1) - >0.82to …9 Symbol Prob

Example: 12 Now, we need to encode the input sequence: “1321” Range 14.4% Range 80% Range 100% Range 0.23 % Tag (float) = (High + Low)/2 = ( )/2 = !!!

Example: (contd.) 13 The range chart is nothing more than a CDF, where Fx(1) = 0.8, Fx(2) = 0.82, and Fx(3) = 1.0. Tag (float) = (High + Low)/2 = ( )/2 = Range = ( – ) = Accuracy in bits > ceil(log2(1/Range)) = ceil(8.7) = 9 bits!! To find the binary word, after the sign, we compute: 2^-1*1+2^-2*1+2^-3*0+2^-4*0+2^-5*0+2^-6*1+2^-7*0+2^-8*1+2^-9*1 Final word: 0| … You may take out the sign at the end!

Example: (contd.) 14 Tag (float) = (High + Low)/2 = and accuracy in bits >9 bits!! Straightforward method=> Tag*2 = ( )*2 = > 1 → 1 st bit = 1 (Ans-1)*2 = (0.5447)*2 = > 1 → 2 nd bit = 1 (Ans-1)*2 = (0.0894)*2 = < 1 → 3 rd bit = 0 (Ans-0)*2 = (0.1788)*2 = < 1 → 4 th bit = 0 (Ans-0)*2 = (0.3576)*2 = < 1 → 5 th bit = 0 (Ans-0)*2 = (0.7152)*2 = > 1 → 6 th bit = 1 (Ans-1)*2 = (0.4304)*2 = < 1 → 7 th bit = 0 (Ans-0)*2 = (0.8608)*2 = > 1 → 8 th bit = 1 (Ans-1)*2 = (0.7728)*2 = > 1 → 9 th bit = 1 –-- stop at the precision only → 9 bits Final word: 0| … You may take out the sign at the end! This value is exactly =

Encoding Pseudo-Code 15 Arithmetic Coding Pseduo-Code: Input symbols to read = 1321 => get from ReadSymbol() CDF = {0; 0.8; 0.82; 1} %0 for the first CDF(n-1) LOW=0.0, HIGH=1.0; while (not EOF) { %EOF --> no more input to the word n = ReadSymbol()+1; %this time symbols are integer RANGE = HIGH – LOW; %calculate the range HIGH = LOW + RANGE * CDF(n); %update HIGH LOW = LOW + RANGE * CDF(n-1);%update LOW } ● Keep track of three values: LOW, HIGH, RANGE ● Any two are sufficient, e.g., only LOW and RANGE. Any value between lower and upper probability can encode the input string

Decoding Procedure: Concept Decoder 16 Decode: Drawback: need to recalculate all thresholds each time

Decoding Procedure: Simplified Decoder 17 decodes the lower end: x =

Simplified Decoder Pseduo Code 18 CDF = {0,0.8,0.82,1} Low = 0; high = 1; x = GetEncodedNumber(); While (x != low) { n = DecodeOneSymbol(x); output symbol n; x = (x - CDF(n-1)) / (CDF(n) - CDF(n-1)); }; But it still needs high-precision floating point operations

Uniqueness and Efficiency 19 How to represent the final tag uniquely and efficiently? Answer: 1- Take the binary a unique value of the tag T(X) 2- Truncate T(X) to (X m is the sequence {x 1 …x m }) to this number of bits 1 bit longer than Shannon code due to selecting the TAG at the middle!! where I(X) is the bits to code a sequence {x1, …, xm} Symbol Prob = ceil(log2(1/(0.8*0.02*0.18))) + 1

Efficiency of Arithmetic Code 20 Assuming i.i.d. Sequence, then L/m → H(X) for large m.

Another Example with a Sequence 21 A= {a 1, a 2, a 3 }, P= {0.7, 0.1, 0.2) Input seq. for encoding:{a 1 a 2 a 3,... } Output Tag for decoding: cdf: F X (1) = 0.7, F X (2) = 0.8, F X (3) = 1.0

Another Example with a Sequence “iou” 22 To send and encode “iou”: Send any # C such that: ≤ C < Using Binary Fraction of (9 bits) i.e., 9 bits for Arithmetic

Another Example with a Sequence “ CAE$ ” 23 A source output symbols {A, B, C, D, E, F, $}. $ is the termination symbol. Their probabilities are as follows. P(A) = 0.2 P(B) = 0.1 P(C) = 0.2 P(D) = 0.05 P(E) = 0.3 P(F) = 0.05 P($) =

Another Example with a Sequence “ CAE$ ” 24 Assume we have an input sequence= C A E $ A B C D E F $ = Code:

Conclusion 25 For IID sequence of length m: Do you see that Arithmetic coding is worse than Huffman??!! So why consider AC: Remember, this is for coding the entire length of m symbols… You’d need N^m codewords in Huffman… which is too much! For Huffman must be kept small but for AC it can be VERY large!