1. CMPT365 Multimedia Systems 1 Arithmetic Coding Additional Material Spring 2015 CMPT 365 Multimedia Systems

2. CMPT365 Multimedia Systems 2 Outline – Part I r Introduction r Basic Encoding and Decoding r Scaling and Incremental Coding r Integer Implementation r Adaptive Arithmetic Coding r Binary Arithmetic Coding r Applications m JBIG, H.264, JPEG 2000

3. CMPT365 Multimedia Systems 3 Limitations of Huffman Code r Need a probability distribution r Hard to adapt to changing statistics r Minimum codeword length is 1 bit m Serious penalty for high-probability symbols m Example: Binary source, P(0)=0.9 Entropy: -0.9*log2(0.9)-0.1*log2(0.1) = 0.469 bit Huffman code: 0, 1  Avg. code length: 1 bit Joint coding is not practical for large alphabet. r Arithmetic coding: m Can resolve all of these problems. m Code a sequence of symbols without having to generate codes for all sequences of that length.

4. CMPT365 Multimedia Systems 4 Introduction 000 010 011 100 1 00 010 011 r Recall table look-up decoding of Huffman code m N: alphabet size m L: Max codeword length m Divide [0, 2^L] into N intervals m One interval for one symbol m Interval size is roughly proportional to symbol prob. r Arithmetic coding applies this idea recursively m Normalizes the range [0, 2^L] to [0, 1]. m Map a sequence to a unique tag in [0, 1). 0 1 abcd….. dcba…..

5. CMPT365 Multimedia Systems 5 Arithmetic Coding r Disjoint and complete partition of the range [0, 1) [0, 0.8), [0.8, 0.82), [0.82, 1) r Each interval corresponds to one symbol r Interval size is proportional to symbol probability r Observation: once the tag falls into an interval, it never gets out of it 0 1 r The first symbol restricts the tag position to be in one of the intervals r The reduced interval is partitioned recursively as more symbols are processed. 0 1 01 0 1 a b c

6. CMPT365 Multimedia Systems 6 Some Questions to think about: r Why compression is achieved this way? r How to implement it efficiently? r How to decode the sequence? r Why is it better than Huffman code?

7. CMPT365 Multimedia Systems 7 Example: SymbolProb. 10.8 20.02 30.18 1 2 3 0 0.8 0.82 1.0 r Map to real line range [0, 1) r Order does not matter m Decoder need to use the same order r Disjoint but complete partition:  1: [0, 0.8):0, 0.799999 … 9  2: [0.8, 0.82):0.8, 0.819999 … 9  3: [0.82, 1): 0.82, 0.999999 … 9 m (Think about the impact to integer implementation)

8. CMPT365 Multimedia Systems 8 Range 0.00288 1 2 3 0.7712 0.773504 0.7735616 0.77408 Range 0.144 1 2 3 0.656 0.7712 0.77408 0.8 Encoding  Input sequence: “ 1321 ” Range 1 Final range: [0.7712, 0.773504): Encode 0.7712 1 2 3 0 0.8 0.82 1.0 Range 0.8 1 2 3 0 0.64 0.656 0.8 Difficulties: 1. Shrinking of interval requires high precision for long sequence. 2. No output is generated until the entire sequence has been processed.

9. CMPT365 Multimedia Systems 9 Cumulative Density Function (CDF) X CDF 1 2 3 4 0.2 0.4 0.8 1.0 Probability Mass Function X 1 2 3 4 0.2 0.4 0.2 r For continuous distribution: r For discrete distribution: r Properties: m Non-decreasing m Piece-wise constant m Each segment is closed at the lower end.

10. CMPT365 Multimedia Systems 10 Encoder Pseudo Code low=0.0, high=1.0; while (not EOF) { n = ReadSymbol(); RANGE = HIGH - LOW; HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-1); } output LOW; InputHIGHLOWRANGE Initial1.00.01.0 10.0+1.0*0.8=0.80.0+1.0*0 = 0.00.8 30.0 + 0.8*1=0.80.0 + 0.8*0.82=0.6560.144 20.656+0.144*0.82=0.774080.656+0.144*0.8=0.77120.00288 10.7712+0.00288*0=0.77120.7712+0.00288*0.8=0.7735040.002304 r Keep track of LOW, HIGH, RANGE m Any two are sufficient, e.g., LOW and RANGE.

11. CMPT365 Multimedia Systems 11 Decode 1 1 2 3 0.7712 0.773504 0.7735616 0.77408 Decode 2 1 2 3 0.656 0.7712 0.77408 0.8 Decoding Decode 1 1 2 3 0 0.8 0.82 1.0 Decode 3 1 2 3 0 0.64 0.656 0.8 Receive 0.7712 Drawback: need to recalculate all thresholds each time.

12. CMPT365 Multimedia Systems 12 1 2 3 0 0.8 0.82 1.0 1 2 3 0 0.8 0.82 1.0 1 2 3 0 0.8 0.82 1.0 1 2 3 0 0.8 0.82 1.0 Receive 0.7712 Decode 1 x =(0.7712-0) / 0.8 = 0.964 Decode 3 Simplified Decoding r Normalize RANGE to [0, 1) each time r No need to recalculate the thresholds. x =(0.964-0.82) / 0.18 = 0.8 Decode 2 x =(0.8-0.8) / 0.02 = 0 Decode 1

13. CMPT365 Multimedia Systems 13 Decoder Pseudo Code Low = 0; high = 1; x = Encoded_number While (x ≠ low) { n = DecodeOneSymbol(x); output symbol n; x = (x - CDF(n-1)) / (CDF(n) - CDF(n-1)); };

14. CMPT365 Multimedia Systems 14 Outline r Introduction r Basic Encoding and Decoding r Scaling and Incremental Coding r Integer Implementation r Adaptive Arithmetic Coding r Binary Arithmetic Coding r Applications m JBIG, H.264, JPEG 2000

15. CMPT365 Multimedia Systems 15 Scaling and Incremental Coding r Problems of Previous examples: m Need high precision m No output is generated until the entire sequence is encoded r Key Observation:  As the RANGE reduces, many MSB ’ s of LOW and HIGH become identical: Example: Binary form of 0.7712 and 0.773504: 0.1100010.., 0.1100011..,  We can output identical MSB ’ s and re-scale the rest:  Incremental encoding m This also allows us to achieve infinite precision with finite-precision integers.

16. CMPT365 Multimedia Systems 16 E1 and E2 Scaling r E1: [LOW HIGH) in [0, 0.5) m LOW: 0.0xxxxxxx (binary), m HIGH: 0.0xxxxxxx. 0 0.5 1.0 r E2: [LOW HIGH) in [0.5, 1) m LOW: 0.1xxxxxxx, m HIGH: 0.1xxxxxxx. r Output 0, then shift left by 1 bit m [0, 0.5)  [0, 1):E1(x) = 2 x 0 0.5 1.0 r Output 1, subtract 0.5, shift left by 1 bit m [0.5, 1)  [0, 1):E2(x) = 2(x - 0.5) 0 0.5 1.0

17. CMPT365 Multimedia Systems 17 Encoding with E1 and E2 0 0.8 1.0 Input 1 0 0.656 0.8 Input 3 E2: Output 1 2(x – 0.5) 0.312 0.5424 0.54816 0.6 Input 2 0.0848 0.09632 E2: Output 1 0.1696 0.19264 E1: 2x, Output 0 E1: Output 0 0.3392 0.38528 0.6784 0.77056 E1: Output 0 0.3568 0.54112 E2: Output 1 Input 1 0.3568 0.504256 Encode any value in the tag, e.g., 0.5 Output 1 All outputs: 1100011 SymbolProb. 10.8 20.02 30.18

18. CMPT365 Multimedia Systems 18 To verify  LOW = 0.5424 (0.10001010... in binary), HIGH = 0.54816 (0.10001100... in binary). r So we can send out 10001 (0.53125) m Equivalent to E2  E1  E1  E1  E2 r After left shift by 5 bits:  LOW = (0.5424 – 0.53125) x 32 = 0.3568  HIGH = (0.54816 – 0.53125) x 32 = 0.54112 m Same as the result in the last page.

19. CMPT365 Multimedia Systems 19 Comparison with Huffman r Input Symbol 1 does not cause any output r Input Symbol 3 generates 1 bit r Input Symbol 2 generates 5 bits r Symbols with larger probabilities generates less number of bits. m Sometimes no bit is generated at all  Advantage over Huffman coding r Large probabilities are desired in arithmetic coding m Can use context-adaptive method to create larger probability and to improve compression ratio. SymbolProb. 10.8 20.02 30.18 r Note: Complete all possible scaling before encoding the next symbol

20. CMPT365 Multimedia Systems 20 Incremental Decoding 0 0.8 1.0 Decode 1: Need ≥ 5 bits (verify) Read 6 bits: Tag: 110001, 0.765625 Input 1100011 0 0.656 0.8 Decode 3, E2 scaling Tag: 100011 (0.546875) 0.312 0.5424 0.54816 0.6 0.0848 0.09632 Decode 2, E2 scaling Tag: 000110 (0.09375) 0.1696 0.19264 E1: Tag: 001100 (0.1875) E1: Tag: 011000 (0.375) 0.3392 0.38528 0.6784 0.77056 E1: Tag: 110000 (0.75) Decode 1 0.3568 0.54112 E2: Tag: 100000 (0.5) r Summary: Complete all possible scaling before further decoding Adjust LOW, HIGH and Tag together.

21. CMPT365 Multimedia Systems 21 Summary – Part I r Introduction r Encoding and Decoding r Scaling and Incremental Coding m E1, E2 r Next: m Integer Implementation E3 scaling m Adaptive Arithmetic Coding m Binary Arithmetic Coding m Applications JBIG, H.264, JPEG 2000

22. CMPT365 Multimedia Systems 22 Outline – Part II r Review r Integer Implementation m Integer representation m E3 Scaling m Minimum word length r Binary Arithmetic Coding r Adaptive Arithmetic Coding

23. CMPT365 Multimedia Systems 23 Range 0.00288 1 2 3 0.7712 0.773504 0.7735616 0.77408 Range 0.144 1 2 3 0.656 0.7712 0.77408 0.8 Encoding Without Scaling  Input sequence: “ 1321 ” Range 1 Final range: [0.7712, 0.773504): Encode 0.7712 1 2 3 0 0.8 0.82 1.0 Range 0.8 1 2 3 0 0.64 0.656 0.8

24. CMPT365 Multimedia Systems 24 E1 and E2 Scaling r E1: [LOW HIGH) in [0, 0.5) m LOW: 0.0xxxxxxx (binary), m HIGH: 0.0xxxxxxx. 0 0.5 1.0 r E2: [LOW HIGH) in [0.5, 1) m LOW: 0.1xxxxxxx, m HIGH: 0.1xxxxxxx. r Output 0, then shift left by 1 bit m [0, 0.5)  [0, 1):E1(x) = 2 x 0 0.5 1.0 r Output 1, subtract 0.5, shift left by 1 bit m [0.5, 1)  [0, 1):E2(x) = 2(x - 0.5) 0 0.5 1.0

25. CMPT365 Multimedia Systems 25 Encoding with E1 and E2 0 0.8 1.0 Input 1 0 0.656 0.8 Input 3 E2: Output 1 2(x – 0.5) 0.312 0.5424 0.54816 0.6 Input 2 0.0848 0.09632 E2: Output 1 0.1696 0.19264 E1: 2x, Output 0 E1: Output 0 0.3392 0.38528 0.6784 0.77056 E1: Output 0 0.3568 0.54112 E2: Output 1 Input 1 0.3568 0.504256 Encode any value in the tag, e.g., 0.5 Output 1 All outputs: 1100011 SymbolProb. 10.8 20.02 30.18

26. CMPT365 Multimedia Systems 26 To verify  LOW = 0.5424 (0.10001010... in binary), HIGH = 0.54816 (0.10001100... in binary). r So we can send out 10001 (0.53125) m Equivalent to E2  E1  E1  E1  E2 r After left shift by 5 bits:  LOW = (0.5424 – 0.53125) x 32 = 0.3568  HIGH = (0.54816 – 0.53125) x 32 = 0.54112 m Same as the result in the last page.

27. CMPT365 Multimedia Systems 27 Encoding Pseudo Code with E1, E2 EncodeSymbol(n) { //Update variables RANGE = HIGH - LOW; HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-1); //keep scaling before encoding next symbol while LOW, HIGH in [0, 0.5) or [0.5, 1) { send 0 for E1 and 1 for E2 scale LOW, HIGH } SymbolProb.CDF 10.8 20.020.82 30.181 (For floating-point implementation)

28. CMPT365 Multimedia Systems 28 Incremental Decoding 0 0.8 1.0 Decode 1: Need ≥ 5 bits (verify) Read 6 bits: Tag: 110001, 0.765625 Input 1100011 0 0.656 0.8 Decode 3, E2 scaling Tag: 100011 (0.546875) 0.312 0.5424 0.54816 0.6 0.0848 0.09632 Decode 2, E2 scaling Tag: 000110 (0.09375) 0.1696 0.19264 E1: Tag: 001100 (0.1875) E1: Tag: 011000 (0.375) 0.3392 0.38528 0.6784 0.77056 E1: Tag: 110000 (0.75) Decode 1 0.3568 0.54112 E2: Tag: 100000 (0.5) r Summary: Complete all possible scaling before further decoding Adjust LOW, HIGH and Tag together.

29. CMPT365 Multimedia Systems 29 Decoding Pseudo Code with E1, E2 DecodeSymbol(Tag) { RANGE = HIGH - LOW; n = 1; While ( (tag - LOW) / RANGE >= CDF(n) ) { n++; } HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-1); //keep scaling before decoding next symbol while LOW, HIGH in [0, 0.5) or [0.5, 1) { scale LOW, HIGH by E1 or E2 rule Left shift Tag and read one more bit to LSB } return n; } SymbolProb.CDF 10.8 20.020.82 30.181 (For floating-point implementation)

30. CMPT365 Multimedia Systems 30 Outline r Review r Integer Implementation m Integer representation m E3 Scaling m Complete Algorithm m Minimum word length r Binary Arithmetic Coding r Adaptive Arithmetic Coding r Applications m JBIG, H.264, JPEG 2000

31. CMPT365 Multimedia Systems 31 Integer Implementation r Old formulas: HIGH  LOW + RANGE * CDF(n); LOW  LOW + RANGE * CDF(n-1); kP( k )nknk Cum(k) 0--0 10.840 20.02141 30.18950 1 2 3 0 40 41 50 r Integer approximation of CDF ( ): m The number of occurrence of each symbol is usually collected by a counter. m Allow adaptive arithmetic coding

32. CMPT365 Multimedia Systems 32 Integer Implementation HIGH  LOW + RANGE * CDF(n); LOW  LOW + RANGE * CDF(n-1);  Why + 1 in RANGE and – 1 in HIGH? m HIGH should be less than the LOW of the next interval  The best integer value is HIGH = (next LOW) – 1  [HIGH, HIGH + 1) still belongs to the current interval, although we could not represent it explicitly. n-1 n LOW2 LOW1 0 HIGH1

33. CMPT365 Multimedia Systems 33 Example kP( k )NkNk Cum(k) 0--0 10.840 20.02141 30.18950 r For 8-bit integers, initial LOW = 0, HIGH = 255  RANGE=256 1 2 3 0 203 208 255 If n = 1:If n = 2:If n = 3:

34. CMPT365 Multimedia Systems 34 E1 Scaling for Integer r E1 Scaling: [0, 0.5)  [0, 1), E1(x) = 2 x. m LOW = 0xxxxxxx,HIGH =0xxxxxxx m Output the MSB value 0, then shift left by 1 bit r Important trick: Shift in 1 to HIGH and 0 to LOW r HIGH: 0xxxxxxx  xxxxxxx1HIGH = (HIGH << 1) + 1; r LOW: 0xxxxxxx  xxxxxxx0LOW = LOW << 1; HIGH 0. 0 x x x x x x x 1 1 1... LOW 0. 0 x x x x x x x 0 0 0...  Always assume HIGH ends with infinite number of 1 ’ s: m So that it approximates the LOW of the next interval. r This also ensures the RANGE is doubled after scaling:  HIGH – LOW + 1  (2 x HIGH + 1 – 2 x LOW + 1) = 2(HIGH – LOW + 1)

35. CMPT365 Multimedia Systems 35 E2 Scaling for Integer r E2 Scaling: [0.5, 1)  [0, 1), E2(x) = 2 (x - 0.5) m LOW = 1xxxxxxx,HIGH =1xxxxxxx m Output the MSB, then shift left by 1 bit (mul by 2) r Same trick: Shift in 1 to HIGH and 0 to LOW r HIGH: 1xxxxxxx  xxxxxxx1 HIGH = (HIGH << 1) + 1; r LOW: 1xxxxxxx  xxxxxxx0LOW = LOW << 1; 0. 1 x x x x x x x 1 1 1... 0. 1 x x x x x x x 0 0 0...

36. CMPT365 Multimedia Systems 36 0 203 255 Input 1 0 167 203 Input 3 E2: Output 1 78 151 Integer Encoding LOW: 167 (10100111)HIGH: 203 (11001011) After E2: (shift in an 1 to HIGH and 0 to LOW) LOW: 1(01001110) 78 (8-bit)HIGH: 1(10010111) 151 (8-bit) In 8.1 format (8 bits for integer, 1 bit for fractional): LOW: (10100111.0) 167HIGH: (11001011.1) 203.5  By shifting in an 1 to HIGH, we can cover the range [203, 203.5].  The entire range [203, 204) can be covered by always shifting in 1 to HIGH. [0, 0.8)  LOW = 0, HIGH = 203. [0.8, 0.82)  LOW = 204, HIGH = 208. Can we represent an interval in [203, 204) ? (Sequence 1333333……)

37. CMPT365 Multimedia Systems 37 Outline r Review r Integer Implementation m Integer representation m E3 Scaling m Complete Algorithm m Minimum word length r Binary Arithmetic Coding r Adaptive Arithmetic Coding

38. CMPT365 Multimedia Systems 38 E3 Scaling: [0.25, 0.75)  [0, 1) r If RANGE straddles 1/2, E1 and E2 cannot be applied, but the range can be quite small m Example: LOW=0.4999, HIGH=0.5001 Binary: LOW=0.01111 …., HIGH=0.1000 … m We may not have enough bits to represent the interval. 01 0.5 0.25 0.75 r E3 Scaling: [0.25, 0.75)  [0, 1): E3(x) = 2(x - 0.25) 01 0.5

39. CMPT365 Multimedia Systems 39 Integer Implementation of E3 01xxxxxx - 01000000 00xxxxxx × 2 0xxxxxx0 LOW: 10xxxxxx - 01000000 01xxxxxx × 2 1xxxxxx0 + 1 1xxxxxx1 HIGH:  Another way to implement E3 (Sayood book pp. 99): m Left shift old LOW and HIGH, complement new MSB. Same trick: Shift in 1 to HIGH and 0 to LOW HIGH = ((HIGH – QUARTER) << 1) + 1; LOW = (LOW - QUARTER) << 1; QUARTER = 2^(M - 2) for m-bit integer. (64 for m = 8 bits) LOW 01xxxxxx  0xxxxxx0 HIGH 10xxxxxx  1xxxxxx1

40. CMPT365 Multimedia Systems 40 Signaling of E3 r What should we send when E3 is used? m Recall: we send 1 if E2 is used, send 0 if E1 is used r What do they mean? m A series of E3 followed by an E1 is equivalent to an E1 followed by a series of E2. m A series of E3 followed by an E2 is equivalent to an E2 followed by a series of E1. r Important relationships: ( www.copro.org/download/ac_en.pdf) Apply n Ej scalings, followed by an Ei scaling.

41. CMPT365 Multimedia Systems 41 Example 0 0.8 1.0 Input 1 0 0.656 0.8 Input 3 0.312 0.5424 0.54816 0.6 Input 2 0.0848 0.09632 E2: Output 1 0.1696 0.19264 E1: Output 0 r Previous example without E3: r With E3: 0.312 0.6 0.124 0.7 E3: (x-0.25)x2 0.1696 0.19264 E2: Output 1 r The range after E2°E3 is the same as that after E1°E2 Input 2 0.5848 0.59632

42. CMPT365 Multimedia Systems 42 Another View of the Equivalence 0 0.25 0.5 0.75 1 r Scaling of a range in [0.5, 0.75) with E1°E2 r Equivalent scaling of the range in [0.5, 0.75) with E2°E3 E2 E1 E3 E2

43. CMPT365 Multimedia Systems 43 A Simple Proof of E2 ° E3 = E1 ° E2 r Given the same range r: m After applying E3: [0.25, 0.75)  [0, 1), the range becomes r3 = (r – 0.25) x 2 m After applying E2, the range becomes r4 = (r3 – 0.5) x 2 = ((r – 0.25) x 2 – 0.5) x 2 = (r – 0.5) x 2 x 2 = r2 r Given an original range r: m After applying E2: [0.5, 1)  [0, 1), the range becomes r1 = (r – 0.5) x 2 m After applying E1: [0, 0.5)  [0, 1), the range becomes r2 = r1 x 2 = ((r – 0.5) x 2) x 2 For formal proof: www.copro.org/download/ac_en.pdfwww.copro.org/download/ac_en.pdf

44. CMPT365 Multimedia Systems 44 Encoding Operation with E3 0.312 0.5424 0.54816 0.6 Input 2 0.0848 0.09632 E2: Output 1 0.1696 0.19264 E1: Output 0 r With E3: 0.312 0.6 0.124 0.7 E3 (no output here) 0.1696 0.19264 E2: Output 1 Input 2 0.5848 0.59632 r Without E3:  Don ’ t send anything when E3 is used, but send a 0 after E2: m The bit stream is identical to that of the old method m Subsequent encoding is also same because of the same final interval Output 0 here!

45. CMPT365 Multimedia Systems 45 Decoding for E3 0 0.8 1.0 Input 1100011 Read 6 bits: Tag: 110001 (0.765625) Decode 1 0 0.656 0.8 Decode 3, E2 scaling Tag: 100011 (0.546875) 0.312 0.5424 0.54816 0.6 0.0848 0.09632 Decode 2, E2 scaling Tag: 000110 (0.09375) 0.1696 0.19264 E1: Tag: 001100 (0.1875) r With E3: Apply E3 whenever it is possible, nothing else is needed 0.312 0.6 Tag: 100011 (0.546875) 0.5848 0.59632 0.124 0.7 E3: Tag: 100110 (0.59375) 0.1696 0.19264 Decode 2, E2 scaling Tag: 001100 (0.1875) Same status as the old method: low, high, range, tag.

46. CMPT365 Multimedia Systems 46 Summary of Different Scalings Need E1 scaling Need E2 scaling Need E3 scaling No scaling is required. Ready to encode/decode the next symbol. 0 0.25 0.5 0.75 1.0

47. CMPT365 Multimedia Systems 47 Outline r Review r Integer Implementation m Integer representation m E3 Scaling m Complete Algorithm m Minimum word length r Binary Arithmetic Coding r Adaptive Arithmetic Coding

48. CMPT365 Multimedia Systems 48 Encoding Pseudo Code with E1, E2, E3 EncodeSymbol(n) { //Update variables RANGE = HIGH - LOW + 1; HIGH = HIGH + RANGE * Cum( n ) / N - 1; LOW = LOW + RANGE * Cum(n-1) / N; //Scaling before encoding next symbol EncoderScaling();//see next slide } (For integer implementation) Round off to integer

49. CMPT365 Multimedia Systems 49 Encoding Pseudo Code with E1, E2, E3 EncoderScaling ( ) { while (E1, E2 or E3 is possible) { if (E3 is possible) { HIGH = ((HIGH - QUARTER) << 1) + 1; LOW = (LOW - QUARTER) << 1; Scale3++; //Save number of E3, but send nothing } if (E1 or E2 is possible) { Let b=0 for E1 and b=1 for E2 send b HIGH = (HIGH << 1) + 1; LOW = (LOW << 1); while (Scale3 > 0) { //send info about E3 now send complement of b // E2 ° ( E3)^n = (E1)^n ° E2 Scale3 --; //Send one bit for each E3 }

50. CMPT365 Multimedia Systems 50 DecodeSymbol(Tag) { RANGE = HIGH - LOW + 1; n = 1; While (Tag > LOW + RANGE * Cum(n) / N - 1) { n++; } HIGH = LOW + RANGE * Cum(n) / N - 1; LOW = LOW + RANGE * Cum(n-1) / N; //keep scaling before decoding next symbol DecoderScaling(Tag);//next slide return n; } Decoding Pseudo Code with E1, E2, E3 (For integer implementation) Intervals: [0, 203], [204, 208], [209, 255] Round off to integer: HIGH of each interval

51. CMPT365 Multimedia Systems 51 Decoding Pseudo Code with E1, E2, E3 DecoderScaling (Tag) { while (E1, E2 or E3 is possible) { if (E1 or E2 is possible) { LOW = LOW << 1; HIGH = (HIGH << 1) + 1; Tag = Tag << 1; Tag = Tag | ReadBits(1); } if (E3 is possible) { LOW = (LOW - QUARTER) << 1; HIGH = ((HIGH - QUARTER) << 1) + 1; Tag = (Tag - QUARTER) << 1; Tag = Tag | ReadBits(1); }

52. CMPT365 Multimedia Systems 52 0 203 255 Input 1 0 167 203 Input 3 E2: Output 1 78 151 E3: Scale3=1 Input 2 28 146 148 175 72 83 E1: Output 0 32 79 E2: Output 1 144 167 E1: Output 0 36 41 E2: Output 1 Output 0 Scale3 = 0 64 159 E1: Output 0 Integer Encoding with E1, E2, E3 E3: Scale3=1 0152 191 Output 0 Output 1 Scale3=0 Output 7 more 0’s Input 1 Final output: 11000100 10000000

53. CMPT365 Multimedia Systems 53 0 203 255 Integer Decoding with E1, E2, E3 Read 8 bits: 11000100 (196) Decode 1 0 167 203 Decode 3 Input: 11000100 10000000 E3: Tag = 10010010 (146) Decode 2 28 146 148 175 78 151 E2: Tag=10001001 (137) 36 41 E2: Tag=00100100 (36) Decode 1 Tag = LOW: stop.

54. CMPT365 Multimedia Systems 54 Outline r Review r Integer Implementation m Integer representation m E3 Scaling m Complete Algorithm m Minimum word length r Binary Arithmetic Coding r Adaptive Arithmetic Coding

55. CMPT365 Multimedia Systems 55 How to decide the word length m? r Need to guarantee non-zero interval for all symbols in the worst case: kP( k )NkNk Cum(k) 0--0 10.840 20.02141 30.18950 1 2 3 0 40 41 50 even when Cum( n ) – cum( n -1) = 1. Otherwise HIGH < LOW. Need  RANGE cannot be too small at any time. Intuitive!

56. CMPT365 Multimedia Systems 56 How to decide the word length m? r Condition: 1/4 (2^m) > N r Example: N = 50,  min m = 8 (1/4M=64) r When do we have the smallest RANGE without triggering a scaling? 0 M/4 M/2 3/4M M M = 2^m m When interval is slightly larger than [M/4, M/2] or [M/2, 3/4M] m None of E1, E2, and E3 can be applied

57. CMPT365 Multimedia Systems 57 Outline r Review r Integer Implementation r Binary Arithmetic Coding r Adaptive Arithmetic Coding

58. CMPT365 Multimedia Systems 58 Binary Arithmetic Coding r Arithmetic coding is slow in general: To decode a symbol, we need a seris of decisions and multiplications: While (Tag > LOW + RANGE * Cum(n) / N - 1) { n++; } r The complexity is greatly reduced if we have only two symbols: 0 and 1. symbol 0 symbol 1 r Only two intervals in [0, 1): [0, x), [x, 1) 0 x 1

59. CMPT365 Multimedia Systems 59 Encoding of Binary Arithmetic Coding Prob(0)=0.6. Sequence: 0110 LOW = 0, HIGH = 1 0 0.6 1 LOW = 0, HIGH = 0.6 0 0.36 0.6 LOW = 0.36, HIGH = 0.6 0.36 0.504 0.6 LOW = 0.504, HIGH = 0.6 0.504 0.5616 0.6 LOW = 0.504, HIGH = 0.5616 Only need to update LOW or HIGH for each symbol.

60. CMPT365 Multimedia Systems 60 Decoding of Binary Arithmetic Coding 0 0.6 1 Tag While (Tag > LOW + RANGE * Cum(n) / N - 1)) { n++; } Only one decision to make: if (Tag > LOW + RANGE * Cum(Symbol 0) / N - 1) { n = 1; } else { n = 0; }

61. CMPT365 Multimedia Systems 61 Applications of Binary Arithmetic Coding r Increasingly popular: m JBIG, JBIG2, JPEG2000, H.264 r Covert non-binary signals into binary first m H.264: Golomb-Rice Code m Bit-plane coding r Various simplifications to avoid multiplication: m H.264: Table look-up for RANGE * Cum(n) / N m JBIG: Eliminate multiplication by assuming the RANGE is close to 1. Scale if RANGE too small.

62. CMPT365 Multimedia Systems 62 Outline r Review r Integer Implementation r Binary Arithmetic Coding r Adaptive Arithmetic Coding

63. CMPT365 Multimedia Systems 63 Adaptive Arithmetic Coding r Observation: The partition of [0, 1) can be different from symbol to symbol r The bit stream can be decoded perfectly as long as both encoder and decoder are synchronized (use the same partition). r General approach: m Starting from a pre-defined probability distribution m Update probability after each symbol r This is very difficult for Huffman coding: m Has to redesign the codebook when prob changes

64. CMPT365 Multimedia Systems 64 Example 0 0.5 1 r Binary sequence: 01111  Initial counters for 0 ’ s and 1 ’ s: C(0)=C(1)=1.  P(0)=P(1)=0.5 0 0.3333 0.5 r After encoding 0: C(0)=2, C(1)=1.  P(0)=2/3, P(1)=1/3 0.3333 0.4167 0.5 r After encoding 01: C(0)=2, C(1)=2.  P(0)=1/2, P(1)=1/2 0.4167 0.45 0.5 r After encoding 011: C(0)=2, C(1)=3.  P(0)=2/5, P(1)=3/5 0.45 0.4667 0.5 r After encoding 0111: C(0)=2, C(1)=4.  P(0)=1/3, P(1)=2/3. r Encode 0.4667.

65. CMPT365 Multimedia Systems 65 Decoding r Input 0.4667.  Initial counters for 0 ’ s and 1 ’ s: C(0)=C(1)=1  P(0)=P(1)=0.5 Decode 0 0 0.5 1 0 0.3333 0.5 r After decoding 0: C(0)=2, C(1)=1.  P(0)=2/3, P(1)=1/3 Decode 1 r After decoding 01: C(0)=2, C(1)=2.  P(0)=1/2, P(1)=1/2 Decode 1 0.3333 0.4167 0.5 r After decoding 011: C(0)=2, C(1)=3.  P(0)=2/5, P(1)=3/5 Decode 1 0.4167 0.45 0.5 r After decoding 0111: C(0)=2, C(1)=4.  P(0)=1/3, P(1)=2/3. Decode 1 0.45 0.4667 0.5

66. CMPT365 Multimedia Systems 66 Context-adaptive Arithmetic Coding r In many cases, a sample has strong correlation with its near neighbors. r Idea: m Collect conditional probability distribution of a symbol for given neighboring symbols (context): P(x(n) | x(n-1), x(n-2), … x(n-k)) m Use this conditional probability to encode the next symbol m More skewed probability distribution can be obtained (desired by arithmetic coding) a b c b c a b 1-D Context template a b c b c a b c b a b c b a 2-D Context template

67. CMPT365 Multimedia Systems 67 Example r Binary sequence: 0, 1 r Neighborhood (template) size: 3  2^3=8 possible combinations (contexts) of 3 neighbors (x(n-1), x(n-2), x(n-3)).  Collect frequencies of 0 ’ s and 1 ’ s under each context 0 1 1 0 1 0 1 ContextC(0)C(1) (0, 0, 0)92 (0, 0, 1)36 (0, 1, 0)10 (0, 1, 1) …… (1, 0, 0) …… (1, 0, 1) …… (1, 1, 0) …… (1, 1, 1) …… 0 1 Each symbol is coded with the probability distribution associated with its context.

68. CMPT365 Multimedia Systems 68 Encoding Pseudo Code InitProbModel(ProbModel); While (not EOF) { n = ReadSymbol( ); //Determine the neighboring combination (context) context = GetContext(); //Encode with the corresponding probability model EncodeSymbol(ProbModel[context], n); //update probability counters for this context UpdateProb(ProbModel[context], n); }

69. CMPT365 Multimedia Systems 69 Decoding Pseudo Code InitProbModel(ProbModel); While (not EOF) { //Determine the neighboring combination (context) context = GetContext(); //Decode with the corresponding probability model n = DecodeSymbol(ProbModel[context]); //update probability counters for this context UpdateProb(ProbModel[context], n); }

70. CMPT365 Multimedia Systems 70 Performance of Arithmetic Coding r For a sequence of length m: r H(X) <= Average Bit Rate <= H(X) + 2/m m Can approaches the entropy very quickly. r Huffman coding: m H(X) <= Average Bit Rate <= H(X) + 1/m m Impractical: need to generate codewords for all sequences of length m.

71. CMPT365 Multimedia Systems 71 Summary – Part II r Arithmetic coding: m Partition the range [0, 1) recursively according to symbol probabilities. r Incremental Encoding and Decoding m E1, E2, E3 scaling r Binary Arithmetic Coding r Context Adaptive Arithmetic Coding r Next: Quantization (lossy compression)