CMPT 365 Multimedia Systems Arithmetic Coding Additional Material Fall 2012

Outline – Part I Introduction Basic Encoding and Decoding Scaling and Incremental Coding Integer Implementation Adaptive Arithmetic Coding Binary Arithmetic Coding Applications JBIG, H.264, JPEG 2000

Limitations of Huffman Code Need a probability distribution Hard to adapt to changing statistics Minimum codeword length is 1 bit Serious penalty for high-probability symbols Example: Binary source, P(0)=0.9 Entropy: -0.9*log2(0.9)-0.1*log2(0.1) = 0.469 bit Huffman code: 0, 1  Avg. code length: 1 bit Joint coding is not practical for large alphabet. Arithmetic coding: Can resolve all of these problems. Code a sequence of symbols without having to generate codes for all sequences of that length.

Introduction abcd….. dcba….. Recall table look-up decoding of Huffman code N: alphabet size L: Max codeword length Divide [0, 2^L] into N intervals One interval for one symbol Interval size is roughly proportional to symbol prob. 1 00 010 011 000 010 011 100 Arithmetic coding applies this idea recursively Normalizes the range [0, 2^L] to [0, 1]. Map a sequence to a unique tag in [0, 1). 0 1 abcd….. dcba…..

Arithmetic Coding Disjoint and complete partition of the range [0, 1) 1 a b c Disjoint and complete partition of the range [0, 1) [0, 0.8), [0.8, 0.82), [0.82, 1) Each interval corresponds to one symbol Interval size is proportional to symbol probability 0 1 The first symbol restricts the tag position to be in one of the intervals The reduced interval is partitioned recursively as more symbols are processed. 1 1 Observation: once the tag falls into an interval, it never gets out of it

Some Questions to think about: Why compression is achieved this way? How to implement it efficiently? How to decode the sequence? Why is it better than Huffman code?

Example: 1 0.8 2 0.02 3 0.18 Map to real line range [0, 1) 1 2 3 0 0.8 0.82 1.0 Symbol Prob. 1 0.8 2 0.02 3 0.18 Map to real line range [0, 1) Order does not matter Decoder need to use the same order Disjoint but complete partition: 1: [0, 0.8): 0, 0.799999…9 2: [0.8, 0.82): 0.8, 0.819999…9 3: [0.82, 1): 0.82, 0.999999…9 (Think about the impact to integer implementation)

Encoding Input sequence: “1321” 1 2 3 0 0.8 0.82 1.0 Range 1 Range 0.8 1 2 3 0 0.64 0.656 0.8 Range 0.144 1 2 3 0.656 0.7712 0.77408 0.8 Range 0.00288 1 2 3 0.7712 0.773504 0.7735616 0.77408 Final range: [0.7712, 0.773504): Encode 0.7712 Difficulties: 1. Shrinking of interval requires high precision for long sequence. 2. No output is generated until the entire sequence has been processed.

Cumulative Density Function (CDF) Probability Mass Function X 1 2 3 4 0.2 0.2 0.4 0.2 For continuous distribution: For discrete distribution: X CDF 1 2 3 4 0.2 0.4 0.8 1.0 Properties: Non-decreasing Piece-wise constant Each segment is closed at the lower end.

Encoder Pseudo Code Keep track of LOW, HIGH, RANGE while (not EOF) { n = ReadSymbol(); RANGE = HIGH - LOW; HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-1); } output LOW; Keep track of LOW, HIGH, RANGE Any two are sufficient, e.g., LOW and RANGE. Input HIGH LOW RANGE Initial 1.0 0.0 1 0.0+1.0*0.8=0.8 0.0+1.0*0 = 0.0 0.8 3 0.0 + 0.8*1=0.8 0.0 + 0.8*0.82=0.656 0.144 2 0.656+0.144*0.82=0.77408 0.656+0.144*0.8=0.7712 0.00288 0.7712+0.00288*0=0.7712 0.7712+0.00288*0.8=0.773504 0.002304

Decoding Drawback: need to recalculate all thresholds each time. Receive 0.7712 1 2 3 0 0.8 0.82 1.0 Decode 1 Decode 3 1 2 3 0 0.64 0.656 0.8 Decode 2 1 2 3 0.656 0.7712 0.77408 0.8 Decode 1 1 2 3 0.7712 0.773504 0.7735616 0.77408 Drawback: need to recalculate all thresholds each time.

Simplified Decoding Normalize RANGE to [0, 1) each time No need to recalculate the thresholds. Receive 0.7712 Decode 1 1 2 3 0 0.8 0.82 1.0 1 2 3 0 0.8 0.82 1.0 x =(0.7712-0) / 0.8 = 0.964 Decode 3 1 2 3 0 0.8 0.82 1.0 x =(0.964-0.82) / 0.18 = 0.8 Decode 2 x =(0.8-0.8) / 0.02 = 0 Decode 1 1 2 3 0 0.8 0.82 1.0

Decoder Pseudo Code Low = 0; high = 1; x = Encoded_number While (x ≠ low) { n = DecodeOneSymbol(x); output symbol n; x = (x - CDF(n-1)) / (CDF(n) - CDF(n-1)); };

Outline Introduction Basic Encoding and Decoding Scaling and Incremental Coding Integer Implementation Adaptive Arithmetic Coding Binary Arithmetic Coding Applications JBIG, H.264, JPEG 2000

Scaling and Incremental Coding Problems of Previous examples: Need high precision No output is generated until the entire sequence is encoded Key Observation: As the RANGE reduces, many MSB’s of LOW and HIGH become identical: Example: Binary form of 0.7712 and 0.773504: 0.1100010.., 0.1100011.., We can output identical MSB’s and re-scale the rest:  Incremental encoding This also allows us to achieve infinite precision with finite-precision integers.

E1 and E2 Scaling E1: [LOW HIGH) in [0, 0.5) LOW: 0.0xxxxxxx (binary), HIGH: 0.0xxxxxxx. 0 0.5 1.0 Output 0, then shift left by 1 bit [0, 0.5) [0, 1): E1(x) = 2 x 0 0.5 1.0 0 0.5 1.0 E2: [LOW HIGH) in [0.5, 1) LOW: 0.1xxxxxxx, HIGH: 0.1xxxxxxx. Output 1, subtract 0.5, shift left by 1 bit [0.5, 1) [0, 1): E2(x) = 2(x - 0.5) 0 0.5 1.0

Encoding with E1 and E2 Input 1 0 0.8 1.0 Input 3 0 0.656 0.8 Symbol Prob. 1 0.8 2 0.02 3 0.18 Input 1 0 0.656 0.8 Input 3 0 0.8 1.0 E2: Output 1 2(x – 0.5) 0.312 0.5424 0.54816 0.6 Input 2 0.0848 0.09632 E2: Output 1 0.1696 0.19264 E1: 2x, Output 0 E1: Output 0 0.3392 0.38528 0.6784 0.77056 E1: Output 0 0.3568 0.54112 E2: Output 1 Input 1 Encode any value in the tag, e.g., 0.5 Output 1 All outputs: 1100011 0.3568 0.504256

To verify LOW = 0.5424    (0.10001010... in binary), HIGH = 0.54816 (0.10001100... in binary). So we can send out 10001 (0.53125) Equivalent to E2E1E1E1E2 After left shift by 5 bits: LOW = (0.5424 – 0.53125) x 32 = 0.3568 HIGH = (0.54816 – 0.53125) x 32 = 0.54112 Same as the result in the last page.

Comparison with Huffman Note: Complete all possible scaling before encoding the next symbol Symbol Prob. 1 0.8 2 0.02 3 0.18 Comparison with Huffman Input Symbol 1 does not cause any output Input Symbol 3 generates 1 bit Input Symbol 2 generates 5 bits Symbols with larger probabilities generates less number of bits. Sometimes no bit is generated at all  Advantage over Huffman coding Large probabilities are desired in arithmetic coding Can use context-adaptive method to create larger probability and to improve compression ratio.

Incremental Decoding Input 1100011 Decode 1: Need ≥ 5 bits (verify) Read 6 bits: Tag: 110001, 0.765625 0 0.656 0.8 0 0.8 1.0 Decode 3, E2 scaling Tag: 100011 (0.546875) 0.312 0.5424 0.54816 0.6 0.0848 0.09632 Decode 2, E2 scaling Tag: 000110 (0.09375) 0.1696 0.19264 E1: Tag: 001100 (0.1875) E1: Tag: 011000 (0.375) 0.3392 0.38528 0.6784 0.77056 E1: Tag: 110000 (0.75) Decode 1 0.3568 0.54112 E2: Tag: 100000 (0.5) Summary: Complete all possible scaling before further decoding Adjust LOW, HIGH and Tag together.

Summary – Part I Introduction Encoding and Decoding Scaling and Incremental Coding E1, E2 Next: Integer Implementation E3 scaling Adaptive Arithmetic Coding Binary Arithmetic Coding Applications JBIG, H.264, JPEG 2000

Outline – Part II Review Integer Implementation Integer representation E3 Scaling Minimum word length Binary Arithmetic Coding Adaptive Arithmetic Coding

Encoding Without Scaling 1 2 3 0 0.8 0.82 1.0 Range 1 Range 0.8 1 2 3 0 0.64 0.656 0.8 Range 0.144 1 2 3 0.656 0.7712 0.77408 0.8 Range 0.00288 1 2 3 0.7712 0.773504 0.7735616 0.77408 Input sequence: “1321” Final range: [0.7712, 0.773504): Encode 0.7712

E1 and E2 Scaling E1: [LOW HIGH) in [0, 0.5) LOW: 0.0xxxxxxx (binary), HIGH: 0.0xxxxxxx. 0 0.5 1.0 Output 0, then shift left by 1 bit [0, 0.5) [0, 1): E1(x) = 2 x 0 0.5 1.0 0 0.5 1.0 E2: [LOW HIGH) in [0.5, 1) LOW: 0.1xxxxxxx, HIGH: 0.1xxxxxxx. Output 1, subtract 0.5, shift left by 1 bit [0.5, 1) [0, 1): E2(x) = 2(x - 0.5) 0 0.5 1.0

Encoding with E1 and E2 Input 1 0 0.8 1.0 Input 3 0 0.656 0.8 Symbol Prob. 1 0.8 2 0.02 3 0.18 Input 1 0 0.656 0.8 Input 3 0 0.8 1.0 E2: Output 1 2(x – 0.5) 0.312 0.5424 0.54816 0.6 Input 2 0.0848 0.09632 E2: Output 1 0.1696 0.19264 E1: 2x, Output 0 E1: Output 0 0.3392 0.38528 0.6784 0.77056 E1: Output 0 0.3568 0.54112 E2: Output 1 Input 1 Encode any value in the tag, e.g., 0.5 Output 1 All outputs: 1100011 0.3568 0.504256

To verify LOW = 0.5424    (0.10001010... in binary), HIGH = 0.54816 (0.10001100... in binary). So we can send out 10001 (0.53125) Equivalent to E2E1E1E1E2 After left shift by 5 bits: LOW = (0.5424 – 0.53125) x 32 = 0.3568 HIGH = (0.54816 – 0.53125) x 32 = 0.54112 Same as the result in the last page.

Encoding Pseudo Code with E1, E2 (For floating-point implementation) Symbol Prob. CDF 1 0.8 2 0.02 0.82 3 0.18 EncodeSymbol(n) { //Update variables RANGE = HIGH - LOW; HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-1); //keep scaling before encoding next symbol while LOW, HIGH in [0, 0.5) or [0.5, 1) { send 0 for E1 and 1 for E2 scale LOW, HIGH }

Incremental Decoding Input 1100011 Decode 1: Need ≥ 5 bits (verify) Read 6 bits: Tag: 110001, 0.765625 0 0.656 0.8 0 0.8 1.0 Decode 3, E2 scaling Tag: 100011 (0.546875) 0.312 0.5424 0.54816 0.6 0.0848 0.09632 Decode 2, E2 scaling Tag: 000110 (0.09375) 0.1696 0.19264 E1: Tag: 001100 (0.1875) E1: Tag: 011000 (0.375) 0.3392 0.38528 0.6784 0.77056 E1: Tag: 110000 (0.75) Decode 1 0.3568 0.54112 E2: Tag: 100000 (0.5) Summary: Complete all possible scaling before further decoding Adjust LOW, HIGH and Tag together.

Decoding Pseudo Code with E1, E2 (For floating-point implementation) DecodeSymbol(Tag) { RANGE = HIGH - LOW; n = 1; While ( (tag - LOW) / RANGE >= CDF(n) ) { n++; } HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-1); //keep scaling before decoding next symbol while LOW, HIGH in [0, 0.5) or [0.5, 1) { scale LOW, HIGH by E1 or E2 rule Left shift Tag and read one more bit to LSB return n; Symbol Prob. CDF 1 0.8 2 0.02 0.82 3 0.18

Outline Review Integer Implementation Binary Arithmetic Coding Integer representation E3 Scaling Complete Algorithm Minimum word length Binary Arithmetic Coding Adaptive Arithmetic Coding Applications JBIG, H.264, JPEG 2000

Integer Implementation Old formulas: HIGH  LOW + RANGE * CDF(n); LOW  LOW + RANGE * CDF(n-1); Integer approximation of CDF ( ): The number of occurrence of each symbol is usually collected by a counter. Allow adaptive arithmetic coding 1 2 3 0 40 41 50 k P(k) nk Cum(k) - 1 0.8 40 2 0.02 41 3 0.18 9 50

Integer Implementation HIGH  LOW + RANGE * CDF(n); LOW  LOW + RANGE * CDF(n-1); Why + 1 in RANGE and – 1 in HIGH? HIGH should be less than the LOW of the next interval The best integer value is HIGH = (next LOW) – 1 [HIGH, HIGH + 1) still belongs to the current interval, although we could not represent it explicitly. n-1 n LOW2 LOW1 HIGH1

Example If n = 1: If n = 2: If n = 3: k P(k) Nk Cum(k) - 1 0.8 40 2 0.02 41 3 0.18 9 50 For 8-bit integers, initial LOW = 0, HIGH = 255  RANGE=256 If n = 1: If n = 2: If n = 3: 1 2 3 0 203 208 255

E1 Scaling for Integer E1 Scaling: [0, 0.5)  [0, 1), E1(x) = 2 x. LOW = 0xxxxxxx, HIGH =0xxxxxxx Output the MSB value 0, then shift left by 1 bit Important trick: Shift in 1 to HIGH and 0 to LOW HIGH: 0xxxxxxx  xxxxxxx1 HIGH = (HIGH << 1) + 1; LOW: 0xxxxxxx  xxxxxxx0 LOW = LOW << 1; Always assume HIGH ends with infinite number of 1’s: So that it approximates the LOW of the next interval. HIGH 0 . 0 x x x x x x x 1 1 1 ... LOW 0 . 0 x x x x x x x 0 0 0 ... This also ensures the RANGE is doubled after scaling: HIGH – LOW + 1  (2 x HIGH + 1 – 2 x LOW + 1) = 2(HIGH – LOW + 1)

E2 Scaling for Integer E2 Scaling: [0.5, 1)  [0, 1), E2(x) = 2 (x - 0.5) LOW = 1xxxxxxx, HIGH =1xxxxxxx Output the MSB, then shift left by 1 bit (mul by 2) Same trick: Shift in 1 to HIGH and 0 to LOW HIGH: 1xxxxxxx  xxxxxxx1 HIGH = (HIGH << 1) + 1; LOW: 1xxxxxxx  xxxxxxx0 LOW = LOW << 1; 0 . 1 x x x x x x x 1 1 1 ... 0 . 1 x x x x x x x 0 0 0 ...

Integer Encoding [0, 0.8)  LOW = 0, HIGH = 203. Can we represent an interval in [203, 204) ? (Sequence 1333333……) Input 1 0 167 203 Input 3 0 203 255 E2: Output 1 78 151 LOW: 167 (10100111) HIGH: 203 (11001011) After E2: (shift in an 1 to HIGH and 0 to LOW) LOW: 1(01001110) 78 (8-bit) HIGH: 1(10010111) 151 (8-bit) In 8.1 format (8 bits for integer, 1 bit for fractional): LOW: (10100111.0) 167 HIGH: (11001011.1) 203.5 By shifting in an 1 to HIGH, we can cover the range [203, 203.5]. The entire range [203, 204) can be covered by always shifting in 1 to HIGH.

Outline Review Integer Implementation Binary Arithmetic Coding Integer representation E3 Scaling Complete Algorithm Minimum word length Binary Arithmetic Coding Adaptive Arithmetic Coding

E3 Scaling: [0.25, 0.75)[0, 1) If RANGE straddles 1/2, E1 and E2 cannot be applied, but the range can be quite small Example: LOW=0.4999, HIGH=0.5001 Binary: LOW=0.01111…., HIGH=0.1000… We may not have enough bits to represent the interval. 1 0.5 0.25 0.75 E3 Scaling: [0.25, 0.75) [0, 1): E3(x) = 2(x - 0.25) 1 0.5

Integer Implementation of E3 Same trick: Shift in 1 to HIGH and 0 to LOW HIGH = ((HIGH – QUARTER) << 1) + 1; LOW = (LOW - QUARTER) << 1; QUARTER = 2^(M - 2) for m-bit integer. (64 for m = 8 bits) 01xxxxxx - 01000000 00xxxxxx × 2 0xxxxxx0 LOW: 10xxxxxx - 01000000 01xxxxxx × 2 1xxxxxx0 + 1 1xxxxxx1 HIGH: LOW 01xxxxxx  0xxxxxx0 HIGH 10xxxxxx  1xxxxxx1 Another way to implement E3 (Sayood book pp. 99): Left shift old LOW and HIGH, complement new MSB.

Signaling of E3 What should we send when E3 is used? Recall: we send 1 if E2 is used, send 0 if E1 is used Important relationships: (www.copro.org/download/ac_en.pdf) Apply n Ej scalings, followed by an Ei scaling. What do they mean? A series of E3 followed by an E1 is equivalent to an E1 followed by a series of E2. A series of E3 followed by an E2 is equivalent to an E2 followed by a series of E1.

Example Previous example without E3: Input 1 0 0.8 1.0 Input 3 0 0.8 1.0 Input 3 0.312 0.5424 0.54816 0.6 Input 2 0 0.656 0.8 0.0848 0.09632 E2: Output 1 0.1696 0.19264 E1: Output 0 With E3: 0.312 0.6 0.124 0.7 E3: (x-0.25)x2 Input 2 0.5848 0.59632 0.1696 0.19264 E2: Output 1 The range after E2°E3 is the same as that after E1°E2

Another View of the Equivalence Scaling of a range in [0.5, 0.75) with E1°E2 0 0.25 0.5 0.75 1 E2 E1 Equivalent scaling of the range in [0.5, 0.75) with E2°E3 0 0.25 0.5 0.75 1 E3 E2

A Simple Proof of E2°E3 = E1°E2 Given an original range r: After applying E2: [0.5, 1) [0, 1), the range becomes r1 = (r – 0.5) x 2 After applying E1: [0, 0.5)  [0, 1), the range becomes r2 = r1 x 2 = ((r – 0.5) x 2) x 2 Given the same range r: After applying E3: [0.25, 0.75)  [0, 1), the range becomes r3 = (r – 0.25) x 2 After applying E2, the range becomes r4 = (r3 – 0.5) x 2 = ((r – 0.25) x 2 – 0.5) x 2 = (r – 0.5) x 2 x 2 = r2 For formal proof: www.copro.org/download/ac_en.pdf

Encoding Operation with E3 Without E3: Input 2 0.0848 0.09632 E2: Output 1 0.312 0.5424 0.54816 0.6 0.1696 0.19264 E1: Output 0 With E3: 0.312 0.6 0.124 0.7 E3 (no output here) Input 2 0.5848 0.59632 0.1696 0.19264 E2: Output 1 Output 0 here! Don’t send anything when E3 is used, but send a 0 after E2: The bit stream is identical to that of the old method Subsequent encoding is also same because of the same final interval

Decoding for E3 Same status as the old method: low, high, range, tag. Input 1100011 Read 6 bits: Tag: 110001 (0.765625) Decode 1 0 0.656 0.8 0 0.8 1.0 Decode 3, E2 scaling Tag: 100011 (0.546875) 0.312 0.5424 0.54816 0.6 0.0848 0.09632 Decode 2, E2 scaling Tag: 000110 (0.09375) 0.1696 0.19264 E1: Tag: 001100 (0.1875) With E3: Apply E3 whenever it is possible, nothing else is needed 0.312 0.6 Tag: 100011 (0.546875) 0.5848 0.59632 0.124 0.7 E3: Tag: 100110 (0.59375) 0.1696 0.19264 Decode 2, E2 scaling Tag: 001100 (0.1875) Same status as the old method: low, high, range, tag.

Summary of Different Scalings 0 0.25 0.5 0.75 1.0 Need E1 scaling Need E2 scaling Need E3 scaling No scaling is required. Ready to encode/decode the next symbol.

Outline Review Integer Implementation Binary Arithmetic Coding Integer representation E3 Scaling Complete Algorithm Minimum word length Binary Arithmetic Coding Adaptive Arithmetic Coding

Encoding Pseudo Code with E1, E2, E3 (For integer implementation) EncodeSymbol(n) { //Update variables RANGE = HIGH - LOW + 1; HIGH = HIGH + RANGE * Cum( n ) / N - 1; LOW = LOW + RANGE * Cum(n-1) / N; //Scaling before encoding next symbol EncoderScaling(); //see next slide } Round off to integer

Encoding Pseudo Code with E1, E2, E3 EncoderScaling( ) { while (E1, E2 or E3 is possible) { if (E3 is possible) { HIGH = ((HIGH - QUARTER) << 1) + 1; LOW = (LOW - QUARTER) << 1; Scale3++; //Save number of E3, but send nothing } if (E1 or E2 is possible) { Let b=0 for E1 and b=1 for E2 send b HIGH = (HIGH << 1) + 1; LOW = (LOW << 1); while (Scale3 > 0) { //send info about E3 now send complement of b //E2 ° (E3)^n = (E1)^n ° E2 Scale3 --; //Send one bit for each E3

Decoding Pseudo Code with E1, E2, E3 (For integer implementation) Intervals: [0, 203], [204, 208], [209, 255] DecodeSymbol(Tag) { RANGE = HIGH - LOW + 1; n = 1; While (Tag > LOW + RANGE * Cum(n) / N - 1) { n++; } HIGH = LOW + RANGE * Cum(n) / N - 1; LOW = LOW + RANGE * Cum(n-1) / N; //keep scaling before decoding next symbol DecoderScaling(Tag); //next slide return n; Round off to integer: HIGH of each interval

Decoding Pseudo Code with E1, E2, E3 DecoderScaling(Tag) { while (E1, E2 or E3 is possible) { if (E1 or E2 is possible) { LOW = LOW << 1; HIGH = (HIGH << 1) + 1; Tag = Tag << 1; Tag = Tag | ReadBits(1); } if (E3 is possible) { LOW = (LOW - QUARTER) << 1; HIGH = ((HIGH - QUARTER) << 1) + 1; Tag = (Tag - QUARTER) << 1;

Integer Encoding with E1, E2, E3 Input 1 0 167 203 Input 3 0 203 255 E2: Output 1 78 151 E3: Scale3=1 Input 2 28 146 148 175 36 41 E2: Output 1 Output 0 Scale3 = 0 72 83 E1: Output 0 144 167 E1: Output 0 32 79 E2: Output 1 64 159 E1: Output 0 E3: Scale3=1 0 152 191 Output 0 Output 1 Scale3=0 Output 7 more 0’s Input 1 Final output: 11000100 10000000

Integer Decoding with E1, E2, E3 Input: 11000100 10000000 Read 8 bits: 11000100 (196) Decode 1 0 167 203 Decode 3 0 203 255 78 151 E2: Tag=10001001 (137) E3: Tag = 10010010 (146) Decode 2 28 146 148 175 36 41 E2: Tag=00100100 (36) Decode 1 Tag = LOW: stop.

Outline Review Integer Implementation Binary Arithmetic Coding Integer representation E3 Scaling Complete Algorithm Minimum word length Binary Arithmetic Coding Adaptive Arithmetic Coding

How to decide the word length m? Need to guarantee non-zero interval for all symbols in the worst case: 1 2 3 0 40 41 50 k P(k) Nk Cum(k) - 1 0.8 40 2 0.02 41 3 0.18 9 50 even when Cum(n) – cum(n -1) = 1. Otherwise HIGH < LOW. Need  RANGE cannot be too small at any time. Intuitive!

How to decide the word length m? When do we have the smallest RANGE without triggering a scaling? 0 M/4 M/2 3/4M M M = 2^m When interval is slightly larger than [M/4, M/2] or [M/2, 3/4M] None of E1, E2, and E3 can be applied Condition: 1/4 (2^m) > N Example: N = 50,  min m = 8 (1/4M=64)

Outline Review Integer Implementation Binary Arithmetic Coding Adaptive Arithmetic Coding

Binary Arithmetic Coding Arithmetic coding is slow in general: To decode a symbol, we need a seris of decisions and multiplications: While (Tag > LOW + RANGE * Cum(n) / N - 1) { n++; } The complexity is greatly reduced if we have only two symbols: 0 and 1. symbol 0 symbol 1 0 x 1 Only two intervals in [0, 1): [0, x), [x, 1)

Encoding of Binary Arithmetic Coding Prob(0)=0.6. Sequence: 0110 LOW = 0, HIGH = 1 0 0.6 1 LOW = 0, HIGH = 0.6 0 0.36 0.6 LOW = 0.36, HIGH = 0.6 0.36 0.504 0.6 LOW = 0.504, HIGH = 0.6 0.504 0.5616 0.6 LOW = 0.504, HIGH = 0.5616 Only need to update LOW or HIGH for each symbol.

Decoding of Binary Arithmetic Coding Tag 0 0.6 1 Only one decision to make: While (Tag > LOW + RANGE * Cum(n) / N - 1)) { n++; } if (Tag > LOW + RANGE * Cum(Symbol 0) / N - 1) { n = 1; } else { n = 0; }

Applications of Binary Arithmetic Coding Increasingly popular: JBIG, JBIG2, JPEG2000, H.264 Covert non-binary signals into binary first H.264: Golomb-Rice Code Bit-plane coding Various simplifications to avoid multiplication: H.264: Table look-up for RANGE * Cum(n) / N JBIG: Eliminate multiplication by assuming the RANGE is close to 1. Scale if RANGE too small.

Outline Review Integer Implementation Binary Arithmetic Coding Adaptive Arithmetic Coding

Adaptive Arithmetic Coding Observation: The partition of [0, 1) can be different from symbol to symbol The bit stream can be decoded perfectly as long as both encoder and decoder are synchronized (use the same partition). General approach: Starting from a pre-defined probability distribution Update probability after each symbol This is very difficult for Huffman coding: Has to redesign the codebook when prob changes

Example Encode 0.4667. Binary sequence: 01111 Initial counters for 0’s and 1’s: C(0)=C(1)=1.  P(0)=P(1)=0.5 0 0.3333 0.5 After encoding 0: C(0)=2, C(1)=1. P(0)=2/3, P(1)=1/3 0 0.5 1 0.3333 0.4167 0.5 After encoding 01: C(0)=2, C(1)=2. P(0)=1/2, P(1)=1/2 0.4167 0.45 0.5 After encoding 011: C(0)=2, C(1)=3. P(0)=2/5, P(1)=3/5 0.45 0.4667 0.5 After encoding 0111: C(0)=2, C(1)=4. P(0)=1/3, P(1)=2/3. Encode 0.4667.

Decoding Input 0.4667. Initial counters for 0’s and 1’s: C(0)=C(1)=1  P(0)=P(1)=0.5 Decode 0 0 0.3333 0.5 After decoding 0: C(0)=2, C(1)=1. P(0)=2/3, P(1)=1/3 Decode 1 0 0.5 1 After decoding 01: C(0)=2, C(1)=2. P(0)=1/2, P(1)=1/2 Decode 1 0.3333 0.4167 0.5 After decoding 011: C(0)=2, C(1)=3. P(0)=2/5, P(1)=3/5 Decode 1 0.4167 0.45 0.5 After decoding 0111: C(0)=2, C(1)=4. P(0)=1/3, P(1)=2/3. Decode 1 0.45 0.4667 0.5

Context-adaptive Arithmetic Coding In many cases, a sample has strong correlation with its near neighbors. Idea: Collect conditional probability distribution of a symbol for given neighboring symbols (context): P(x(n) | x(n-1), x(n-2), … x(n-k)) Use this conditional probability to encode the next symbol More skewed probability distribution can be obtained (desired by arithmetic coding) a b c b c a b c b a b c b a a b c b c a b 1-D Context template 2-D Context template

Example 0 1 1 0 1 0 1 Binary sequence: 0, 1 Neighborhood (template) size: 3 2^3=8 possible combinations (contexts) of 3 neighbors (x(n-1), x(n-2), x(n-3)). Collect frequencies of 0’s and 1’s under each context Context C(0) C(1) (0, 0, 0) 9 2 (0, 0, 1) 3 6 (0, 1, 0) 10 (0, 1, 1) … (1, 0, 0) (1, 0, 1) (1, 1, 0) (1, 1, 1) 0 1 0 1 0 1 Each symbol is coded with the probability distribution associated with its context.

Encoding Pseudo Code InitProbModel(ProbModel); While (not EOF) { n = ReadSymbol( ); //Determine the neighboring combination (context) context = GetContext(); //Encode with the corresponding probability model EncodeSymbol(ProbModel[context], n); //update probability counters for this context UpdateProb(ProbModel[context], n); }

Decoding Pseudo Code InitProbModel(ProbModel); While (not EOF) { //Determine the neighboring combination (context) context = GetContext(); //Decode with the corresponding probability model n = DecodeSymbol(ProbModel[context]); //update probability counters for this context UpdateProb(ProbModel[context], n); }

Performance of Arithmetic Coding For a sequence of length m: H(X) <= Average Bit Rate <= H(X) + 2/m Can approaches the entropy very quickly. Huffman coding: H(X) <= Average Bit Rate <= H(X) + 1/m Impractical: need to generate codewords for all sequences of length m.

Summary – Part II Arithmetic coding: Incremental Encoding and Decoding Partition the range [0, 1) recursively according to symbol probabilities. Incremental Encoding and Decoding E1, E2, E3 scaling Binary Arithmetic Coding Context Adaptive Arithmetic Coding Next: Quantization (lossy compression)