CMPT365 Multimedia Systems 1 Arithmetic Coding Additional Material Spring 2015 CMPT 365 Multimedia Systems

CMPT365 Multimedia Systems 2 Outline – Part I r Introduction r Basic Encoding and Decoding r Scaling and Incremental Coding r Integer Implementation r Adaptive Arithmetic Coding r Binary Arithmetic Coding r Applications m JBIG, H.264, JPEG 2000

CMPT365 Multimedia Systems 3 Limitations of Huffman Code r Need a probability distribution r Hard to adapt to changing statistics r Minimum codeword length is 1 bit m Serious penalty for high-probability symbols m Example: Binary source, P(0)=0.9 Entropy: -0.9*log2(0.9)-0.1*log2(0.1) = bit Huffman code: 0, 1  Avg. code length: 1 bit Joint coding is not practical for large alphabet. r Arithmetic coding: m Can resolve all of these problems. m Code a sequence of symbols without having to generate codes for all sequences of that length.

CMPT365 Multimedia Systems 4 Introduction r Recall table look-up decoding of Huffman code m N: alphabet size m L: Max codeword length m Divide [0, 2^L] into N intervals m One interval for one symbol m Interval size is roughly proportional to symbol prob. r Arithmetic coding applies this idea recursively m Normalizes the range [0, 2^L] to [0, 1]. m Map a sequence to a unique tag in [0, 1). 0 1 abcd….. dcba…..

CMPT365 Multimedia Systems 5 Arithmetic Coding r Disjoint and complete partition of the range [0, 1) [0, 0.8), [0.8, 0.82), [0.82, 1) r Each interval corresponds to one symbol r Interval size is proportional to symbol probability r Observation: once the tag falls into an interval, it never gets out of it 0 1 r The first symbol restricts the tag position to be in one of the intervals r The reduced interval is partitioned recursively as more symbols are processed a b c

CMPT365 Multimedia Systems 6 Some Questions to think about: r Why compression is achieved this way? r How to implement it efficiently? r How to decode the sequence? r Why is it better than Huffman code?

CMPT365 Multimedia Systems 7 Example: SymbolProb r Map to real line range [0, 1) r Order does not matter m Decoder need to use the same order r Disjoint but complete partition:  1: [0, 0.8):0, … 9  2: [0.8, 0.82):0.8, … 9  3: [0.82, 1): 0.82, … 9 m (Think about the impact to integer implementation)

CMPT365 Multimedia Systems 8 Range Range Encoding  Input sequence: “ 1321 ” Range 1 Final range: [0.7712, ): Encode Range Difficulties: 1. Shrinking of interval requires high precision for long sequence. 2. No output is generated until the entire sequence has been processed.

CMPT365 Multimedia Systems 9 Cumulative Density Function (CDF) X CDF Probability Mass Function X r For continuous distribution: r For discrete distribution: r Properties: m Non-decreasing m Piece-wise constant m Each segment is closed at the lower end.

CMPT365 Multimedia Systems 10 Encoder Pseudo Code low=0.0, high=1.0; while (not EOF) { n = ReadSymbol(); RANGE = HIGH - LOW; HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-1); } output LOW; InputHIGHLOWRANGE Initial *0.8= *0 = *1= *0.82= *0.82= *0.8= *0= *0.8= r Keep track of LOW, HIGH, RANGE m Any two are sufficient, e.g., LOW and RANGE.

CMPT365 Multimedia Systems 11 Decode Decode Decoding Decode Decode Receive Drawback: need to recalculate all thresholds each time.

CMPT365 Multimedia Systems Receive Decode 1 x =( ) / 0.8 = Decode 3 Simplified Decoding r Normalize RANGE to [0, 1) each time r No need to recalculate the thresholds. x =( ) / 0.18 = 0.8 Decode 2 x =( ) / 0.02 = 0 Decode 1

CMPT365 Multimedia Systems 13 Decoder Pseudo Code Low = 0; high = 1; x = Encoded_number While (x ≠ low) { n = DecodeOneSymbol(x); output symbol n; x = (x - CDF(n-1)) / (CDF(n) - CDF(n-1)); };

CMPT365 Multimedia Systems 14 Outline r Introduction r Basic Encoding and Decoding r Scaling and Incremental Coding r Integer Implementation r Adaptive Arithmetic Coding r Binary Arithmetic Coding r Applications m JBIG, H.264, JPEG 2000

CMPT365 Multimedia Systems 15 Scaling and Incremental Coding r Problems of Previous examples: m Need high precision m No output is generated until the entire sequence is encoded r Key Observation:  As the RANGE reduces, many MSB ’ s of LOW and HIGH become identical: Example: Binary form of and : , ,  We can output identical MSB ’ s and re-scale the rest:  Incremental encoding m This also allows us to achieve infinite precision with finite-precision integers.

CMPT365 Multimedia Systems 16 E1 and E2 Scaling r E1: [LOW HIGH) in [0, 0.5) m LOW: 0.0xxxxxxx (binary), m HIGH: 0.0xxxxxxx r E2: [LOW HIGH) in [0.5, 1) m LOW: 0.1xxxxxxx, m HIGH: 0.1xxxxxxx. r Output 0, then shift left by 1 bit m [0, 0.5)  [0, 1):E1(x) = 2 x r Output 1, subtract 0.5, shift left by 1 bit m [0.5, 1)  [0, 1):E2(x) = 2(x - 0.5)

CMPT365 Multimedia Systems 17 Encoding with E1 and E Input Input 3 E2: Output 1 2(x – 0.5) Input E2: Output E1: 2x, Output 0 E1: Output E1: Output E2: Output 1 Input Encode any value in the tag, e.g., 0.5 Output 1 All outputs: SymbolProb

CMPT365 Multimedia Systems 18 To verify  LOW = ( in binary), HIGH = ( in binary). r So we can send out ( ) m Equivalent to E2  E1  E1  E1  E2 r After left shift by 5 bits:  LOW = ( – ) x 32 =  HIGH = ( – ) x 32 = m Same as the result in the last page.

CMPT365 Multimedia Systems 19 Comparison with Huffman r Input Symbol 1 does not cause any output r Input Symbol 3 generates 1 bit r Input Symbol 2 generates 5 bits r Symbols with larger probabilities generates less number of bits. m Sometimes no bit is generated at all  Advantage over Huffman coding r Large probabilities are desired in arithmetic coding m Can use context-adaptive method to create larger probability and to improve compression ratio. SymbolProb r Note: Complete all possible scaling before encoding the next symbol

CMPT365 Multimedia Systems 20 Incremental Decoding Decode 1: Need ≥ 5 bits (verify) Read 6 bits: Tag: , Input Decode 3, E2 scaling Tag: ( ) Decode 2, E2 scaling Tag: ( ) E1: Tag: (0.1875) E1: Tag: (0.375) E1: Tag: (0.75) Decode E2: Tag: (0.5) r Summary: Complete all possible scaling before further decoding Adjust LOW, HIGH and Tag together.

CMPT365 Multimedia Systems 21 Summary – Part I r Introduction r Encoding and Decoding r Scaling and Incremental Coding m E1, E2 r Next: m Integer Implementation E3 scaling m Adaptive Arithmetic Coding m Binary Arithmetic Coding m Applications JBIG, H.264, JPEG 2000

CMPT365 Multimedia Systems 22 Outline – Part II r Review r Integer Implementation m Integer representation m E3 Scaling m Minimum word length r Binary Arithmetic Coding r Adaptive Arithmetic Coding

CMPT365 Multimedia Systems 23 Range Range Encoding Without Scaling  Input sequence: “ 1321 ” Range 1 Final range: [0.7712, ): Encode Range

CMPT365 Multimedia Systems 24 E1 and E2 Scaling r E1: [LOW HIGH) in [0, 0.5) m LOW: 0.0xxxxxxx (binary), m HIGH: 0.0xxxxxxx r E2: [LOW HIGH) in [0.5, 1) m LOW: 0.1xxxxxxx, m HIGH: 0.1xxxxxxx. r Output 0, then shift left by 1 bit m [0, 0.5)  [0, 1):E1(x) = 2 x r Output 1, subtract 0.5, shift left by 1 bit m [0.5, 1)  [0, 1):E2(x) = 2(x - 0.5)

CMPT365 Multimedia Systems 25 Encoding with E1 and E Input Input 3 E2: Output 1 2(x – 0.5) Input E2: Output E1: 2x, Output 0 E1: Output E1: Output E2: Output 1 Input Encode any value in the tag, e.g., 0.5 Output 1 All outputs: SymbolProb

CMPT365 Multimedia Systems 26 To verify  LOW = ( in binary), HIGH = ( in binary). r So we can send out ( ) m Equivalent to E2  E1  E1  E1  E2 r After left shift by 5 bits:  LOW = ( – ) x 32 =  HIGH = ( – ) x 32 = m Same as the result in the last page.

CMPT365 Multimedia Systems 27 Encoding Pseudo Code with E1, E2 EncodeSymbol(n) { //Update variables RANGE = HIGH - LOW; HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-1); //keep scaling before encoding next symbol while LOW, HIGH in [0, 0.5) or [0.5, 1) { send 0 for E1 and 1 for E2 scale LOW, HIGH } SymbolProb.CDF (For floating-point implementation)

CMPT365 Multimedia Systems 28 Incremental Decoding Decode 1: Need ≥ 5 bits (verify) Read 6 bits: Tag: , Input Decode 3, E2 scaling Tag: ( ) Decode 2, E2 scaling Tag: ( ) E1: Tag: (0.1875) E1: Tag: (0.375) E1: Tag: (0.75) Decode E2: Tag: (0.5) r Summary: Complete all possible scaling before further decoding Adjust LOW, HIGH and Tag together.

CMPT365 Multimedia Systems 29 Decoding Pseudo Code with E1, E2 DecodeSymbol(Tag) { RANGE = HIGH - LOW; n = 1; While ( (tag - LOW) / RANGE >= CDF(n) ) { n++; } HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-1); //keep scaling before decoding next symbol while LOW, HIGH in [0, 0.5) or [0.5, 1) { scale LOW, HIGH by E1 or E2 rule Left shift Tag and read one more bit to LSB } return n; } SymbolProb.CDF (For floating-point implementation)

CMPT365 Multimedia Systems 30 Outline r Review r Integer Implementation m Integer representation m E3 Scaling m Complete Algorithm m Minimum word length r Binary Arithmetic Coding r Adaptive Arithmetic Coding r Applications m JBIG, H.264, JPEG 2000

CMPT365 Multimedia Systems 31 Integer Implementation r Old formulas: HIGH  LOW + RANGE * CDF(n); LOW  LOW + RANGE * CDF(n-1); kP( k )nknk Cum(k) r Integer approximation of CDF ( ): m The number of occurrence of each symbol is usually collected by a counter. m Allow adaptive arithmetic coding

CMPT365 Multimedia Systems 32 Integer Implementation HIGH  LOW + RANGE * CDF(n); LOW  LOW + RANGE * CDF(n-1);  Why + 1 in RANGE and – 1 in HIGH? m HIGH should be less than the LOW of the next interval  The best integer value is HIGH = (next LOW) – 1  [HIGH, HIGH + 1) still belongs to the current interval, although we could not represent it explicitly. n-1 n LOW2 LOW1 0 HIGH1

CMPT365 Multimedia Systems 33 Example kP( k )NkNk Cum(k) r For 8-bit integers, initial LOW = 0, HIGH = 255  RANGE= If n = 1:If n = 2:If n = 3:

CMPT365 Multimedia Systems 34 E1 Scaling for Integer r E1 Scaling: [0, 0.5)  [0, 1), E1(x) = 2 x. m LOW = 0xxxxxxx,HIGH =0xxxxxxx m Output the MSB value 0, then shift left by 1 bit r Important trick: Shift in 1 to HIGH and 0 to LOW r HIGH: 0xxxxxxx  xxxxxxx1HIGH = (HIGH << 1) + 1; r LOW: 0xxxxxxx  xxxxxxx0LOW = LOW << 1; HIGH 0. 0 x x x x x x x LOW 0. 0 x x x x x x x  Always assume HIGH ends with infinite number of 1 ’ s: m So that it approximates the LOW of the next interval. r This also ensures the RANGE is doubled after scaling:  HIGH – LOW + 1  (2 x HIGH + 1 – 2 x LOW + 1) = 2(HIGH – LOW + 1)

CMPT365 Multimedia Systems 35 E2 Scaling for Integer r E2 Scaling: [0.5, 1)  [0, 1), E2(x) = 2 (x - 0.5) m LOW = 1xxxxxxx,HIGH =1xxxxxxx m Output the MSB, then shift left by 1 bit (mul by 2) r Same trick: Shift in 1 to HIGH and 0 to LOW r HIGH: 1xxxxxxx  xxxxxxx1 HIGH = (HIGH << 1) + 1; r LOW: 1xxxxxxx  xxxxxxx0LOW = LOW << 1; 0. 1 x x x x x x x x x x x x x x

CMPT365 Multimedia Systems Input Input 3 E2: Output Integer Encoding LOW: 167 ( )HIGH: 203 ( ) After E2: (shift in an 1 to HIGH and 0 to LOW) LOW: 1( ) 78 (8-bit)HIGH: 1( ) 151 (8-bit) In 8.1 format (8 bits for integer, 1 bit for fractional): LOW: ( ) 167HIGH: ( )  By shifting in an 1 to HIGH, we can cover the range [203, 203.5].  The entire range [203, 204) can be covered by always shifting in 1 to HIGH. [0, 0.8)  LOW = 0, HIGH = 203. [0.8, 0.82)  LOW = 204, HIGH = 208. Can we represent an interval in [203, 204) ? (Sequence ……)

CMPT365 Multimedia Systems 37 Outline r Review r Integer Implementation m Integer representation m E3 Scaling m Complete Algorithm m Minimum word length r Binary Arithmetic Coding r Adaptive Arithmetic Coding

CMPT365 Multimedia Systems 38 E3 Scaling: [0.25, 0.75)  [0, 1) r If RANGE straddles 1/2, E1 and E2 cannot be applied, but the range can be quite small m Example: LOW=0.4999, HIGH= Binary: LOW= …., HIGH= … m We may not have enough bits to represent the interval r E3 Scaling: [0.25, 0.75)  [0, 1): E3(x) = 2(x )

CMPT365 Multimedia Systems 39 Integer Implementation of E3 01xxxxxx xxxxxx × 2 0xxxxxx0 LOW: 10xxxxxx xxxxxx × 2 1xxxxxx xxxxxx1 HIGH:  Another way to implement E3 (Sayood book pp. 99): m Left shift old LOW and HIGH, complement new MSB. Same trick: Shift in 1 to HIGH and 0 to LOW HIGH = ((HIGH – QUARTER) << 1) + 1; LOW = (LOW - QUARTER) << 1; QUARTER = 2^(M - 2) for m-bit integer. (64 for m = 8 bits) LOW 01xxxxxx  0xxxxxx0 HIGH 10xxxxxx  1xxxxxx1

CMPT365 Multimedia Systems 40 Signaling of E3 r What should we send when E3 is used? m Recall: we send 1 if E2 is used, send 0 if E1 is used r What do they mean? m A series of E3 followed by an E1 is equivalent to an E1 followed by a series of E2. m A series of E3 followed by an E2 is equivalent to an E2 followed by a series of E1. r Important relationships: ( Apply n Ej scalings, followed by an Ei scaling.

CMPT365 Multimedia Systems 41 Example Input Input Input E2: Output E1: Output 0 r Previous example without E3: r With E3: E3: (x-0.25)x E2: Output 1 r The range after E2°E3 is the same as that after E1°E2 Input

CMPT365 Multimedia Systems 42 Another View of the Equivalence r Scaling of a range in [0.5, 0.75) with E1°E2 r Equivalent scaling of the range in [0.5, 0.75) with E2°E3 E2 E1 E3 E2

CMPT365 Multimedia Systems 43 A Simple Proof of E2 ° E3 = E1 ° E2 r Given the same range r: m After applying E3: [0.25, 0.75)  [0, 1), the range becomes r3 = (r – 0.25) x 2 m After applying E2, the range becomes r4 = (r3 – 0.5) x 2 = ((r – 0.25) x 2 – 0.5) x 2 = (r – 0.5) x 2 x 2 = r2 r Given an original range r: m After applying E2: [0.5, 1)  [0, 1), the range becomes r1 = (r – 0.5) x 2 m After applying E1: [0, 0.5)  [0, 1), the range becomes r2 = r1 x 2 = ((r – 0.5) x 2) x 2 For formal proof:

CMPT365 Multimedia Systems 44 Encoding Operation with E Input E2: Output E1: Output 0 r With E3: E3 (no output here) E2: Output 1 Input r Without E3:  Don ’ t send anything when E3 is used, but send a 0 after E2: m The bit stream is identical to that of the old method m Subsequent encoding is also same because of the same final interval Output 0 here!

CMPT365 Multimedia Systems 45 Decoding for E Input Read 6 bits: Tag: ( ) Decode Decode 3, E2 scaling Tag: ( ) Decode 2, E2 scaling Tag: ( ) E1: Tag: (0.1875) r With E3: Apply E3 whenever it is possible, nothing else is needed Tag: ( ) E3: Tag: ( ) Decode 2, E2 scaling Tag: (0.1875) Same status as the old method: low, high, range, tag.

CMPT365 Multimedia Systems 46 Summary of Different Scalings Need E1 scaling Need E2 scaling Need E3 scaling No scaling is required. Ready to encode/decode the next symbol

CMPT365 Multimedia Systems 47 Outline r Review r Integer Implementation m Integer representation m E3 Scaling m Complete Algorithm m Minimum word length r Binary Arithmetic Coding r Adaptive Arithmetic Coding

CMPT365 Multimedia Systems 48 Encoding Pseudo Code with E1, E2, E3 EncodeSymbol(n) { //Update variables RANGE = HIGH - LOW + 1; HIGH = HIGH + RANGE * Cum( n ) / N - 1; LOW = LOW + RANGE * Cum(n-1) / N; //Scaling before encoding next symbol EncoderScaling();//see next slide } (For integer implementation) Round off to integer

CMPT365 Multimedia Systems 49 Encoding Pseudo Code with E1, E2, E3 EncoderScaling ( ) { while (E1, E2 or E3 is possible) { if (E3 is possible) { HIGH = ((HIGH - QUARTER) << 1) + 1; LOW = (LOW - QUARTER) << 1; Scale3++; //Save number of E3, but send nothing } if (E1 or E2 is possible) { Let b=0 for E1 and b=1 for E2 send b HIGH = (HIGH << 1) + 1; LOW = (LOW << 1); while (Scale3 > 0) { //send info about E3 now send complement of b // E2 ° ( E3)^n = (E1)^n ° E2 Scale3 --; //Send one bit for each E3 }

CMPT365 Multimedia Systems 50 DecodeSymbol(Tag) { RANGE = HIGH - LOW + 1; n = 1; While (Tag > LOW + RANGE * Cum(n) / N - 1) { n++; } HIGH = LOW + RANGE * Cum(n) / N - 1; LOW = LOW + RANGE * Cum(n-1) / N; //keep scaling before decoding next symbol DecoderScaling(Tag);//next slide return n; } Decoding Pseudo Code with E1, E2, E3 (For integer implementation) Intervals: [0, 203], [204, 208], [209, 255] Round off to integer: HIGH of each interval

CMPT365 Multimedia Systems 51 Decoding Pseudo Code with E1, E2, E3 DecoderScaling (Tag) { while (E1, E2 or E3 is possible) { if (E1 or E2 is possible) { LOW = LOW << 1; HIGH = (HIGH << 1) + 1; Tag = Tag << 1; Tag = Tag | ReadBits(1); } if (E3 is possible) { LOW = (LOW - QUARTER) << 1; HIGH = ((HIGH - QUARTER) << 1) + 1; Tag = (Tag - QUARTER) << 1; Tag = Tag | ReadBits(1); }

CMPT365 Multimedia Systems Input Input 3 E2: Output E3: Scale3=1 Input E1: Output E2: Output E1: Output E2: Output 1 Output 0 Scale3 = E1: Output 0 Integer Encoding with E1, E2, E3 E3: Scale3= Output 0 Output 1 Scale3=0 Output 7 more 0’s Input 1 Final output:

CMPT365 Multimedia Systems Integer Decoding with E1, E2, E3 Read 8 bits: (196) Decode Decode 3 Input: E3: Tag = (146) Decode E2: Tag= (137) E2: Tag= (36) Decode 1 Tag = LOW: stop.

CMPT365 Multimedia Systems 54 Outline r Review r Integer Implementation m Integer representation m E3 Scaling m Complete Algorithm m Minimum word length r Binary Arithmetic Coding r Adaptive Arithmetic Coding

CMPT365 Multimedia Systems 55 How to decide the word length m? r Need to guarantee non-zero interval for all symbols in the worst case: kP( k )NkNk Cum(k) even when Cum( n ) – cum( n -1) = 1. Otherwise HIGH < LOW. Need  RANGE cannot be too small at any time. Intuitive!

CMPT365 Multimedia Systems 56 How to decide the word length m? r Condition: 1/4 (2^m) > N r Example: N = 50,  min m = 8 (1/4M=64) r When do we have the smallest RANGE without triggering a scaling? 0 M/4 M/2 3/4M M M = 2^m m When interval is slightly larger than [M/4, M/2] or [M/2, 3/4M] m None of E1, E2, and E3 can be applied

CMPT365 Multimedia Systems 57 Outline r Review r Integer Implementation r Binary Arithmetic Coding r Adaptive Arithmetic Coding

CMPT365 Multimedia Systems 58 Binary Arithmetic Coding r Arithmetic coding is slow in general: To decode a symbol, we need a seris of decisions and multiplications: While (Tag > LOW + RANGE * Cum(n) / N - 1) { n++; } r The complexity is greatly reduced if we have only two symbols: 0 and 1. symbol 0 symbol 1 r Only two intervals in [0, 1): [0, x), [x, 1) 0 x 1

CMPT365 Multimedia Systems 59 Encoding of Binary Arithmetic Coding Prob(0)=0.6. Sequence: 0110 LOW = 0, HIGH = LOW = 0, HIGH = LOW = 0.36, HIGH = LOW = 0.504, HIGH = LOW = 0.504, HIGH = Only need to update LOW or HIGH for each symbol.

CMPT365 Multimedia Systems 60 Decoding of Binary Arithmetic Coding Tag While (Tag > LOW + RANGE * Cum(n) / N - 1)) { n++; } Only one decision to make: if (Tag > LOW + RANGE * Cum(Symbol 0) / N - 1) { n = 1; } else { n = 0; }

CMPT365 Multimedia Systems 61 Applications of Binary Arithmetic Coding r Increasingly popular: m JBIG, JBIG2, JPEG2000, H.264 r Covert non-binary signals into binary first m H.264: Golomb-Rice Code m Bit-plane coding r Various simplifications to avoid multiplication: m H.264: Table look-up for RANGE * Cum(n) / N m JBIG: Eliminate multiplication by assuming the RANGE is close to 1. Scale if RANGE too small.

CMPT365 Multimedia Systems 62 Outline r Review r Integer Implementation r Binary Arithmetic Coding r Adaptive Arithmetic Coding

CMPT365 Multimedia Systems 63 Adaptive Arithmetic Coding r Observation: The partition of [0, 1) can be different from symbol to symbol r The bit stream can be decoded perfectly as long as both encoder and decoder are synchronized (use the same partition). r General approach: m Starting from a pre-defined probability distribution m Update probability after each symbol r This is very difficult for Huffman coding: m Has to redesign the codebook when prob changes

CMPT365 Multimedia Systems 64 Example r Binary sequence:  Initial counters for 0 ’ s and 1 ’ s: C(0)=C(1)=1.  P(0)=P(1)= r After encoding 0: C(0)=2, C(1)=1.  P(0)=2/3, P(1)=1/ r After encoding 01: C(0)=2, C(1)=2.  P(0)=1/2, P(1)=1/ r After encoding 011: C(0)=2, C(1)=3.  P(0)=2/5, P(1)=3/ r After encoding 0111: C(0)=2, C(1)=4.  P(0)=1/3, P(1)=2/3. r Encode

CMPT365 Multimedia Systems 65 Decoding r Input  Initial counters for 0 ’ s and 1 ’ s: C(0)=C(1)=1  P(0)=P(1)=0.5 Decode r After decoding 0: C(0)=2, C(1)=1.  P(0)=2/3, P(1)=1/3 Decode 1 r After decoding 01: C(0)=2, C(1)=2.  P(0)=1/2, P(1)=1/2 Decode r After decoding 011: C(0)=2, C(1)=3.  P(0)=2/5, P(1)=3/5 Decode r After decoding 0111: C(0)=2, C(1)=4.  P(0)=1/3, P(1)=2/3. Decode

CMPT365 Multimedia Systems 66 Context-adaptive Arithmetic Coding r In many cases, a sample has strong correlation with its near neighbors. r Idea: m Collect conditional probability distribution of a symbol for given neighboring symbols (context): P(x(n) | x(n-1), x(n-2), … x(n-k)) m Use this conditional probability to encode the next symbol m More skewed probability distribution can be obtained (desired by arithmetic coding) a b c b c a b 1-D Context template a b c b c a b c b a b c b a 2-D Context template

CMPT365 Multimedia Systems 67 Example r Binary sequence: 0, 1 r Neighborhood (template) size: 3  2^3=8 possible combinations (contexts) of 3 neighbors (x(n-1), x(n-2), x(n-3)).  Collect frequencies of 0 ’ s and 1 ’ s under each context ContextC(0)C(1) (0, 0, 0)92 (0, 0, 1)36 (0, 1, 0)10 (0, 1, 1) …… (1, 0, 0) …… (1, 0, 1) …… (1, 1, 0) …… (1, 1, 1) …… 0 1 Each symbol is coded with the probability distribution associated with its context.

CMPT365 Multimedia Systems 68 Encoding Pseudo Code InitProbModel(ProbModel); While (not EOF) { n = ReadSymbol( ); //Determine the neighboring combination (context) context = GetContext(); //Encode with the corresponding probability model EncodeSymbol(ProbModel[context], n); //update probability counters for this context UpdateProb(ProbModel[context], n); }

CMPT365 Multimedia Systems 69 Decoding Pseudo Code InitProbModel(ProbModel); While (not EOF) { //Determine the neighboring combination (context) context = GetContext(); //Decode with the corresponding probability model n = DecodeSymbol(ProbModel[context]); //update probability counters for this context UpdateProb(ProbModel[context], n); }

CMPT365 Multimedia Systems 70 Performance of Arithmetic Coding r For a sequence of length m: r H(X) <= Average Bit Rate <= H(X) + 2/m m Can approaches the entropy very quickly. r Huffman coding: m H(X) <= Average Bit Rate <= H(X) + 1/m m Impractical: need to generate codewords for all sequences of length m.

CMPT365 Multimedia Systems 71 Summary – Part II r Arithmetic coding: m Partition the range [0, 1) recursively according to symbol probabilities. r Incremental Encoding and Decoding m E1, E2, E3 scaling r Binary Arithmetic Coding r Context Adaptive Arithmetic Coding r Next: Quantization (lossy compression)