1. CMPT 365 Multimedia Systems
Arithmetic Coding Additional Material Fall 2012

2. Outline – Part I Introduction Basic Encoding and Decoding
Scaling and Incremental Coding
Integer Implementation
Adaptive Arithmetic Coding
Binary Arithmetic Coding
Applications
JBIG, H.264, JPEG 2000

3. Limitations of Huffman Code
Need a probability distribution
Hard to adapt to changing statistics
Minimum codeword length is 1 bit
Serious penalty for high-probability symbols
Example: Binary source, P(0)=0.9
Entropy: -0.9*log2(0.9)-0.1*log2(0.1) = bit
Huffman code: 0, 1  Avg. code length: 1 bit
Joint coding is not practical for large alphabet.
Arithmetic coding:
Can resolve all of these problems.
Code a sequence of symbols without having to generate codes for all sequences of that length.

4. Introduction abcd….. dcba…..
Recall table look-up decoding of Huffman code
N: alphabet size
L: Max codeword length
Divide [0, 2^L] into N intervals
One interval for one symbol
Interval size is roughly
proportional to symbol prob. 1 00
Arithmetic coding applies this idea recursively
Normalizes the range [0, 2^L] to [0, 1].
Map a sequence to a unique tag in [0, 1).
abcd…..
dcba…..

5. Arithmetic Coding Disjoint and complete partition of the range [0, 1)1
a b c
Disjoint and complete partition of the range [0, 1)
[0, 0.8), [0.8, 0.82), [0.82, 1)
Each interval corresponds to one symbol
Interval size is proportional to symbol probability
The first symbol restricts the tag
position to be in one of the intervals
The reduced interval is partitioned
recursively as more symbols are
processed. 1 1
Observation: once the tag falls into an interval, it never gets out of it

6. Some Questions to think about:
Why compression is achieved this way?
How to implement it efficiently?
How to decode the sequence?
Why is it better than Huffman code?

7. Example: 1 0.8 2 0.02 3 0.18 Map to real line range [0, 1)
Symbol
Prob. 1
0.8 2
0.02 3
0.18
Map to real line range [0, 1)
Order does not matter
Decoder need to use the same order
Disjoint but complete partition:
1: [0, 0.8): 0, …9
2: [0.8, 0.82): 0.8, …9
3: [0.82, 1): , …9
(Think about the impact to integer implementation)

8. Encoding Input sequence: “1321”
Range 1
Range 0.8
Range 0.144
Range
Final range: [0.7712, ): Encode
Difficulties: 1. Shrinking of interval requires high precision for long sequence.
2. No output is generated until the entire sequence has been processed.

9. Cumulative Density Function (CDF)
Probability Mass Function X
0.4
0.2
For continuous distribution:
For discrete distribution: X
CDF
0.2
0.4
0.8
1.0
Properties:
Non-decreasing
Piece-wise constant
Each segment is closed at the lower end.

10. Encoder Pseudo Code Keep track of LOW, HIGH, RANGE
while (not EOF) {
n = ReadSymbol();
RANGE = HIGH - LOW;
HIGH = LOW + RANGE * CDF(n);
LOW = LOW + RANGE * CDF(n-1); }
output LOW;
Keep track of LOW, HIGH, RANGE
Any two are sufficient, e.g., LOW and RANGE.
Input
HIGH
LOW
RANGE
Initial
1.0
0.0 1
*0.8=0.8
*0 = 0.0
0.8 3
*1=0.8
*0.82=0.656
0.144 2
*0.82=
*0.8=0.7712
*0=0.7712
*0.8=

11. Decoding Drawback: need to recalculate all thresholds each time.
Receive
Decode 1
Decode 3
Decode 2
Decode 1
Drawback: need to recalculate all thresholds each time.

12. Simplified Decoding Normalize RANGE to [0, 1) each time
No need to recalculate the thresholds.
Receive
Decode 1
x =( ) / 0.8
= 0.964
Decode 3
x =( ) / 0.18
= 0.8
Decode 2
x =( ) / 0.02
= 0
Decode 1

13. Decoder Pseudo Code Low = 0; high = 1; x = Encoded_number
While (x ≠ low) {
n = DecodeOneSymbol(x);
output symbol n;
x = (x - CDF(n-1)) / (CDF(n) - CDF(n-1)); };

14. Outline Introduction Basic Encoding and Decoding
Scaling and Incremental Coding
Integer Implementation
Adaptive Arithmetic Coding
Binary Arithmetic Coding
Applications
JBIG, H.264, JPEG 2000

15. Scaling and Incremental Coding
Problems of Previous examples:
Need high precision
No output is generated until the entire sequence is encoded
Key Observation:
As the RANGE reduces, many MSB’s of LOW and HIGH become identical:
Example: Binary form of and :
, ,
We can output identical MSB’s and re-scale the rest:
 Incremental encoding
This also allows us to achieve infinite precision with finite-precision integers.

16. E1 and E2 Scaling E1: [LOW HIGH) in [0, 0.5)
LOW: 0.0xxxxxxx (binary),
HIGH: 0.0xxxxxxx.
Output 0, then shift left by 1 bit
[0, 0.5) [0, 1): E1(x) = 2 x
E2: [LOW HIGH) in [0.5, 1)
LOW: 0.1xxxxxxx,
HIGH: 0.1xxxxxxx.
Output 1, subtract 0.5,
shift left by 1 bit
[0.5, 1) [0, 1): E2(x) = 2(x - 0.5)

17. Encoding with E1 and E2 Input 1 0 0.8 1.0 Input 3 0 0.656 0.8
Symbol
Prob. 1
0.8 2
0.02 3
0.18
Input 1
Input 3
E2: Output 1
2(x – 0.5)
Input 2
E2: Output 1
E1: 2x, Output 0
E1: Output 0
E1: Output 0
E2: Output 1
Input 1
Encode any value
in the tag, e.g., 0.5
Output 1
All outputs:

18. To verify
LOW =     ( in binary), HIGH = ( in binary).
So we can send out ( )
Equivalent to E2E1E1E1E2
After left shift by 5 bits:
LOW = ( – ) x 32 =
HIGH = ( – ) x 32 =
Same as the result in the last page.

19. Comparison with Huffman
Note: Complete all possible scaling before
encoding the next symbol
Symbol
Prob. 1
0.8 2
0.02 3
0.18
Comparison with Huffman
Input Symbol 1 does not cause any output
Input Symbol 3 generates 1 bit
Input Symbol 2 generates 5 bits
Symbols with larger probabilities generates less number of bits.
Sometimes no bit is generated at all
 Advantage over Huffman coding
Large probabilities are desired in arithmetic coding
Can use context-adaptive method to create larger probability
and to improve compression ratio.

20. Incremental Decoding Input 1100011 Decode 1: Need ≥ 5 bits (verify)
Read 6 bits:
Tag: ,
Decode 3, E2 scaling
Tag: ( )
Decode 2, E2 scaling
Tag: ( )
E1: Tag: (0.1875)
E1: Tag: (0.375)
E1: Tag: (0.75)
Decode 1
E2: Tag: (0.5)
Summary: Complete all possible scaling before further decoding
Adjust LOW, HIGH and Tag together.

21. Summary – Part I Introduction Encoding and Decoding
Scaling and Incremental Coding
E1, E2
Next:
Integer Implementation
E3 scaling
Adaptive Arithmetic Coding
Binary Arithmetic Coding
Applications
JBIG, H.264, JPEG 2000

22. Outline – Part II Review Integer Implementation
Integer representation
E3 Scaling
Minimum word length
Binary Arithmetic Coding
Adaptive Arithmetic Coding

23. Encoding Without Scaling
Range 1
Range 0.8
Range 0.144
Range
Input sequence: “1321”
Final range: [0.7712, ): Encode

24. E1 and E2 Scaling E1: [LOW HIGH) in [0, 0.5)
LOW: 0.0xxxxxxx (binary),
HIGH: 0.0xxxxxxx.
Output 0, then shift left by 1 bit
[0, 0.5) [0, 1): E1(x) = 2 x
E2: [LOW HIGH) in [0.5, 1)
LOW: 0.1xxxxxxx,
HIGH: 0.1xxxxxxx.
Output 1, subtract 0.5,
shift left by 1 bit
[0.5, 1) [0, 1): E2(x) = 2(x - 0.5)

25. Encoding with E1 and E2 Input 1 0 0.8 1.0 Input 3 0 0.656 0.8
Symbol
Prob. 1
0.8 2
0.02 3
0.18
Input 1
Input 3
E2: Output 1
2(x – 0.5)
Input 2
E2: Output 1
E1: 2x, Output 0
E1: Output 0
E1: Output 0
E2: Output 1
Input 1
Encode any value
in the tag, e.g., 0.5
Output 1
All outputs:

26. To verify
LOW =     ( in binary), HIGH = ( in binary).
So we can send out ( )
Equivalent to E2E1E1E1E2
After left shift by 5 bits:
LOW = ( – ) x 32 =
HIGH = ( – ) x 32 =
Same as the result in the last page.

27. Encoding Pseudo Code with E1, E2
(For floating-point implementation)
Symbol
Prob.
CDF 1
0.8 2
0.02
0.82 3
0.18
EncodeSymbol(n) {
//Update variables
RANGE = HIGH - LOW;
HIGH = LOW + RANGE * CDF(n);
LOW = LOW + RANGE * CDF(n-1);
//keep scaling before encoding next symbol
while LOW, HIGH in [0, 0.5) or [0.5, 1) {
send 0 for E1 and 1 for E2
scale LOW, HIGH }

28. Incremental Decoding Input 1100011 Decode 1: Need ≥ 5 bits (verify)
Read 6 bits:
Tag: ,
Decode 3, E2 scaling
Tag: ( )
Decode 2, E2 scaling
Tag: ( )
E1: Tag: (0.1875)
E1: Tag: (0.375)
E1: Tag: (0.75)
Decode 1
E2: Tag: (0.5)
Summary: Complete all possible scaling before further decoding
Adjust LOW, HIGH and Tag together.

29. Decoding Pseudo Code with E1, E2
(For floating-point implementation)
DecodeSymbol(Tag) {
RANGE = HIGH - LOW;
n = 1;
While ( (tag - LOW) / RANGE >= CDF(n) ) {
n++; }
HIGH = LOW + RANGE * CDF(n);
LOW = LOW + RANGE * CDF(n-1);
//keep scaling before decoding next symbol
while LOW, HIGH in [0, 0.5) or [0.5, 1) {
scale LOW, HIGH by E1 or E2 rule
Left shift Tag and read one more bit to LSB
return n;
Symbol
Prob.
CDF 1
0.8 2
0.02
0.82 3
0.18

30. Outline Review Integer Implementation Binary Arithmetic Coding
Integer representation
E3 Scaling
Complete Algorithm
Minimum word length
Binary Arithmetic Coding
Adaptive Arithmetic Coding
Applications
JBIG, H.264, JPEG 2000

31. Integer Implementation
Old formulas:
HIGH  LOW + RANGE * CDF(n);
LOW  LOW + RANGE * CDF(n-1);
Integer approximation of CDF ( ):
The number of occurrence of each symbol is usually collected by a counter.
Allow adaptive arithmetic coding k
P(k) nk
Cum(k) - 1
0.8 40 2
0.02 41 3
0.18 9 50

32. Integer Implementation
HIGH  LOW + RANGE * CDF(n);
LOW  LOW + RANGE * CDF(n-1);
Why + 1 in RANGE and – 1 in HIGH?
HIGH should be less than the LOW of the next interval
The best integer value is HIGH = (next LOW) – 1
[HIGH, HIGH + 1) still belongs to the current interval,
although we could not represent it explicitly.
n-1 n
LOW2
LOW1
HIGH1

33. Example If n = 1: If n = 2: If n = 3:k
P(k) Nk
Cum(k) - 1
0.8 40 2
0.02 41 3
0.18 9 50
For 8-bit integers, initial LOW = 0, HIGH = 255  RANGE=256
If n = 1:
If n = 2:
If n = 3:

34. E1 Scaling for Integer E1 Scaling: [0, 0.5)  [0, 1), E1(x) = 2 x.
LOW = 0xxxxxxx, HIGH =0xxxxxxx
Output the MSB value 0, then shift left by 1 bit
Important trick: Shift in 1 to HIGH and 0 to LOW
HIGH: 0xxxxxxx  xxxxxxx1 HIGH = (HIGH << 1) + 1;
LOW: 0xxxxxxx  xxxxxxx0 LOW = LOW << 1;
Always assume HIGH ends with infinite number of 1’s:
So that it approximates the LOW of the next interval.
HIGH x x x x x x x
LOW x x x x x x x
This also ensures the RANGE is doubled after scaling:
HIGH – LOW + 1  (2 x HIGH + 1 – 2 x LOW + 1) = 2(HIGH – LOW + 1)

35. E2 Scaling for Integer
E2 Scaling: [0.5, 1)  [0, 1), E2(x) = 2 (x - 0.5)
LOW = 1xxxxxxx, HIGH =1xxxxxxx
Output the MSB, then shift left by 1 bit (mul by 2)
Same trick: Shift in 1 to HIGH and 0 to LOW
HIGH: 1xxxxxxx  xxxxxxx1 HIGH = (HIGH << 1) + 1;
LOW: 1xxxxxxx  xxxxxxx0 LOW = LOW << 1;
0 . 1 x x x x x x x
0 . 1 x x x x x x x

36. Integer Encoding [0, 0.8)  LOW = 0, HIGH = 203.
Can we represent an interval in [203, 204) ? (Sequence ……)
Input 1
Input 3
E2: Output 1
LOW: 167 ( ) HIGH: 203 ( )
After E2: (shift in an 1 to HIGH and 0 to LOW)
LOW: 1( ) 78 (8-bit) HIGH: 1( ) 151 (8-bit)
In 8.1 format (8 bits for integer, 1 bit for fractional):
LOW: ( ) 167 HIGH: ( )
By shifting in an 1 to HIGH, we can cover the range [203, 203.5].
The entire range [203, 204) can be covered by always shifting in 1 to HIGH.

37. Outline Review Integer Implementation Binary Arithmetic Coding
Integer representation
E3 Scaling
Complete Algorithm
Minimum word length
Binary Arithmetic Coding
Adaptive Arithmetic Coding

38. E3 Scaling: [0.25, 0.75)[0, 1)
If RANGE straddles 1/2, E1 and E2 cannot be applied,
but the range can be quite small
Example: LOW=0.4999, HIGH=0.5001
Binary: LOW= …., HIGH=0.1000…
We may not have enough bits to represent the interval. 1
0.5
0.25
0.75
E3 Scaling:
[0.25, 0.75) [0, 1):
E3(x) = 2(x ) 1
0.5

39. Integer Implementation of E3
Same trick: Shift in 1 to HIGH and 0 to LOW
HIGH = ((HIGH – QUARTER) << 1) + 1;
LOW = (LOW - QUARTER) << 1;
QUARTER = 2^(M - 2) for m-bit integer. (64 for m = 8 bits)
01xxxxxx
00xxxxxx ×
0xxxxxx0
LOW:
10xxxxxx
01xxxxxx ×
1xxxxxx0
1xxxxxx1
HIGH:
LOW 01xxxxxx  0xxxxxx0
HIGH 10xxxxxx  1xxxxxx1
Another way to implement E3 (Sayood book pp. 99):
Left shift old LOW and HIGH, complement new MSB.

40. Signaling of E3 What should we send when E3 is used?
Recall: we send 1 if E2 is used, send 0 if E1 is used
Important relationships: (
Apply n Ej scalings, followed by an Ei scaling.
What do they mean?
A series of E3 followed by an E1 is equivalent to an E1 followed by a series of E2.
A series of E3 followed by an E2 is equivalent to an E2 followed by a series of E1.

41. Example Previous example without E3: Input 1 0 0.8 1.0 Input 3
Input 3
Input 2
E2: Output 1
E1: Output 0
With E3:
E3: (x-0.25)x2
Input 2
E2: Output 1
The range after E2°E3 is the same as that after E1°E2

42. Another View of the Equivalence
Scaling of a range in [0.5, 0.75) with E1°E2 E2 E1
Equivalent scaling of the range in [0.5, 0.75) with E2°E3 E3 E2

43. A Simple Proof of E2°E3 = E1°E2
Given an original range r:
After applying E2: [0.5, 1) [0, 1), the range becomes
r1 = (r – 0.5) x 2
After applying E1: [0, 0.5)  [0, 1), the range becomes
r2 = r1 x 2 = ((r – 0.5) x 2) x 2
Given the same range r:
After applying E3: [0.25, 0.75)  [0, 1), the range becomes
r3 = (r – 0.25) x 2
After applying E2, the range becomes
r4 = (r3 – 0.5) x 2 = ((r – 0.25) x 2 – 0.5) x 2
= (r – 0.5) x 2 x 2
= r2
For formal proof:

44. Encoding Operation with E3
Without E3:
Input 2
E2: Output 1
E1: Output 0
With E3: E3
(no output here)
Input 2
E2: Output 1
Output 0 here!
Don’t send anything when E3 is used, but send a 0 after E2:
The bit stream is identical to that of the old method
Subsequent encoding is also same because of the same final interval

45. Decoding for E3 Same status as the old method: low, high, range, tag.
Input
Read 6 bits:
Tag: ( )
Decode 1
Decode 3, E2 scaling
Tag: ( )
Decode 2, E2 scaling
Tag: ( )
E1: Tag: (0.1875)
With E3: Apply E3 whenever it is possible, nothing else is needed
Tag: ( )
E3:
Tag: ( )
Decode 2, E2 scaling
Tag: (0.1875)
Same status as the old method: low, high, range, tag.

46. Summary of Different Scalings
Need E1 scaling
Need E2 scaling
Need E3 scaling
No scaling is required.
Ready to encode/decode the next symbol.

47. Outline Review Integer Implementation Binary Arithmetic Coding
Integer representation
E3 Scaling
Complete Algorithm
Minimum word length
Binary Arithmetic Coding
Adaptive Arithmetic Coding

48. Encoding Pseudo Code with E1, E2, E3
(For integer implementation)
EncodeSymbol(n) {
//Update variables
RANGE = HIGH - LOW + 1;
HIGH = HIGH + RANGE * Cum( n ) / N - 1;
LOW = LOW + RANGE * Cum(n-1) / N;
//Scaling before encoding next symbol
EncoderScaling(); //see next slide }
Round off to integer

49. Encoding Pseudo Code with E1, E2, E3
EncoderScaling( ) {
while (E1, E2 or E3 is possible) {
if (E3 is possible) {
HIGH = ((HIGH - QUARTER) << 1) + 1;
LOW = (LOW - QUARTER) << 1;
Scale3++; //Save number of E3, but send nothing }
if (E1 or E2 is possible) {
Let b=0 for E1 and b=1 for E2
send b
HIGH = (HIGH << 1) + 1;
LOW = (LOW << 1);
while (Scale3 > 0) { //send info about E3 now
send complement of b //E2 ° (E3)^n = (E1)^n ° E2
Scale3 --; //Send one bit for each E3

50. Decoding Pseudo Code with E1, E2, E3
(For integer implementation)
Intervals: [0, 203], [204, 208], [209, 255]
DecodeSymbol(Tag) {
RANGE = HIGH - LOW + 1;
n = 1;
While (Tag > LOW + RANGE * Cum(n) / N - 1) {
n++; }
HIGH = LOW + RANGE * Cum(n) / N - 1;
LOW = LOW + RANGE * Cum(n-1) / N;
//keep scaling before decoding next symbol
DecoderScaling(Tag); //next slide
return n;
Round off to integer: HIGH of each interval

51. Decoding Pseudo Code with E1, E2, E3
DecoderScaling(Tag) {
while (E1, E2 or E3 is possible) {
if (E1 or E2 is possible) {
LOW = LOW << 1;
HIGH = (HIGH << 1) + 1;
Tag = Tag << 1;
Tag = Tag | ReadBits(1); }
if (E3 is possible) {
LOW = (LOW - QUARTER) << 1;
HIGH = ((HIGH - QUARTER) << 1) + 1;
Tag = (Tag - QUARTER) << 1;

52. Integer Encoding with E1, E2, E3
Input 1
Input 3
E2: Output 1
E3: Scale3=1
Input 2
E2: Output 1
Output 0
Scale3 = 0
E1: Output 0
E1: Output 0
E2: Output 1
E1: Output 0
E3: Scale3=1
Output 0
Output 1
Scale3=0
Output 7 more 0’s
Input 1
Final output:

53. Integer Decoding with E1, E2, E3
Input:
Read 8 bits:
(196)
Decode 1
Decode 3
E2: Tag= (137)
E3: Tag = (146)
Decode 2
E2: Tag= (36)
Decode 1
Tag = LOW: stop.

54. Outline Review Integer Implementation Binary Arithmetic Coding
Integer representation
E3 Scaling
Complete Algorithm
Minimum word length
Binary Arithmetic Coding
Adaptive Arithmetic Coding

55. How to decide the word length m?
Need to guarantee non-zero interval for all symbols
in the worst case: k
P(k) Nk
Cum(k) - 1
0.8 40 2
0.02 41 3
0.18 9 50
even when Cum(n) – cum(n -1) = 1.
Otherwise HIGH < LOW.
Need
 RANGE cannot be too small at any time. Intuitive!

56. How to decide the word length m?
When do we have the smallest RANGE without triggering a scaling?
M/ M/ /4M M
M = 2^m
When interval is slightly larger than [M/4, M/2] or [M/2, 3/4M]
None of E1, E2, and E3 can be applied
Condition: 1/4 (2^m) > N
Example: N = 50,  min m = 8 (1/4M=64)

57. Outline Review Integer Implementation Binary Arithmetic Coding
Adaptive Arithmetic Coding

58. Binary Arithmetic Coding
Arithmetic coding is slow in general:
To decode a symbol, we need a seris of decisions and multiplications:
While (Tag > LOW + RANGE * Cum(n) / N - 1) {
n++; }
The complexity is greatly reduced if we have only two symbols: 0 and 1.
symbol symbol 1
0 x
Only two intervals in [0, 1): [0, x), [x, 1)

59. Encoding of Binary Arithmetic Coding
Prob(0)=0.6. Sequence: 0110
LOW = 0, HIGH = 1
LOW = 0, HIGH = 0.6
LOW = 0.36, HIGH = 0.6
LOW = 0.504, HIGH = 0.6
LOW = 0.504, HIGH =
Only need to update LOW or HIGH for each symbol.

60. Decoding of Binary Arithmetic Coding
Tag
Only one decision to make:
While (Tag > LOW + RANGE * Cum(n) / N - 1)) {
n++; }
if (Tag > LOW + RANGE * Cum(Symbol 0) / N - 1) {
n = 1;
} else {
n = 0; }

61. Applications of Binary Arithmetic Coding
Increasingly popular:
JBIG, JBIG2, JPEG2000, H.264
Covert non-binary signals into binary first
H.264: Golomb-Rice Code
Bit-plane coding
Various simplifications to avoid multiplication:
H.264: Table look-up for RANGE * Cum(n) / N
JBIG: Eliminate multiplication by assuming the RANGE is close to 1. Scale if RANGE too small.

62. Outline Review Integer Implementation Binary Arithmetic Coding
Adaptive Arithmetic Coding

63. Adaptive Arithmetic Coding
Observation: The partition of [0, 1) can be different from symbol to symbol
The bit stream can be decoded perfectly as long as both encoder and decoder are synchronized (use the same partition).
General approach:
Starting from a pre-defined probability distribution
Update probability after each symbol
This is very difficult for Huffman coding:
Has to redesign the codebook when prob changes

64. Example Encode 0.4667. Binary sequence: 01111
Initial counters for 0’s and 1’s:
C(0)=C(1)=1.
 P(0)=P(1)=0.5
After encoding 0: C(0)=2, C(1)=1. P(0)=2/3, P(1)=1/3
After encoding 01: C(0)=2, C(1)=2. P(0)=1/2, P(1)=1/2
After encoding 011: C(0)=2, C(1)=3. P(0)=2/5, P(1)=3/5
After encoding 0111: C(0)=2, C(1)=4. P(0)=1/3, P(1)=2/3.
Encode

65. Decoding Input 0.4667. Initial counters for 0’s and 1’s:
C(0)=C(1)=1  P(0)=P(1)=0.5
Decode 0
After decoding 0: C(0)=2, C(1)=1. P(0)=2/3, P(1)=1/3
Decode 1
After decoding 01: C(0)=2, C(1)=2. P(0)=1/2, P(1)=1/2
Decode 1
After decoding 011: C(0)=2, C(1)=3. P(0)=2/5, P(1)=3/5
Decode 1
After decoding 0111: C(0)=2, C(1)= P(0)=1/3, P(1)=2/3.
Decode 1

66. Context-adaptive Arithmetic Coding
In many cases, a sample has strong correlation with its near neighbors.
Idea:
Collect conditional probability distribution of a symbol for given neighboring symbols (context):
P(x(n) | x(n-1), x(n-2), … x(n-k))
Use this conditional probability to encode the next symbol
More skewed probability distribution can be obtained (desired by arithmetic coding)
a b c b c a b
c b a b c b a
a b c b c a b
1-D
Context
template
2-D
Context
template

67. Example 0 1 1 0 1 0 1 Binary sequence: 0, 1
Neighborhood (template) size: 3
2^3=8 possible combinations (contexts) of 3 neighbors
(x(n-1), x(n-2), x(n-3)).
Collect frequencies of 0’s and 1’s under each context
Context
C(0)
C(1)
(0, 0, 0) 9 2
(0, 0, 1) 3 6
(0, 1, 0) 10
(0, 1, 1) …
(1, 0, 0)
(1, 0, 1)
(1, 1, 0)
(1, 1, 1)
Each symbol is coded with
the probability distribution
associated with its context.

68. Encoding Pseudo Code InitProbModel(ProbModel); While (not EOF) {
n = ReadSymbol( );
//Determine the neighboring combination (context)
context = GetContext();
//Encode with the corresponding probability model
EncodeSymbol(ProbModel[context], n);
//update probability counters for this context
UpdateProb(ProbModel[context], n); }

69. Decoding Pseudo Code InitProbModel(ProbModel); While (not EOF) {
//Determine the neighboring combination (context)
context = GetContext();
//Decode with the corresponding probability model
n = DecodeSymbol(ProbModel[context]);
//update probability counters for this context
UpdateProb(ProbModel[context], n); }

70. Performance of Arithmetic Coding
For a sequence of length m:
H(X) <= Average Bit Rate <= H(X) + 2/m
Can approaches the entropy very quickly.
Huffman coding:
H(X) <= Average Bit Rate <= H(X) + 1/m
Impractical: need to generate codewords for all sequences of length m.

71. Summary – Part II Arithmetic coding: Incremental Encoding and Decoding
Partition the range [0, 1) recursively according to symbol probabilities.
Incremental Encoding and Decoding
E1, E2, E3 scaling
Binary Arithmetic Coding
Context Adaptive Arithmetic Coding
Next: Quantization (lossy compression)