1. EE465: Introduction to Digital Image Processing
Data Compression: Advanced Topics
Huffman Coding Algorithm
Motivation
Procedure
Examples
EE465: Introduction to Digital Image Processing

2. EE465: Introduction to Digital Image Processing
Recall: Variable Length Codes (VLC)
Recall:
Self-information
It follows from the above formula that a small-probability event contains
much information and therefore worth many bits to represent it. Conversely,
if some event frequently occurs, it is probably a good idea to use as few bits
as possible to represent it. Such observation leads to the idea of varying the
code lengths based on the events’ probabilities.
Assign a long codeword to an event with small probability
Assign a short codeword to an event with large probability
EE465: Introduction to Digital Image Processing

3. EE465: Introduction to Digital Image Processing
Two Goals of VLC design
• achieve optimal code length (i.e., minimal redundancy)
For an event x with probability of p(x), the optimal
code-length is , where x denotes the
smallest integer larger than x (e.g., 3.4=4 )
–log2p(x) 
code redundancy:
Unless probabilities of events are all power of 2,
we often have r>0
• satisfy uniquely decodable (prefix) condition
EE465: Introduction to Digital Image Processing

4. EE465: Introduction to Digital Image Processing
“Big Question”
How can we simultaneously achieve minimum redundancy
and uniquely decodable conditions?
D. Huffman was the first one to think about this problem
and come up with a systematic solution.
EE465: Introduction to Digital Image Processing

5. EE465: Introduction to Digital Image Processing
Huffman Coding (Huffman’1952)
Coding Procedures for an N-symbol source
Source reduction
List all probabilities in a descending order
Merge the two symbols with smallest probabilities into a new compound symbol
Repeat the above two steps for N-2 steps
Codeword assignment
Start from the smallest source and work back to the original source
Each merging point corresponds to a node in binary codeword tree
EE465: Introduction to Digital Image Processing

6. EE465: Introduction to Digital Image Processing
Example-I
Step 1: Source reduction
symbol x
p(x)
0.5
0.5
0.5 S N
0.25
0.25
0.5 E
0.125
(NEW)
0.25 W
0.125
(EW)
compound symbols
EE465: Introduction to Digital Image Processing

7. EE465: Introduction to Digital Image Processing
Example-I (Con’t)
Step 2: Codeword assignment
symbol x
p(x)
codeword 1
NEW S
0.5
0.5
0.5 1 S N
0.25
0.25 10 1
0.5 EW 10 E N
0.125
110 1 1
0.25 W
111
110
0.125 1
111 W E
EE465: Introduction to Digital Image Processing

8. EE465: Introduction to Digital Image Processing
Example-I (Con’t) 1 1
NEW
NEW 1 1 S 1 S or EW 10 EW 01 N N 1 1
110
001
000 W E W E
The codeword assignment is not unique. In fact, at each
merging point (node), we can arbitrarily assign “0” and “1”
to the two branches (average code length is the same).
EE465: Introduction to Digital Image Processing

9. EE465: Introduction to Digital Image Processing
Example-II
Step 1: Source reduction
symbol x
p(x)
0.4
0.4
0.4
0.6 e
(aiou) a
0.2
0.2
0.4
0.4
(iou) i
0.2
0.2
0.2 o
0.1
0.2 u
0.1
(ou)
compound symbols
EE465: Introduction to Digital Image Processing

10. EE465: Introduction to Digital Image Processing
Example-II (Con’t)
Step 2: Codeword assignment
symbol x
p(x)
codeword
0.4
0.4
0.4
0.6 1 e
(aiou) 01 a
0.2
0.2
0.4 1
0.4
(iou)
000 i
0.2
0.2
0.2
0010 o
0.1
0.2 u
0.1
(ou)
0011
compound symbols
EE465: Introduction to Digital Image Processing

11. EE465: Introduction to Digital Image Processing
Example-II (Con’t) 1
(aiou) e 00 01
(iou) a
000
001 i
(ou)
0010
0011 o u
binary codeword tree representation
EE465: Introduction to Digital Image Processing

12. EE465: Introduction to Digital Image Processing
Example-II (Con’t)
symbol x
p(x)
codeword
length 1 1
0.4 e a
0.2 01 2 3 i
0.2
000 o
0.1
0010 4 u
0.1
0011 4
If we use fixed-length codes, we have to spend three bits per
sample, which gives code redundancy of =0.878bps
EE465: Introduction to Digital Image Processing

13. EE465: Introduction to Digital Image Processing
Example-III
Step 1: Source reduction
compound symbol
EE465: Introduction to Digital Image Processing

14. EE465: Introduction to Digital Image Processing
Example-III (Con’t)
Step 2: Codeword assignment
compound symbol
EE465: Introduction to Digital Image Processing

15. Summary of Huffman Coding Algorithm
Achieve minimal redundancy subject to the constraint that the source symbols be coded one at a time
Sorting symbols in descending probabilities is the key in the step of source reduction
The codeword assignment is not unique. Exchange the labeling of “0” and “1” at any node of binary codeword tree would produce another solution that equally works well
Only works for a source with finite number of symbols (otherwise, it does not know where to start)
EE465: Introduction to Digital Image Processing