1. Variable-Length Codes: Huffman Codes
Chapter 4
Variable-Length Codes: Huffman Codes

2. Outline 4.1 Introduction 4.2 Unique Decoding 4.3 Instantaneous Codes
4.4 Construction of Instantaneous Codes
4.5 The Kraft Inequality
4.6 Huffman Codes

3. 4.1 Introduction
Consider the problem of efficient coding of message to be sent over a “noiseless” channel.
maximize the number of messages that can be sent in a given period of time.
transmit a message in the shortest possible time.
make the codeword as short as possible.

4. 4.2 Unique Decoding Source symbols (alphabet): { s1, . . . , sq }
Codes alphabet: { C1, C2, , Cr }
X is a random variable
X→{ s1, , sq } with probabilities {p1, , pq}
X is observed over and over again, i.e., it generates a sequence of symbols {s1, , sq}
Ex: s1 → 000
s2 → 111
encode Si
Ci Cj · · · Ck

5. The collection of all codewords is called a ‘code’. Our Objective:
Minimize the average codeword length
Unique decodability − the received message must have a single, unique possible interpretation.
Ex s1 → Source alphabet:{s1, s2, s3, s4}
s2 → Code alphabet:{0,1}
s3 → 11
s4 → 00
Then 0011
So it doesn’t satisfy unique decodability
s4 s3
s1 s1 s3

6. Ex
s1 → 0
s2 → Then 010
s3 → 01
s4 → It also doesn’t satisfy unique decodability
s2 → 01
s3 → It is a unique decodable code
s4 → 111
s1 s4 s2
s3 s1

7. Definition:
The nth extension of a code is simple all possible concatenations of n symbols of the original source code.
No two encoded concatenations can be the same, even for different extensions.
Every finite sequence of code characters corresponds at most one message.
≡every distinct sequence of source symbols has a corresponding encoded sequence that is unique.

8. 4.3 Instantaneous Codes Decision (Decoding) tree: s1 = 0 s2 = 10
Initial state s2 1 s3 1 1 s4

9. Note that: each bit of the received stream is examined only once and that the terminal states of this tree are the four source symbols s1, s2, s3 and s4.
Definition: A code is instantaneous if it is decodable without lookahead (i.e., a word can be recognized as soon as complete).
When a complete symbol is received, the receiver immediately know this, and do not have to look further before deciding what message symbol you received.
A code is instantaneous iff no codeword si is a prefix of another codeword sj.

10. the existence of the decoding tree
≡ the existence of the instantaneous decodability
Ex. Let n be a positive integer. A comma code is a code with codewords
“1” becomes a comma to represent end of a codeword.
Because a comma code is prefix-free, it is a instantaneous code.
1, 01, 001, 0001, , 00···01, 00···0
n-1 n

11. So it had better use comma code.
Ex:
ex: 01111·····111
So it had better use comma code.
s1 → 0
s2 → 01
s3 → 011
s4 → 111
Not instantaneous code, but it still be uniquely decodable code.
U.D.
I.C. is better than U.D.
I.C.
s1 → 1
s2 → 01
s3 → 001
s4 → 001 s4 s4

12. 4.4 Construction of Instantaneous Codes
Given five symbols si in the source code S.
Both C1 and C2 are Instantaneous Codes, which one is better?
Answer: Depends on the frequency of occurrence of the symbols
s1 → 0
s2 → 10
s3 → 110
s4 → 1110
s5 → 1111
s1 → 00
s2 → 01
s3 → 10
s4 → 110
s5 → 111 C1 C2

13. 4.5 Kraft Inequality
Theorem: A necessary and sufficient condition for the existence of an instantaneous code S of q symbols si
(i = 1, .., q) with encoded words of length
l1  l2  ···  lq is where r is the radix (number of symbols) of the alphabet of the encoded symbols.

14. Thm: An instantaneous code with word length n1 , n2,
Thm: An instantaneous code with word length n1 , n2, . . ., nM exits iff where D is the size of the code alphabet. () For simplicity, we assume D = 2 (1) when M = 2, n1 = 1 and n2 = s1 s21 s1
is OK for tree of length 1 1 s2

15. (2)If M  k is OK,
then k’  k”  1 when M = k + 1, By induction method, the inequality is true. K’ k K”

16. Another proof: () C = {c1, c2, …, cM} with codeword lengths l1, …, lM Let L = max{ li } If where yj are any code symbols, cannot be in C because ci is a prefix of x. x has possibilities.
words (length of L) not in C

17. => If there are 1 number of words with length 1 then 1 r.
(2  r2 - 1r) Infer that, 3  r3 - 1r2 - 2r. => 1  r 1r + 2  r 1r2 + 2r+ 3  r 1rn-1 + 2rn-2 + n  rn So if it satisfy the last equation, then all the equations hold. …
=> It satisfy Kraft’s inequality.

18. Note: A code may obey Kraft inequality still not be instantaneous
Note: A code may obey Kraft inequality still not be instantaneous. EX: EX: Binary Block codes (Error Correcting Codes) ( ) ( )
But it is not I.C. k h
b : 2

19. Ex: Comma code D {……. } length 1 1 (It must to be
Ex: Comma code D {…….} length (It must to be.) length D-1 length (D-1)2 length k D(D-1)k-1
Kraft inequality can be extended to any uniquely decodable codes. … …

20. McMillan Inequality: Thm: A uniquely decodable code has word length l1, l2, …, lq exits iff (r is the size of the code alphabet)
()Trivial. Because I.C. is one kind of U.C. () where l is the length of the longest symbol, i.e., and Nk is the number of code symbols (of radix r) of length k.

21. (the number of distinct sequences of length k in radix r) If k > 1, we can find a n s.t. kn > nl →←

22. 4.6 Huffman Codes
Lemma: If a code C is optimal within the class of instantaneous codes, then C is optimal with the entire class of U.D. codes.
pf: Suppose C ’ is a U.D. code. C ’ has a smaller average codeword length than C.
Let n1’, n2’ , , nM’ be the codeword length of C ’
So, C is not optimal in I.C. →←
(It satisfy Kraft Inequality)

23. Optimal Codes:
Given a binary I.C. C with codeword length n1, …, nM associated with probability p1, …, pM.
For convenience, let {p1 ≥ p2 ··· ≥ pM-1 ≥ pM}
(ni ≤ ni+1≤ ··· ≤ ni+r if pi = pi+1 = ··· = pi+r)
Then C is optimal within the class of I.C., C must have the following properties:

24. (a) Higher probable symbols have shorter codewords.
i.e. if pj > pk => nj ≤ nk
(b) The 2 least probable symbols have codewords of equal length, i.e., nM-1 = nM
(c) Among the codewords of length nM, 2 codes the agree in all digits except the least one.
Ex:
x1 → 0
x2 → 100
x3 → 101
x4 → 1101
x5 → 1110
Don’t satisfy (c), it have to be
x4 → 1101
x5 → 1110

25. pf:
(a) if ( pj > pk ) ( nj > nk ) then we can construct a better codes C ’ by interchange codewords j, k.
(b) From (a) if pM-1 > pM then nM-1 ≤ nM & By assumption if pM-1 = pM then nM-1 ≤ nM .We may make nM-1 = nM and still have in I.C. better than the original one.
(c) If condition (c) is not true, we may drop the least digit of all such codewords to obtain a better code.
Huffman coding─ Construction of Optimal (instantaneous) codes

26. (1) Combine xM-1, xM into xM-1,M with probability pM-1+pM
Let x1, …, xM be an array of symbols with probabilities p1, …, pM ( p1 ≥ p2 ≥ ··· ≥ pM)
(1) Combine xM-1, xM into xM-1,M with probability pM-1+pM
(2) Assume we can construct an O.C. C2 for x1, x2, …, xM-1,M
(3) Now construct a code C1 for x1, …, xM as follows
The codeword associated with x1, …, xM-2 in C1 is exactly the same as the corresponding codewords of C2
Let wM-1,M be the codeword of xM-1,M in C2
The codewords for xM-1, xM in C1 is
either wM-1,M 0 → xM-1 or
wM-1,M 1 → xM

27. Claim: C1 is an optimal code for the set of probability p1, …, pM.
Ex:
x3,4,5,6 0.45 x x
x1,
x3,4,5,6 0.45
x1 0.3
x2 0.25
x3 0.2
x4 0.1
x5 0.1
x6 0.05 x x x
x5,6 0.15 x x x
x4,5,6 0.25 x

28. x x
x1,2 0
x3,4,5,6 1 x
x4,5,6 11
x3,4,5,6 1 x
x5,6 111
x5 1110
x6 1111

29. pf: We assume that C1 is not optimal.
Let C1’ be an optimal instantaneous code for x1, …, xM.
Then C1’ has codewords w1’, w2’, …, wM’ with length n1’, n2’, …, nM’.
If there are only two symbols of maximum length in a tree, they must have their last decision node in common, and they must be the two least probable symbols. Before we reduce a tree, the two symbols contribute nM( pM + pM-1) and after the reduction they contribute (nM - 1)( pM + pM-1).
So that the code length is reduced by ( pM + pM-1).
Average length of C1 > Average length of C1’ --- (1)
After reduction,
Average length of C2 > Average length of C2’
(The terms of (1) minus pM+pM-1)
But C2 is optimal →←

30. Huffman encoding is not unique.
If there are more than two symbols of the maximum length, we can use the following proposition:
Symbols having the same length may be inter-change without changing the average code length.
We can use the biggest two probable symbols to encode like the way before.
Huffman encoding is not unique.

31. Ex:
p1 = 0.4 → 00
p2 = 0.2 → 10
p3 = 0.2 → 11
p4 = 0.1 → 010
p5 = 0.1 → 011
Average length :
L = 0.4·2+0.2·2+0.2·2+0.1·3+0.1·3
= 2.2 Or
p1 = 0.4 → 1
p2 = 0.2 → 01
p3 = 0.2 → 000
p4 = 0.1 → 0010
p5 = 0.1 → 0011
Average length :
L = 0.4·1+0.2·2+0.2·3+0.1·4+0.1·4
= 2.2

32. Which encoding way is better?
Var( I ) = 0.4(2-2.2) (2-2.2) (2-2.2) (3-2.2) (3-2.2)2 = 0.16 (Good!)
Var( II ) = 0.4(1-2.2) (2-2.2)2 +
0.2(3-2.2) (4-2.2) (4-2.2)2 = 1.36