1. Compression Algorithms
CSCI 2720
Spring 2005
Eileen Kraemer

2. When we last met … We looked at string encoding
And noted that if we use same number of bits per character in our alphabet, then
the number of bits required to encode a character of the alphabet is
log2(ceil(sizeof(alphabet))
And we don’t need to transmit or store the mapping from encodings to characters

3. What if …
the string we encode doesn’t use all the letters in the alphabet?
log2(ceil(sizeof(set_of_characters_used))
But then also need to store / transmit the mapping from encodings to characters
… and is typically close to size of alphabet

4. And we also looked at: … Huffman Encoding
Assumes encoding on a per-character basis
Observation: assigning shorter codes to frequently used characters (which requires assigning longer codes to rarely used characters) can result in overall shorter encodings of strings
Problem:
when decoding, need to know how many bits to read off for each character.
Solution:
Choose an encoding that ensures that no character encoding is the prefix of any other character encoding. An encoding tree has this property.

5. A Huffman Encoding Tree21 1 9 12 E 1 5 7 1 1 3 2 3 4 A T R N

6. 12 21 9 7 4 3 5 2 A T R N E 1 A
000 T
001 R
010 N
011 E 1

7. Weighted path length A 000 T 001 R 010 N 011 E 1
Weighted path = Len(code(A)) * f(A) +
Len(code(T)) * f(T) + Len(code(R) ) * f(R) +
Len(code(N)) * f(N) + Len(code(E)) * f(E)
= (3 * 3) + ( 2 * 3) + (3 * 3) + (4 *3) + (9*1)
= = 45
Claim (proof in text) : no other encoding can result in a shorter weighted path length A
000 T
001 R
010 N
011 E 1

8. Taking a step back … Why do we need compression?
rate of creation of image and video data
image data from digital camera
today 1k by 1.5 k is common = 1.5 mbytes
need 2k by 3k to equal 35mm slide = 6 mbytes
video at even low resolution of
512 by 512 and 3 bytes per pixel, 30 frames/second

9. Compression basics video data rate mpeg-1 compresses
23.6 mbytes/second
2 hours of video = 169 gigabytes
mpeg-1 compresses
23.6 mbytesdown to 187 kbytes per second
169 gigabytes down to 1.3 gigabytes
compression is essential for both storage and transmission of data

10. Compression basics compression is very widely used
jpeg, giff for single images
mpeg1, 2, 3, 4 for video sequence
zip for computer data
mp3 for sound
based on two fundamental principles
spatial coherence and temporal coherence
similarity with spatial neighbour
similarity with temporal neighbour

11. Basics of compression character = basic data unit in the input stream
represents byte, bit, etc.
strings = sequences of characters
encoding = compression
decoding = decompression
codeword = data elements used to represent input characters or character strings
codetable = list of codewords

12. Codeword encoding/compression takes decoder/decompressor takes
characters/strings as input and use codetable to decide on which codewords to produce
decoder/decompressor takes
codewords as input and uses same codetable to decide on which characters/strings to produce

13. Codetable
clearly both encoder and decoder must pass the encoded data as a series of codewords
also must pass the codetable
the codetable can be passed explicitly or implicitly
that is we either
pass it across
agree on it beforehand (hard wired)
recreate it from the codewords (clever!)

14. Basic definitions compression ratio = lossless compression
size of original data / compressed data
basically higher compression ratio the better
lossless compression
output data is exactly same as input data
essential for encoding computer processed data
lossy compression
output data not same as input data
acceptable for data that is only viewed or heard

15. Lossless versus lossy
human visual system less sensitive to high frequency losses and to losses in color
lossy compression acceptable for visual data
degree of loss is usually a parameter of the compression algorithm
tradeoff - loss versus compression
higher compression => more loss
lower compression => less loss

16. Symmetric versus asymmetric
encoding time == decoding time
essential for real-time applications (ie. video or audio on demand)
asymmetric
encoding time >> decoding
ok for write-once, read-many situations

17. Entropy encoding
compression that does not take into account what is being compressed
normally is also lossless encoding
most common types of entropy encoding
run length encoding
Huffman encoding
modified Huffman (fax…)
Lempel Ziv

18. Source encoding takes into account type of data (ie. visual)
normally is lossy but can also be lossless
most common types in use:
JPEG, GIF = single images
MPEG = sequence of images (video)
MP3 = sound sequence
often uses entropy encoding as a sub-routine

19. Run length encoding one of simplest and earliest types of compression
take account of repeating data (called runs)
runs are represented by a count along with the original data
eg. AAAABB => 4A2B
do you run length encode a single character?
no, use a special prefix character to represent start of runs

20. Run length encoding runs are represented as prefix char itself becomes
<prefix char><repeat count><run char>
prefix char itself becomes
<prefix char>1<prefix char>
want a prefix char that is not too common
an example early use is MacPaint file format
run length encoding is lossless and has fixed length codewords

21. MacPaint File Format

22. Run length encoding works best for images with solid background
good example of such an image is a cartoon
does not work as well for natural images
does not work well for English text
however, is almost always a part of a larger compression system

23. Huffman encoding
assume we know the frequency of each character in the input stream
then encode each character as a variable length bit string, with the length inversely proportional to the character frequency
variable length codewords are used; early example is Morse code
Huffman produced an algorithm for assigning codewords optimally

24. Huffman encoding
input = probabilities of occurrence of each input character (frequencies of occurrence)
output is a binary tree
each leaf node is an input character
each branch is a zero or one bit
codeword for a leaf is the concatenation of bits for the path from the root to the leaf
codeword is a variable length bit string
a very good compression ratio (optimal)?

25. Huffman encoding Basic algorithm
Mark all characters as free tree nodes
While there is more than one free node
Take two nodes with lowest freq. of occurrence
Create a new tree node with these nodes as children and with freq. equal to the sum of their freqs.
Remove the two children from the free node list.
Add the new parent to the free node list
Last remaining free node is the root of the binary tree used for encoding/decoding

26. Huffman example a series of colours in an 8 by 8 screen
colours are red, green, cyan, blue, magenta, yellow, and black
sequence is
rkkkkkkk gggmcbrr
kkkrrkkk bbbmybbr
kkrrrrgg gggggggr
kkbcccrr grrrrgrr

27. Huffman example

28. Huffman example

29. Huffman example

30. Huffman example

31. Fixed versus variable length codewords
run length codewords are fixed length
Huffman codewords are variable length
length inversely proportional to frequency
all variable length compression schemes have the prefix property
one code can not be the prefix of another
binary tree structure guarantees that this is the case (a leaf node is a leaf node!)

32. Huffman encoding advantages disadvantages
maximum compression ratio assuming correct probabilities of occurrence
easy to implement and fast
disadvantages
need two passes for both encoder and decoder
one to create the frequency distribution
one to encode/decode the data
can avoid this by sending tree (takes time) or by having unchanging frequencies

33. Modified Huffman encoding
if we know frequency of occurrences, then Huffman works very well
consider case of a fax; mostly long white spaces with short bursts of black
do the following
run length encode each string of bits on a line
Huffman encode these run length codewords
use a predefined frequency distribution
combination run length, then Huffman

34. Lempel Ziv Welsch (LZW)
previous methods worked only on characters
LZW works by encoding strings
some strings are replaced by a single codeword
for now assume codeword is fixed (12 bits)
for 8 bit characters, first 256 (or less) entries in table are reserved for the characters
rest of table ( ) represent strings

35. LZW compression
trick is that strings to codeword mapping is created dynamically by the encoder
also recreated dynamically by the decoder
need not pass the code table between the two
is a lossless compression algorithm
degree of compression hard to predict
depends on data, but gets better as codeword table contains more strings

36. LZW encoder

37. Demonstrations
A nice animated version of Lempel-Ziv

38. LZW encoder example
compress the string BABAABAAA

39. LZW decoder

40. Lempel-Ziv compression
a lossless compression algorithm
All encodings have the same length
But may represent more than one character
Uses a “dictionary” approach – keeps track of characters and character strings already encountered

41. LZW decoder example
decompress the string <66><65><256><257><65><260>

42. LZW Issues compression better as the code table grows
what happens when all 4096 locations in string table are used?
A number of options, but encoder and decoder must agree to do the same thing
do not add any more entries to table (as is)
clear codeword table and start again
clear codeword table and start again with larger table/longer codewords (GIF format)

43. LZW advantages/disadvantages
simple, fast and good compression
can do compression in one pass
dynamic codeword table built for each file
decompression recreates the codeword table so it does not need to be passed
disadvantages
not the optimum compression ratio
actual compression hard to predict

44. Entropy methods all previous methods are lossless and entropy based
lossless methods are essential for computer data (zip, gnuzip, etc.)
combination of run length encoding/huffman is a standard tool
are often used as a subroutine by other lossy methods (Jpeg, Mpeg)

45. Lempel-Ziv compression
a lossless compression algorithm
All encodings have the same length
But may represent more than one character
Uses a “dictionary” approach – keeps track of characters and character strings already encountered