1. Source Coding
Binit Mohanty
Ketan Rajawat

2. Information What is information? Can it be “measured”?
Fundamental Unit of Info – Bits
Represents an ON/OFF state
Complex Info – Array of Bits

3. Quantizing Information
Example I:
Class of 32 people
Need log2(32) = 5 bits
Not enough!
Need Info definition and a Dictionary
What about a class of 25 people? 5 people?

4. Information and Uncertainty
Information: Extra knowledge you gain
Uncertainty: Knowledge that you do not have
Before the knowledge of the outcome you have – uncertainty
After it you have Information!
They are the same thing!

5. Quantizing Information
Example II
Class of 32 people – 6 females, 26 males
Choose one at random
Give the knowledge of gender
By what amount is uncertainty reduced?

6. Quantizing Info… Uncertainty = log(1/Probability) Average this
Entropy!

7. Entropy for a single event

8. Source Coding To compress information Exploit recurrence of sequences
Lossless: Perfect reconstruction possible
Eg: GIF (LZ), WINRAR
Lossy: Tolerable degradation allowed
Eg: JPEG, MP3
Exploit recurrence of sequences

9. Huffman Code David A. Huffman of MIT Short codes for high probability
Longer codes for low probability
An Example:
P(A)=0.5, P(B)=0.25, P(C)=0.25
A=0, B=10,C=11

10. LZ Code By Abraham Lempel & Jacob Ziv Two versions LZ-77 & LZ-78
Parse:
Insert a comma after every new sequence
Transmit
New bit and previous occurrence index

11. Some Other Codes LZ – 77 LZW - Text based coding Elias Coding
Lynch-Davisson Codes