2. Entropy estimation and lossless compression

3. Structure and Entropy of English How much lossless compression can be achieved for a given image? How much lossless compression can be achieved for a given image? It is bounded by the entropy value, but how do we measure the entropy value? It is bounded by the entropy value, but how do we measure the entropy value? Take English as an example Take English as an example S={A, B, C, …, Z, } S={A, B, C, …, Z, } DMS DMS p(s i ) = 1/27 p(s i ) = 1/27 H(S) = log 2 (27) = 4.75 bits/symbol H(S) = log 2 (27) = 4.75 bits/symbol

4. Structure and Entropy of English DMS based on symbol probabilities DMS based on symbol probabilities H(S) = p(s i )log 2 (1/p(s i )) = 4.03 bits/symbol H(S) = p(s i )log 2 (1/p(s i )) = 4.03 bits/symbol Symbol Probability Symbol Probability Symbol Probability Symbol Probability Space0.1589 N 0.0574 Space0.1589 N 0.0574 A0.0642 O 0.0632 B 0.0127 P 0.0152 C 0.0218 Q 0.0008 D 0.0317 R 0.0484 E 0.1031 S 0.0514 F 0.0208 T 0.0796 G 0.0152 U 0.0228 H 0.0467 V 0.0083 I 0.0575 W 0.0157 J 0.0008 X 0.0013 K 0.0049 Y 0.0164 L 0.0321 Z 0.0005 M 0.0198

5. Structure and Entropy of English First-order Markov Process First-order Markov Process 2nd-order Markov Process 2nd-order Markov Process

6. Rate-Distortion Theory and Lossy Compression

7. What is the minimum bit rate require to encode a source while keeping the resulting degradation below a certain level? What is the minimum bit rate require to encode a source while keeping the resulting degradation below a certain level? Rate-distortion theory Rate-distortion theory Rate-distortion function R(D) satisfies Rate-distortion function R(D) satisfies For any given level of distortion D, it is possible to find a coding scheme with rate arbitrarily close to R(D) and average distortion arbitrarily close to D For any given level of distortion D, it is possible to find a coding scheme with rate arbitrarily close to R(D) and average distortion arbitrarily close to D It is impossible to find a code that archives reproduction with distortion D ( or better ) at a rate below R(D) It is impossible to find a code that archives reproduction with distortion D ( or better ) at a rate below R(D)

8. Rate-Distortion Theory and Lossy Compression