2. INFORMATION THEORY POLYNESIAN REVISITED Thomas Tiahrt, MA, PhD CSC492 – Advanced Text Analytics

3. Models vs. Reality 2 Simplified Polynesian: Not actually a random variable But can be modeled by a random variable

4. Models vs. Reality 3

5. Polynesian Syllable Model 4

6. Joint and Marginal Distributions 5 ptkaiu Per letter probabilities 1/163/81/161/41/8 Per syllable (marginal) 1/83/41/81/21/4

7. Joint Entropy 6

8. 7

9. 8 ptk Per syllable (marginal) 1/83/41/8

10. 9 ptk Per syllable (marginal) 1/83/41/8

11. 10

12. 11

13. 12

14. 13

15. 14

16. Joint Entropy 15

17. Entropy Rate 16

18. Entropy of Human Language 17

19. References 18 Sources: Foundations of Statistical Natural Language Processing, by Christopher Manning and Hinrich Schütze The MIT Press Fundamentals of Information Theory and Coding Design, by Roberto Togneri and Christopher J.S. deSilva Chapman & Hall / CRC

20. The end of the Conditional Entropy slide show has come. End of the Slides 19