1. 10/13/2017

2. Topic 1: Ketchup https://www.superlectures.com/interspeech2016/
Jurafsky uses history of ketchup (& ice cream elsewhere)
to shed light on currently popular methods in speech and language
He traces etymology of “ketchup” from an Asian fish sauce
Advances in (sailing) technology made it possible to replace anchovies with less expensive tomatoes and sugar from the west
The ice cream story combines fruit syrups (Sharbat) from Persia
with gun powder from China and advances in refrigeration technology
Big Tent
Sept 9, 2017
Topics in HLT

3. The Speech Invasion
At speech meetings (Interspeech-2016, as opposed to NAACL-2009),
Jurafsky credits speech researchers for transferring currently popular techniques from speech to language
Some of these people were probably also involved in transferring similar methods from information theory into speech (and perhaps hedge funds)
Sept 9, 2017
Topics in HLT

4. What happened in 1988?
Sept 9, 2017
Topics in HLT

5. What happened in1988?
Sept 9, 2017
Topics in HLT

6. What happened in 1975? Jurafsky’s story is nice & simple,
But history is “complicated”
IMHO,
speech did onto language,
what was done onto them
What happened in 1975?
The same thing that happened to language in 1988
(and to hedge funds in 1990s)?
Sept 9, 2017
Topics in HLT

7. Speech  Language Shannon’s: Noisy Channel Model
I  Noisy Channel  O
I΄ ≈ ARGMAXI Pr(I|O) = ARGMAXI Pr(I) Pr(O|I)
Trigram Language Model
Word
Rank
More likely alternatives We 9
The This One Two A Three Please In
need 7
are will the would also do to 1
resolve 85
have know do…
all of 2
the
important
657
document question first…
issues 14
thing point to
Channel Model
Application
Input
Output
Speech Recognition
writer
rider
OCR (Optical Character Recognition)
all
a1l
Spelling Correction
government
goverment
Application
Independent
Eurospeech 2003

8. Didn’t have the guts to use this slide at Eurospeech (Geneva)
Speech Invasion: Speech  Language Using (Abusing) Shannon’s Noisy Channel Model: Part of Speech Tagging and Machine Translation
Speech
Words  Noisy Channel  Acoustics
OCR
Words  Noisy Channel  Optics
Spelling Correction
Words  Noisy Channel  Typos
Part of Speech Tagging (POS):
POS  Noisy Channel  Words
Machine Translation: “Made in America”
English  Noisy Channel  French
Didn’t have the guts to use this slide at Eurospeech (Geneva)
Eurospeech 2003

9. HMM Example: Dictionary Attack
Wikipedia
HMM View
In computer science, a one-way function is a function that is
easy to compute on every input,
but hard to invert 
In cryptanalysis and computer security,
a dictionary attack is a technique for defeating a cipher or authentication mechanism
by trying hundreds or sometimes millions of likely possibilities,
such as words in a dictionary.
Authentication System
password  Noisy Channel  hash
Unix used to publish
hash function (not even noisy)
encrypted passwords (/etc/passwd)
Dict attacks exposed weakness
I  Noisy Channel  O
I΄ ≈ ARGMAXI Pr(I|O) = ARGMAXI Pr(I) Pr(O|I)
10/13/2017

10. Motivating Hidden Markov Models (HMMs)
10/13/2017

11. Norvig’s Blog http://norvig.com/spell-correct.html
10/13/2017

12. How It Works: Some Probability Theory http://norvig. com/spell-correct
10/13/2017

13. Prior / Language Model Channel Model
10/13/2017

14. Spelling Correction << Code Breaking http://norvig
10/13/2017

15. Hypo: HMMs: Bletchley  IDA  IBM  Speech  NLP
10/13/2017

16. Spelling Correction
10/13/2017

17. Channel Model http://norvig.com/ngrams
10/13/2017

18. 10/13/2017

19. I  Noisy Channel  O I΄ ≈ ARGMAXI Pr(I|O) = ARGMAXI Pr(I) Pr(O|I)
Language
Model
Channel
Model
I  Noisy Channel  O I΄ ≈ ARGMAXI Pr(I|O) = ARGMAXI Pr(I) Pr(O|I)
Prior /
Languag Model
Channel
Model
10/13/2017

20. 10/13/2017

21. 10/13/2017

22. http://www. computational- logic
logic.org/iccl/master/lectures/summer06/nlp/part-of-speech- tagging.pdf
10/13/2017

23. 10/13/2017

24. 10/13/2017

25. 10/13/2017

26. 10/13/2017

27. 10/13/2017

28. 10/13/2017

29. 10/13/2017

30. 10/13/2017

31. 10/13/2017

32. 10/13/2017

33. 10/13/2017

34. 10/13/2017

35. 10/13/2017

36. Part of Speech Tagging is Stuck? Why?
Programs appear to be working as well as people
But when a computer disagrees with a a person,
the computer is wrong
And when two people disagree,
it is a difference of opinion
Evaluation needs to distinguish errors from differences of opinion
Current evaluation method for tagging
Humans label text with a single correct answer (Penn TreeBank)
Computer is judged correct if it produces that answer
Other tasks (IR, Web Search) allow for multiple answers
NDCG: Scoring method
Machine returns top 10 candidate ans
Human judges grade each candidate
perfect
excellent
good
fair
poor
NDCG: candidates + grades  score
Ideally, top 10 candidates have lots of good grades, especially in first few positions
NDCG is expensive
Need to judge all candidates returned by all versions of all systems (unlike Treebanks)
10/13/2017

37. 10/13/2017

38. 10/13/2017

39. 10/13/2017

40. 10/13/2017

41. 10/13/2017

42. http://www- stat. wharton. upenn
stat.wharton.upenn.edu/~steele/Courses/956/Resource/HiddenMark ovModels.htm
10/13/2017