1. Speech and Language Processing Lecture 4 Chapter 4 of SLP

2. 03/26/10 Speech and Language Processing - Jurafsky and Martin 2 Today  Word prediction task  Language modeling (N-grams)‏  N-gram intro  The chain rule  Model evaluation  Smoothing

3. 03/26/10 Speech and Language Processing - Jurafsky and Martin 3 Word Prediction  Guess the next word... ... I notice three guys standing on the ???  There are many sources of knowledge that can be used to inform this task, including arbitrary world knowledge.  But it turns out that you can do pretty well by simply looking at the preceding words and keeping track of some fairly simple counts.

4. 03/26/10 Speech and Language Processing - Jurafsky and Martin 4 Word Prediction  We can formalize this task using what are called N-gram models.  N-grams are token sequences of length N.  Our earlier example contains the following 2-grams (aka bigrams)‏  (I notice), (notice three), (three guys), (guys standing), (standing on), (on the)‏  Given knowledge of counts of N-grams such as these, we can guess likely next words in a sequence.

5. 03/26/10 Speech and Language Processing - Jurafsky and Martin 5 N-Gram Models  More formally, we can use knowledge of the counts of N-grams to assess the conditional probability of candidate words as the next word in a sequence.  Or, we can use them to assess the probability of an entire sequence of words.  Pretty much the same thing as we’ll see...

6. 03/26/10 Speech and Language Processing - Jurafsky and Martin 6 Applications  It turns out that being able to predict the next word (or any linguistic unit) in a sequence is an extremely useful thing to be able to do.  As we’ll see, it lies at the core of the following applications  Automatic speech recognition  Handwriting and character recognition  Spelling correction  Machine translation  And many more.

7. 03/26/10 Speech and Language Processing - Jurafsky and Martin 7 Counting  Simple counting lies at the core of any probabilistic approach. So let’s first take a look at what we’re counting.  He stepped out into the hall, was delighted to encounter a water brother.  13 tokens, 15 if we include “,” and “.” as separate tokens.  Assuming we include the comma and period, how many bigrams are there?

8. 03/26/10 Speech and Language Processing - Jurafsky and Martin 8 Counting  Not always that simple  I do uh main- mainly business data processing  Spoken language poses various challenges.  Should we count “uh” and other fillers as tokens?  What about the repetition of “mainly”? Should such do- overs count twice or just once?  The answers depend on the application.  If we’re focusing on something like ASR to support indexing for search, then “uh” isn’t helpful (it’s not likely to occur as a query).  But filled pauses are very useful in dialog management, so we might want them there.

9. 03/26/10 Speech and Language Processing - Jurafsky and Martin 9 Counting: Types and Tokens  How about  They picnicked by the pool, then lay back on the grass and looked at the stars.  18 tokens (again counting punctuation)‏  But we might also note that “the” is used 3 times, so there are only 16 unique types (as opposed to tokens).  In going forward, we’ll have occasion to focus on counting both types and tokens of both words and N-grams.

10. 03/26/10 Speech and Language Processing - Jurafsky and Martin 10 Counting: Wordforms  Should “cats” and “cat” count as the same when we’re counting?  How about “geese” and “goose”?  Some terminology:  Lemma: a set of lexical forms having the same stem, major part of speech, and rough word sense  Wordform: fully inflected surface form  Again, we’ll have occasion to count both lemmas and wordforms

11. 03/26/10 Speech and Language Processing - Jurafsky and Martin 11 Counting: Corpora  So what happens when we look at large bodies of text instead of single utterances?  Brown et al (1992) large corpus of English text  583 million wordform tokens  293,181 wordform types  Google  Crawl of 1,024,908,267,229 English tokens  13,588,391 wordform types  That seems like a lot of types... After all, even large dictionaries of English have only around 500k types. Why so many here? Numbers Misspellings Names Acronyms etc

12. 03/26/10 Speech and Language Processing - Jurafsky and Martin 12 Language Modeling  Back to word prediction  We can model the word prediction task as the ability to assess the conditional probability of a word given the previous words in the sequence  P(w n |w 1,w 2 …w n-1 )‏  We’ll call a statistical model that can assess this a Language Model

13. 03/26/10 Speech and Language Processing - Jurafsky and Martin 13 Language Modeling  How might we go about calculating such a conditional probability?  One way is to use the definition of conditional probabilities and look for counts. So to get  P(the | its water is so transparent that)‏  By definition that’s P(its water is so transparent that the)‏ P(its water is so transparent that)‏ We can get each of those from counts in a large corpus.

14. 03/26/10 Speech and Language Processing - Jurafsky and Martin 14 Very Easy Estimate  How to estimate?  P(the | its water is so transparent that)‏ P(the | its water is so transparent that) = Count(its water is so transparent that the)‏ Count(its water is so transparent that)‏

15. 03/26/10 Speech and Language Processing - Jurafsky and Martin 15 Very Easy Estimate  According to Google those counts are 5/9.  Unfortunately... 2 of those were to these slides... So maybe it’s really  3/7  In any case, that’s not terribly convincing due to the small numbers involved.

16. 03/26/10 Speech and Language Processing - Jurafsky and Martin 16 Language Modeling  Unfortunately, for most sequences and for most text collections we won’t get good estimates from this method.  What we’re likely to get is 0. Or worse 0/0.  Clearly, we’ll have to be a little more clever.  Let’s use the chain rule of probability  And a particularly useful independence assumption.

17. 03/26/10 Speech and Language Processing - Jurafsky and Martin 17 The Chain Rule  Recall the definition of conditional probabilities  Rewriting:  For sequences...  P(A,B,C,D) = P(A)P(B|A)P(C|A,B)P(D|A,B,C)‏  In general  P(x 1,x 2,x 3,…x n ) = P(x 1 )P(x 2 |x 1 )P(x 3 |x 1,x 2 )…P(x n |x 1 …x n-1 )‏

18. 03/26/10 Speech and Language Processing - Jurafsky and Martin 18 The Chain Rule P(its water was so transparent)= P(its)* P(water|its)* P(was|its water)* P(so|its water was)* P(transparent|its water was so)‏

19. 03/26/10 Speech and Language Processing - Jurafsky and Martin 19 Unfortunately  There are still a lot of possible sentences  In general, we’ll never be able to get enough data to compute the statistics for those longer prefixes  Same problem we had for the strings themselves

20. 03/26/10 Speech and Language Processing - Jurafsky and Martin 20 Independence Assumption  Make the simplifying assumption  P(lizard|the,other,day,I,was,walking,along,an d,saw,a) = P(lizard|a)‏  Or maybe  P(lizard|the,other,day,I,was,walking,along,an d,saw,a) = P(lizard|saw,a)‏  That is, the probability in question is independent of its earlier history.

21. 03/26/10 Speech and Language Processing - Jurafsky and Martin 21 Independence Assumption  This particular kind of independence assumption is called a Markov assumption after the Russian mathematician Andrei Markov.

22. 03/26/10 Speech and Language Processing - Jurafsky and Martin 22 So for each component in the product replace with the approximation (assuming a prefix of N)‏ Bigram version Markov Assumption

23. 03/26/10 Speech and Language Processing - Jurafsky and Martin 23 Estimating Bigram Probabilities  The Maximum Likelihood Estimate (MLE)‏

24. 03/26/10 Speech and Language Processing - Jurafsky and Martin 24 An Example  I am Sam  Sam I am  I do not like green eggs and ham

25. 03/26/10 Speech and Language Processing - Jurafsky and Martin 25 Maximum Likelihood Estimates  The maximum likelihood estimate of some parameter of a model M from a training set T  Is the estimate that maximizes the likelihood of the training set T given the model M  Suppose the word Chinese occurs 400 times in a corpus of a million words (Brown corpus)‏  What is the probability that a random word from some other text from the same distribution will be “Chinese”  MLE estimate is 400/1000000 =.004  This may be a bad estimate for some other corpus  But it is the estimate that makes it most likely that “Chinese” will occur 400 times in a million word corpus.

26. 03/26/10 Speech and Language Processing - Jurafsky and Martin 26 Berkeley Restaurant Project Sentences  can you tell me about any good cantonese restaurants close by  mid priced thai food is what i’m looking for  tell me about chez panisse  can you give me a listing of the kinds of food that are available  i’m looking for a good place to eat breakfast  when is caffe venezia open during the day

27. 03/26/10 Speech and Language Processing - Jurafsky and Martin 27 Bigram Counts  Out of 9222 sentences  Eg. “I want” occurred 827 times

28. 03/26/10 Speech and Language Processing - Jurafsky and Martin 28 Bigram Probabilities  Divide bigram counts by prefix unigram counts to get probabilities.

29. 03/26/10 Speech and Language Processing - Jurafsky and Martin 29 Bigram Estimates of Sentence Probabilities  P( I want english food ) = P(i| )* P(want|I)* P(english|want)* P(food|english)* P( |food)* =.000031

30. 03/26/10 Speech and Language Processing - Jurafsky and Martin 30 Kinds of Knowledge  P(english|want) =.0011  P(chinese|want) =.0065  P(to|want) =.66  P(eat | to) =.28  P(food | to) = 0  P(want | spend) = 0  P (i | ) =.25  As crude as they are, N-gram probabilities capture a range of interesting facts about language. World knowledge Syntax Discourse

31. 03/26/10 Speech and Language Processing - Jurafsky and Martin 31 Shannon’s Method  Assigning probabilities to sentences is all well and good, but it’s not terribly illuminating. A more interesting task is to turn the model around and use it to generate random sentences that are like the sentences from which the model was derived.  Generally attributed to Claude Shannon.

32. 03/26/10 Speech and Language Processing - Jurafsky and Martin 32 Shannon’s Method  Sample a random bigram (, w) according to its probability  Now sample a random bigram (w, x) according to its probability  Where the prefix w matches the suffix of the first.  And so on until we randomly choose a (y, )‏  Then string the words together  I I want want to to eat eat Chinese Chinese food food

33. 03/26/10 Speech and Language Processing - Jurafsky and Martin 33 Shakespeare

34. 03/26/10 Speech and Language Processing - Jurafsky and Martin 34 Shakespeare as a Corpus  N=884,647 tokens, V=29,066  Shakespeare produced 300,000 bigram types out of V 2 = 844 million possible bigrams...  So, 99.96% of the possible bigrams were never seen (have zero entries in the table)‏  This is the biggest problem in language modeling; we’ll come back to it.  Quadrigrams are worse: What's coming out looks like Shakespeare because it is Shakespeare

35. 03/26/10 Speech and Language Processing - Jurafsky and Martin 35 The Wall Street Journal is Not Shakespeare

36. 03/26/10 Speech and Language Processing - Jurafsky and Martin 36 Evaluation  How do we know if our models are any good?  And in particular, how do we know if one model is better than another.  Well Shannon’s game gives us an intuition.  The generated texts from the higher order models sure look better. That is, they sound more like the text the model was obtained from.  But what does that mean? Can we make that notion operational?

37. 03/26/10 Speech and Language Processing - Jurafsky and Martin 37 Evaluation  Standard method  Train parameters of our model on a training set.  Look at the models performance on some new data  This is exactly what happens in the real world; we want to know how our model performs on data we haven’t seen  So use a test set. A dataset which is different than our training set, but is drawn from the same source  Then we need an evaluation metric to tell us how well our model is doing on the test set.  One such metric is perplexity (to be introduced below)‏

38. 03/26/10 Speech and Language Processing - Jurafsky and Martin 38 Unknown Words  But once we start looking at test data, we’ll run into words that we haven’t seen before (pretty much regardless of how much training data you have.  With an Open Vocabulary task  Create an unknown word token  Training of probabilities  Create a fixed lexicon L, of size V  From a dictionary or  A subset of terms from the training set  At text normalization phase, any training word not in L changed to  Now we count that like a normal word  At test time  Use UNK counts for any word not in training

39. 03/26/10 Speech and Language Processing - Jurafsky and Martin 39 Perplexity  Perplexity is the probability of the test set (assigned by the language model), normalized by the number of words:  Chain rule:  For bigrams:  Minimizing perplexity is the same as maximizing probability  The best language model is one that best predicts an unseen test set

40. 03/26/10 Speech and Language Processing - Jurafsky and Martin 40 Lower perplexity means a better model  Training 38 million words, test 1.5 million words, WSJ

41. 03/26/10 Speech and Language Processing - Jurafsky and Martin 41 Evaluating N-Gram Models  Best evaluation for a language model  Put model A into an application  For example, a speech recognizer  Evaluate the performance of the application with model A  Put model B into the application and evaluate  Compare performance of the application with the two models  Extrinsic evaluation

42. 03/26/10 Speech and Language Processing - Jurafsky and Martin 42 Difficulty of extrinsic (in-vivo) evaluation of N-gram models  Extrinsic evaluation  This is really time-consuming  Can take days to run an experiment  So  As a temporary solution, in order to run experiments  To evaluate N-grams we often use an intrinsic evaluation, an approximation called perplexity  But perplexity is a poor approximation unless the test data looks just like the training data  So is generally only useful in pilot experiments (generally is not sufficient to publish)‏  But is helpful to think about.

43. 03/26/10 Speech and Language Processing - Jurafsky and Martin 43 Zero Counts  Back to Shakespeare  Recall that Shakespeare produced 300,000 bigram types out of V 2 = 844 million possible bigrams...  So, 99.96% of the possible bigrams were never seen (have zero entries in the table)‏  Does that mean that any sentence that contains one of those bigrams should have a probability of 0?

44. 03/26/10 Speech and Language Processing - Jurafsky and Martin 44 Zero Counts  Some of those zeros are really zeros...  Things that really can’t or shouldn’t happen.  On the other hand, some of them are just rare events.  If the training corpus had been a little bigger they would have had a count (probably a count of 1!).  Zipf’s Law (long tail phenomenon):  A small number of events occur with high frequency  A large number of events occur with low frequency  You can quickly collect statistics on the high frequency events  You might have to wait an arbitrarily long time to get valid statistics on low frequency events  Result:  Our estimates are sparse! We have no counts at all for the vast bulk of things we want to estimate!  Answer:  Estimate the likelihood of unseen (zero count) N-grams!

45. 03/26/10 Speech and Language Processing - Jurafsky and Martin 45 Laplace Smoothing  Also called add-one smoothing  Just add one to all the counts!  Very simple  MLE estimate:  Laplace estimate:  Reconstructed counts:

46. 03/26/10 Speech and Language Processing - Jurafsky and Martin 46 Laplace-Smoothed Bigram Counts

47. 03/26/10 Speech and Language Processing - Jurafsky and Martin 47 Laplace-Smoothed Bigram Probabilities

48. 03/26/10 Speech and Language Processing - Jurafsky and Martin 48 Reconstituted Counts

49. 03/26/10 Speech and Language Processing - Jurafsky and Martin 49 Big Change to the Counts!  C(count to) went from 608 to 238!  P(to|want) from.66 to.26!  Discount d= c*/c  d for “chinese food” =.10!!! A 10x reduction  So in general, Laplace is a blunt instrument  Could use more fine-grained method (add-k)‏  But Laplace smoothing not used for N-grams, as we have much better methods  Despite its flaws Laplace (add-k) is however still used to smooth other probabilistic models in NLP, especially  For pilot studies  in domains where the number of zeros isn’t so huge.

50. 03/26/10 Speech and Language Processing - Jurafsky and Martin 50 Better Smoothing  Intuition used by many smoothing algorithms  Good-Turing  Kneser-Ney  Witten-Bell  Is to use the count of things we’ve seen once to help estimate the count of things we’ve never seen

51. 03/26/10 Speech and Language Processing - Jurafsky and Martin 51 Good-Turing Josh Goodman Intuition  Imagine you are fishing  There are 8 species: carp, perch, whitefish, trout, salmon, eel, catfish, bass  You have caught  10 carp, 3 perch, 2 whitefish, 1 trout, 1 salmon, 1 eel = 18 fish  How likely is it that the next fish caught is from a new species (one not seen in our previous catch)?  3/18  Assuming so, how likely is it that next species is trout?  Must be less than 1/18 Slide adapted from Josh Goodman

52. 03/26/10 Speech and Language Processing - Jurafsky and Martin 52 Good-Turing  Notation: N x is the frequency-of-frequency-x  So N 10 =1  Number of fish species seen 10 times is 1 (carp)‏  N 1 =3  Number of fish species seen 1 is 3 (trout, salmon, eel)‏  To estimate total number of unseen species  Use number of species (words) we’ve seen once  c 0 * =c 1 p 0 = N 1 /N  All other estimates are adjusted (down) to give probabilities for unseen Slide from Josh Goodman

53. 03/26/10 Speech and Language Processing - Jurafsky and Martin 53 Good-Turing Intuition  Notation: N x is the frequency-of-frequency-x  So N 10 =1, N 1 =3, etc  To estimate total number of unseen species  Use number of species (words) we’ve seen once  c 0 * =c 1 p 0 = N 1 /Np 0 =N 1 /N=3/18  All other estimates are adjusted (down) to give probabilities for unseen P(eel) = c*(1) = (1+1) 1/ 3 = 2/3 Slide from Josh Goodman

54. 03/26/10 Speech and Language Processing - Jurafsky and Martin 54 GT Fish Example

55. 03/26/10 Speech and Language Processing - Jurafsky and Martin 55 Bigram Frequencies of Frequencies and GT Re-estimates

56. 03/26/10 Speech and Language Processing - Jurafsky and Martin 56 Complications  In practice, assume large counts (c>k for some k) are reliable:  That complicates c*, making it:  Also: we assume singleton counts c=1 are unreliable, so treat N- grams with count of 1 as if they were count=0  Also, need the Nk to be non-zero, so we need to smooth (interpolate) the Nk counts before computing c* from them

57. 03/26/10 Speech and Language Processing - Jurafsky and Martin 57 Backoff and Interpolation  Another really useful source of knowledge  If we are estimating:  trigram p(z|x,y)  but count(xyz) is zero  Use info from:  Bigram p(z|y)‏  Or even:  Unigram p(z)‏  How to combine this trigram, bigram, unigram info in a valid fashion?

58. 03/26/10 Speech and Language Processing - Jurafsky and Martin 58 Backoff Vs. Interpolation  Backoff: use trigram if you have it, otherwise bigram, otherwise unigram  Interpolation: mix all three

59. 03/26/10 Speech and Language Processing - Jurafsky and Martin 59 Interpolation  Simple interpolation  Lambdas conditional on context:

60. 03/26/10 Speech and Language Processing - Jurafsky and Martin 60 How to Set the Lambdas?  Use a held-out, or development, corpus  Choose lambdas which maximize the probability of some held-out data  I.e. fix the N-gram probabilities  Then search for lambda values  That when plugged into previous equation  Give largest probability for held-out set  Can use EM to do this search

61. 03/26/10 Speech and Language Processing - Jurafsky and Martin 61 Katz Backoff

62. 03/26/10 Speech and Language Processing - Jurafsky and Martin 62 Why discounts P* and alpha?  MLE probabilities sum to 1  So if we used MLE probabilities but backed off to lower order model when MLE prob is zero  We would be adding extra probability mass  And total probability would be greater than 1

63. 03/26/10 Speech and Language Processing - Jurafsky and Martin 63 GT Smoothed Bigram Probabilities

64. 03/26/10 Speech and Language Processing - Jurafsky and Martin 64 Intuition of Backoff+Discounting  How much probability to assign to all the zero trigrams?  Use GT or other discounting algorithm to tell us  How to divide that probability mass among different contexts?  Use the N-1 gram estimates to tell us  What do we do for the unigram words not seen in training?  Out Of Vocabulary = OOV words

65. 03/26/10 Speech and Language Processing - Jurafsky and Martin 65 OOV words: word  Out Of Vocabulary = OOV words  We don’t use GT smoothing for these  Because GT assumes we know the number of unseen events  Instead: create an unknown word token  Training of probabilities  Create a fixed lexicon L of size V  At text normalization phase, any training word not in L changed to  Now we train its probabilities like a normal word  At decoding time  If text input: Use UNK probabilities for any word not in training

66. 03/26/10 Speech and Language Processing - Jurafsky and Martin 66 Practical Issues  We do everything in log space  Avoid underflow  (also adding is faster than multiplying)‏

67. 03/26/10 Speech and Language Processing - Jurafsky and Martin 67 Google N-Gram Release

68. 03/26/10 Speech and Language Processing - Jurafsky and Martin 68 Google N-Gram Release  serve as the incoming 92  serve as the incubator 99  serve as the independent 794  serve as the index 223  serve as the indication 72  serve as the indicator 120  serve as the indicators 45  serve as the indispensable 111  serve as the indispensible 40  serve as the individual 234

69. 03/26/10 Speech and Language Processing - Jurafsky and Martin 69 Google Caveat  Remember the lesson about test sets and training sets... Test sets should be similar to the training set (drawn from the same distribution) for the probabilities to be meaningful.  So... The Google corpus is fine if your application deals with arbitrary English text on the Web.  If not then a smaller domain specific corpus is likely to yield better results.