LING 388: Language and Computers Sandiway Fong Lecture 26: 12/1

Administrivia Homework #4 –returned on Monday (11/29) Homework #5 –goes out Monday (12/6) –due Monday (12/13) - exam period

Last Time the shape of uncertainty N-grams –unigrams, bigrams, trigrams, quadrigrams –conditional probability –chain rule p(w 1 w 2 w 3...w n ) = p(w 1 ) p(w 2 |w 1 ) p(w 3 |w 1 w 2 )... p(w n |w 1...w n-2 w n-1 )

N-grams Chain rule –bigram/trigram approximation 1st/2nd order Markov Model just look at the preceding one or two words only bigram: p(w n |w n-1 ) trigram:p(w n |w n-2 w n-1 ) –Relative frequency/Maximum Likelihood Estimate (MLE) bigram:f(w n-1 w n )/ f(w n-1 ) trigram:f(w n-2 w n-1 w n )/ f(w n-2 w n-1 ) f = frequency count

N-grams and Corpora Corpus –New York Times (AP) article –November 29th 2004 –Gas Prices Decline to an Average $1.95 Per Gallon

N-grams and Corpora

Contents note: blank lines separate paragraphs, new sentences do not always begin with a new line The retail price of gasoline fell for the fourth straight week to average $1.95 per gallon nationwide, the Energy Department reported Monday. The government survey said the average price nationwide of regular-grade unleaded gasoline declined 0.3 cent last week to $ Prices are 45.5 cents, or 31 percent, higher than a year ago. Pump prices are highest on the West Coast, averaging $2.157 per gallon, and cheapest on the Gulf Coast, averaging $1.841 per gallon. In the Midwest, gas averages $1.884 per gallon. One of the key factors behind the high price of gasoline is the soaring cost of oil -- the result of strong demand, and tight supplies of heating oil. The price of light crude for December delivery rose 32 cents to settle at $49.76 a barrel on the New York Mercantile Exchange. Oil is roughly 66 percent more expensive than a year ago. In other Nymex trading, November unleaded gasoline climbed less than a penny to $ per gallon. The nation's supply of commercially available gasoline is 1.7 percent higher than last year at million barrels, according to the Energy Department. Crude inventories stand at million barrels, or 2 percent above last year at this time.

N-grams and Corpora Contents The retail price of gasoline fell for the fourth straight week to average $1.95 per gallon nationwide, the Energy Department reported Monday. The government survey said the average price nationwide of regular-grade unleaded gasoline declined 0.3 cent last week to $ Prices are 45.5 cents, or 31 percent, higher than a year ago. Pump prices are highest on the West Coast, averaging $2.157 per gallon, and cheapest on the Gulf Coast, averaging $1.841 per gallon. In the Midwest, gas averages $1.884 per gallon. One of the key factors behind the high price of gasoline is the soaring cost of oil -- the result of strong demand, and tight supplies of heating oil. The price of light crude for December delivery rose 32 cents to settle at $49.76 a barrel on the New York Mercantile Exchange. Oil is roughly 66 percent more expensive than a year ago. In other Nymex trading, November unleaded gasoline climbed less than a penny to $ per gallon. The nation's supply of commercially available gasoline is 1.7 percent higher than last year at million barrels, according to the Energy Department. Crude inventories stand at million barrels, or 2 percent above last year at this time.

N-grams and Corpora Contents (StarT = dummy word for start of sentence) StarT the retail price of gasoline fell for the fourth straight week to average $1.95 per gallon nationwide, the Energy Department reported Monday. StarT the government survey said the average price nationwide of regular-grade unleaded gasoline declined 0.3 cent last week to $ StarT prices are 45.5 cents, or 31 percent, higher than a year ago. StarT pump prices are highest on the West Coast, averaging $2.157 per gallon, and cheapest on the Gulf Coast, averaging $1.841 per gallon. StarT in the Midwest, gas averages $1.884 per gallon. StarT one of the key factors behind the high price of gasoline is the soaring cost of oil -- the result of strong demand, and tight supplies of heating oil. StarT the price of light crude for December delivery rose 32 cents to settle at $49.76 a barrel on the New York Mercantile Exchange. StarT Oil is roughly 66 percent more expensive than a year ago. StarT in other Nymex trading, November unleaded gasoline climbed less than a penny to $ per gallon. StarT the nation's supply of commercially available gasoline is 1.7 percent higher than last year at million barrels, according to the Energy Department. StarT crude inventories stand at million barrels, or 2 percent above last year at this time.

N-grams and Corpora Ted Petersen’s Perl-based Ngram Statistics Package (NSP) – Computes N-gram statistics –Usage: count.pl [OPTIONS] DESTINATION SOURCE [[, SOURCE]...] Counts up the frequency of all n-grams occurring in SOURCE. Sends to DESTINATION the list of n-grams found, along with the frequencies of combinations of the n tokens that the n-gram is composed of. If SOURCE is a directory, all text files in it are counted. –OPTIONS: --ngram N –Creates n-grams of N tokens each. N = 2 by default. --newLine –Prevents n-grams from spanning across the new-line character.

N-grams and Corpora perl count.pl --ngram 1 --newline nyt.ug nyt2.txt Unigram stats 259.<>23 the<>16 StarT<>11,<>11 of<>9 1<>6 to<>5 gallon<>5 per<>5 gasoline<>5 price<>4 at<>4 a<>4

N-grams and Corpora Bigram stats 247 1<>.<> per<>gallon<>5 5 5 StarT<>the<> gallon<>.<> than<>a<>3 4 4 on<>the<> price<>of<>3 4 9 week<>to<>2 2 5 Coast<>,<>2 2 11,<>averaging<> higher<>than<>2 2 4 unleaded<>gasoline<>2 2 5 last<>year<>2 3 4 year<>at<>2 4 4 # n-grams w 1 <> w 2 <> f 1 f 2 f 3 (<> = separator) f 1 = bigram freq f 2 = freq of w 1 occuring in a bigram as the 1st word f 3 = freq of w 2 occuring in a bigram as the 2nd word

N-grams and Corpora Trigram stats 235 per<>gallon<>.<> year<>ago<>.<> million<>barrels<>,<> to<>1<>.<> a<>year<>ago<> than<>a<>year<> last<>year<>at<> price<>of<>gasoline<> Coast<>,<>averaging<> <>.<>7<> and<>tight<>supplies<> w 1 <> w 2 <> w 3 <> f 1 f 2 f 3 f 4 f 5 f 6 f 7 f 1 = trigram freq f 2 = freq of w 1 occuring in a trigram as the 1st word f 3 = freq of w 2 occuring in a trigram as the 2nd word f 4 = freq of w 3 occuring in a trigram as the 3rd word...

N-grams and Corpora Probability calculations –Relative frequency/Maximum Likelihood Estimate (MLE) bigram:f(w n-1 w n )/ f(w n-1 ) trigram:f(w n-2 w n-1 w n )/ f(w n-2 w n-1 ) f = frequency count

Language Models and N-grams Approximating Shakespeare –generate random sentences using n-grams –train on complete Works of Shakespeare Unigram (pick random, unconnected words) Bigram

Language Models and N-grams Approximating Shakespeare –generate random sentences using n-grams –train on complete Works of Shakespeare Trigram Quadrigram Remarks: dataset size problem training set is small 884,647 words 29,066 different words 29,066 2 = 844,832,356 possible bigrams for the random sentence generator, this means very limited choices for possible continuations, which means program can’t be very innovative for higher n

Language Models and N-grams Aside:

Language Models and N-grams Fall ‘04 LING 538 student project (Tom Meservy & Kelly Fadel) Java code for generating random sentences –trained on raw text Shakespeare, Confucius, Bible, and Beatles lyrics –Demo... java ling538.Processor Lyrics.txt.DS_Store Lyrics.txt Calculating probabilities Probability Sequence~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ E- 4~ ~ ~ ~ ~ ~ E- 4~ ~ ~ ~ ~ ~ ~ ~ Unigram Sentence~you ~the ~love ~i ~my ~you ~and ~to ~i ~i ~a ~my ~and ~i ~my ~my ~lies ~now ~i ~i ~i ~i ~and ~the ~you ~to ~to ~you ~need ~back ~my ~i ~i ~me ~the ~i ~bed, ~you ~i ~i ~and ~i ~spain" ~and ~with ~see ~my ~i ~me ~i Bigram Sentence~you ~know ~she ~was ~waiting ~for ~your ~eyes ~now ~you're ~no ~time ~ago ~your ~emotional ~rescue ~i ~knew ~her ~in ~the ~world ~you ~can ~i ~don't ~come ~on ~back ~your ~emotional ~rescue ~yeah, ~oh ~yeah, ~yeah, ~why ~you ~know ~i ~can ~i ~awoke ~i ~really ~knows ~turn ~you ~can ~i Trigram Sentence~you ~want to ~change the ~harlem shuffle. ~hello little ~help from ~me to ~make it ~won't be ~belong, yeah, ~i said ~so she's ~an angel ~sent to ~me i ~gotta do ~what he ~left it ~be let ~me down ~i'm really ~got a ~loser and ~i'll be ~your man ~that's the ~day i ~want you ~oh, did ~she understand ~his song ~that was ~waiting for ~me dream ~would take ~a chance ~with me, ~hold me, ~martha my ~name you ~know what ~can i ~notice i ~don't know ~you made ~up you're ~gonna say ~you love ~you and ~i will Quadrigram Sentence~you ~know, you know ~of thee, ah ~i got a ~chance with me ~mine, i me ~too much and ~i like you ~i want you ~know, you know ~what it's like ~i please you ~love me too ~much it's all ~his nowhere plans ~and schemes, lost ~themselves instead please ~believe me when ~i tell you ~i want you ~i want you ~i want you ~love me too ~fast you gotta ~carry that weight ~a long time ~it's getting better ~all the time ~that was so ~i must be ~the way things ~to tell her ~majesty's a pretty ~nice girl but ~like you heard ~the word is ~good spread the ~sky with diamonds ~lucy in the ~sky with diamonds ~pictures yourself in ~love with you ~i didn't catch ~will you walk ~out and make ~you tell me, ~oh every little ~help from my ~car yes i'm ~in love with

Next Time Lab Class –Location: SBSRI Computer Lab.