Lecture 1, 7/21/2005Natural Language Processing1 CS60057 Speech &Natural Language Processing Autumn 2007 Lecture 16 5 September 2007

Lecture 1, 7/21/2005Natural Language Processing2 A Brief History  Machine translation was one of the first applications envisioned for computers  Warren Weaver (1949): “I have a text in front of me which is written in Russian but I am going to pretend that it is really written in English and that it has been coded in some strange symbols. All I need to do is strip off the code in order to retrieve the information contained in the text.”  First demonstrated by IBM in 1954 with a basic word-for-word translation system

Lecture 1, 7/21/2005Natural Language Processing3 Rule-Based vs. Statistical MT  Rule-based MT: Hand-written transfer rules Rules can be based on lexical or structural transfer Pro: firm grip on complex translation phenomena Con: Often very labor-intensive -> lack of robustness  Statistical MT Mainly word or phrase-based translations Translation are learned from actual data Pro: Translations are learned automatically Con: Difficult to model complex translation phenomena

Lecture 1, 7/21/2005Natural Language Processing4 Parallel Resources  Newswire: DE-News (German-English), Hong-Kong News, Xinhua News (Chinese-English),  Government: Canadian-Hansards (French-English), Europarl (Danish, Dutch, English, Finnish, French, German, Greek, Italian, Portugese, Spanish, Swedish), UN Treaties (Russian, English, Arabic,... )  Manuals: PHP, KDE, OpenOffice (all from OPUS, many languages)  Web pages: STRAND project (Philip Resnik)

Lecture 1, 7/21/2005Natural Language Processing5 Parallel Corpus  Example from DE-News (8/1/1996) EnglishGerman Diverging opinions about planned tax reform Unterschiedliche Meinungen zur geplanten Steuerreform The discussion around the envisaged major tax reform continues. Die Diskussion um die vorgesehene grosse Steuerreform dauert an. The FDP economics expert, Graf Lambsdorff, today came out in favor of advancing the enactment of significant parts of the overhaul, currently planned for Der FDP - Wirtschaftsexperte Graf Lambsdorff sprach sich heute dafuer aus, wesentliche Teile der fuer 1999 geplanten Reform vorzuziehen.

Lecture 1, 7/21/2005Natural Language Processing6 Three Problems for Statistical MT  Language model Given an English string e, assigns P(e) by formula good English string -> high P(e) random word sequence -> low P(e)  Translation model Given a pair of strings, assigns P(f | e) by formula look like translations -> high P(f | e) don’t look like translations -> low P(f | e)  Decoding algorithm Given a language model, a translation model, and a new sentence f … find translation e maximizing P(e) * P(f | e) Slide from Kevin Knight

Lecture 1, 7/21/2005Natural Language Processing7 The Classic Language Model Word N-Grams Goal of the language model -- choose among: He is on the soccer field He is in the soccer field Is table the on cup the The cup is on the table Rice shrine American shrine Rice company American company Slide from Kevin Knight

Lecture 1, 7/21/2005Natural Language Processing8 Language Modeling  Determines the probability of some English sequence of length l  P(e) is hard to estimate directly, unless l is very small  P(e) is normally approximated as: where m is size of the context, i.e. number of previous words that are considered, normally m=2 (tri-gram language model

Lecture 1, 7/21/2005Natural Language Processing9 Word-Level Alignments  Given a parallel sentence pair we can link (align) words or phrases that are translations of each other:

Lecture 1, 7/21/2005Natural Language Processing10 Sentence Alignment  If document D e is translation of document D f how do we find the translation for each sentence?  The n-th sentence in D e is not necessarily the translation of the n-th sentence in document D f  In addition to 1:1 alignments, there are also 1:0, 0:1, 1:n, and n:1 alignments  Approximately 90% of the sentence alignments are 1:1

Lecture 1, 7/21/2005Natural Language Processing11 Sentence Alignment (c’ntd)  There are several sentence alignment algorithms: Align (Gale & Church): Aligns sentences based on their character length (shorter sentences tend to have shorter translations then longer sentences). Works astonishingly well Char-align: (Church): Aligns based on shared character sequences. Works fine for similar languages or technical domains K-Vec (Fung & Church): Induces a translation lexicon from the parallel texts based on the distribution of foreign-English word pairs.

Lecture 1, 7/21/2005Natural Language Processing12 Computing Translation Probabilities  Given a parallel corpus we can estimate P(e | f) The maximum likelihood estimation of P(e | f) is: freq(e,f)/freq(f)  Way too specific to get any reasonable frequencies! Vast majority of unseen data will have zero counts!  P(e | f ) could be re-defined as:  Problem: The English words maximizing P(e | f ) might not result in a readable sentence

Lecture 1, 7/21/2005Natural Language Processing13 Computing Translation Probabilities (c’tnd)  We can account for adequacy: each foreign word translates into its most likely English word  We cannot guarantee that this will result in a fluent English sentence  Solution: transform P(e | f) with Bayes’ rule: P(e | f) = P(e) P(f | e) / P(f)  P(f | e) accounts for adequacy  P(e) accounts for fluency

Lecture 1, 7/21/2005Natural Language Processing14 Decoding  The decoder combines the evidence from P(e) and P(f | e) to find the sequence e that is the best translation:  The choice of word e’ as translation of f’ depends on the translation probability P(f’ | e’) and on the context, i.e. other English words preceding e’

Lecture 1, 7/21/2005Natural Language Processing15 Noisy Channel Model for Translation

Lecture 1, 7/21/2005Natural Language Processing16 Translation Modeling  Determines the probability that the foreign word f is a translation of the English word e  How to compute P(f | e) from a parallel corpus?  Statistical approaches rely on the co-occurrence of e and f in the parallel data: If e and f tend to co-occur in parallel sentence pairs, they are likely to be translations of one another

Lecture 1, 7/21/2005Natural Language Processing17 Finding Translations in a Parallel Corpus  Into which foreign words f,..., f’ does e translate?  Commonly, four factors are used: How often do e and f co-occur? (translation) How likely is a word occurring at position i to translate into a word occurring at position j? (distortion) For example: English is a verb-second language, whereas German is a verb-final language How likely is e to translate into more than one word? (fertility) For example: defeated can translate into eine Niederlage erleiden How likely is a foreign word to be spuriously generated? (null translation)

Lecture 1, 7/21/2005Natural Language Processing18 Translation Steps

Lecture 1, 7/21/2005Natural Language Processing19 IBM Models 1–5  Model 1: Bag of words Unique local maxima Efficient EM algorithm (Model 1–2)  Model 2: General alignment:  Model 3: fertility: n(k | e) No full EM, count only neighbors (Model 3–5) Deficient (Model 3–4)  Model 4: Relative distortion, word classes  Model 5: Extra variables to avoid deficiency

Lecture 1, 7/21/2005Natural Language Processing20 How to get Word Alignments  Word alignment: a mapping between the source words and the target words in a set of parallel sentences.  Restriction: each foreign word comes from exactly 1 English word  Advantage: represent an alignment by the index of the English word that the French word comes from  Alignment above is thus 2,3,4,5,6,6,6

Lecture 1, 7/21/2005Natural Language Processing21 One addition: spurious words  A word in the foreign sentence  That doesn’t align with any word in the English sentence  Is called a spurious word.  We model these by pretending they are generated by an English word e 0 :

Lecture 1, 7/21/2005Natural Language Processing22 More sophisticated models of alignment

Lecture 1, 7/21/2005Natural Language Processing23 Computing word alignments: IBM Model 1  A word alignment algorithm gives us P(F,E)  We want this to train our phrase probabilities  (f j |e i ) as part of P(F|E)  But a word-alignment algorithm can also be part of a mini-translation model itself.

Lecture 1, 7/21/2005Natural Language Processing24 IBM Models 1–5  Model 1: Bag of words Unique local maxima Efficient EM algorithm (Model 1–2)  Model 2: General alignment:  Model 3: fertility: n(k | e) No full EM, count only neighbors (Model 3–5) Deficient (Model 3–4)  Model 4: Relative distortion, word classes  Model 5: Extra variables to avoid deficiency

Lecture 1, 7/21/2005Natural Language Processing25 IBM Model 1  Given an English sentence e 1... e l and a foreign sentence f 1... f m  We want to find the ’best’ alignment a, where a is a set pairs of the form {(i, j),..., (i’, j’)}, 0<= i, i’ <= l and 1<= j, j’<= m  Note that if (i, j), (i’, j) are in a, then i equals i’, i.e. no many-to-one alignments are allowed  Note we add a spurious NULL word to the English sentence at position 0  In total there are (l + 1) m different alignments A  Allowing for many-to-many alignments results in (2 l ) m possible alignments A

Lecture 1, 7/21/2005Natural Language Processing26 IBM Model 1  Simplest of the IBM models  Does not consider word order (bag-of-words approach)  Does not model one-to-many alignments  Computationally inexpensive  Useful for parameter estimations that are passed on to more elaborate models

Lecture 1, 7/21/2005Natural Language Processing27 IBM Model 1  Translation probability in terms of alignments: where: and:

Lecture 1, 7/21/2005Natural Language Processing28 IBM Model 1  We want to find the most likely alignment:  Since P(a | e) is the same for all a:  Problem: We still have to enumerate all alignments

Lecture 1, 7/21/2005Natural Language Processing29 IBM Model 1  Since P(f j | e i ) is independent from P(f j’ | e i’ ) we can find the maximum alignment by looking at the individual translation probabilities only  Let, then for each a j :  The best alignment can computed in a quadratic number of steps: (l+1 x m)

Lecture 1, 7/21/2005Natural Language Processing30 Computing Model 1 Parameters  How to compute translation probabilities for model 1 from a parallel corpus?  Step 1: Determine candidates. For each English word e collect all foreign words f that co-occur at least once with e  Step 2: Initialize P(f | e) uniformly, i.e. P(f | e) = 1/(no of co-occurring foreign words)

Lecture 1, 7/21/2005Natural Language Processing31 Computing Model 1 Parameters  Step 3: Iteratively refine translation probablities: 1 for n iterations 2 set tc to zero 3 for each sentence pair (e,f) of lengths (l,m) 4 for j=1 to m 5 total=0; 6 for i=1 to l 7 total += P(f j | e i ); 8 for i=1 to l 9 tc(f j | e i ) += P(f j | e i )/total; 10 for each word e 11 total=0; 12 for each word f s.t. tc(f | e) is defined 13 total += tc(f | e); 14 for each word f s.t. tc(f | e) is defined 15 P(f | e) = tc(f | e)/total;

Lecture 1, 7/21/2005Natural Language Processing32 IBM Model 1 Example  Parallel ‘corpus’: the dog :: le chien the cat :: le chat  Step 1+2 (collect candidates and initialize uniformly): P(le | the) = P(chien | the) = P(chat | the) = 1/3 P(le | dog) = P(chien | dog) = P(chat | dog) = 1/3 P(le | cat) = P(chien | cat) = P(chat | cat) = 1/3 P(le | NULL) = P(chien | NULL) = P(chat | NULL) = 1/3

Lecture 1, 7/21/2005Natural Language Processing33 IBM Model 1 Example  Step 3: Iterate  NULL the dog :: le chien j=1 total = P(le | NULL)+P(le | the)+P(le | dog)= 1 tc(le | NULL) += P(le | NULL)/1 = 0 +=.333/1 = tc(le | the) += P(le | the)/1= 0 +=.333/1 = tc(le | dog) += P(le | dog)/1= 0 +=.333/1 = j=2 total = P(chien | NULL)+P(chien | the)+P(chien | dog)=1 tc(chien | NULL) += P(chien | NULL)/1 = 0 +=.333/1 = tc(chien | the) += P(chien | the)/1 = 0 +=.333/1 = tc(chien | dog) += P(chien | dog)/1 = 0 +=.333/1 = 0.333

Lecture 1, 7/21/2005Natural Language Processing34 IBM Model 1 Example  NULL the cat :: le chat j=1 total = P(le | NULL)+P(le | the)+P(le | cat)=1 tc(le | NULL) += P(le | NULL)/1 = =.333/1 = tc(le | the) += P(le | the)/1 = =.333/1 = tc(le | cat) += P(le | cat)/1 = 0 +=.333/1 = j=2 total = P(chien | NULL)+P(chien | the)+P(chien | dog)=1 tc(chat | NULL) += P(chat | NULL)/1 = 0 +=.333/1 = tc(chat | the) += P(chat | the)/1 = 0 +=.333/1 = tc(chat | cat) += P(chat | dog)/1 = 0 +=.333/1 = 0.333

Lecture 1, 7/21/2005Natural Language Processing35 IBM Model 1 Example  Re-compute translation probabilities total(the) = tc(le | the) + tc(chien | the) + tc(chat | the) = = P(le | the) = tc(le | the)/total(the) = / = 0.5 P(chien | the) = tc(chien | the)/total(the) = 0.333/ P(chat | the) = tc(chat | the)/total(the) = 0.333/ total(dog) = tc(le | dog) + tc(chien | dog) = P(le | dog) = tc(le | dog)/total(dog) = / = 0.5 P(chien | dog) = tc(chien | dog)/total(dog) = / = 0.5

Lecture 1, 7/21/2005Natural Language Processing36 IBM Model 1 Example  Iteration 2:  NULL the dog :: le chien j=1 total = P(le | NULL)+P(le | the)+P(le | dog)= 1.5 = = 1.5 tc(le | NULL) += P(le | NULL)/1 = 0 +=.5/1.5 = tc(le | the) += P(le | the)/1= 0 +=.5/1.5 = tc(le | dog) += P(le | dog)/1= 0 +=.5/1.5 = j=2 total = P(chien | NULL)+P(chien | the)+P(chien | dog)=1 = = 1 tc(chien | NULL) += P(chien | NULL)/1 = 0 +=.25/1 = 0.25 tc(chien | the) += P(chien | the)/1 = 0 +=.25/1 = 0.25 tc(chien | dog) += P(chien | dog)/1 = 0 +=.5/1 = 0.5

Lecture 1, 7/21/2005Natural Language Processing37 IBM Model 1 Example  NULL the cat :: le chat j=1 total = P(le | NULL)+P(le | the)+P(le | cat)= 1.5 = = 1.5 tc(le | NULL) += P(le | NULL)/1 = =.5/1 = tc(le | the) += P(le | the)/1= =.5/1 = tc(le | cat) += P(le | cat)/1= 0 +=.5/1 = 0.5 j=2 total = P(chat | NULL)+P(chat | the)+P(chat | cat)=1 = = 1 tc(chat | NULL) += P(chat | NULL)/1 = 0 +=.25/1 = 0.25 tc(chat | the) += P(chat | the)/1 = 0 +=.25/1 = 0.25 tc(chat | cat) += P(chat | cat)/1 = 0 +=.5/1 = 0.5

Lecture 1, 7/21/2005Natural Language Processing38 IBM Model 1 Example  Re-compute translations (iteration 2): total(the) = tc(le | the) + tc(chien | the) + tc(chat | the) = = P(le | the) = tc(le | the)/total(the) =.833 / = P(chien | the) = tc(chien | the)/total(the) = 0.25/1.333 = P(chat | the) = tc(chat | the)/total(the) = 0.25/1.333 = total(dog) = tc(le | dog) + tc(chien | dog) = = P(le | dog) = tc(le | dog)/total(dog) = / = 0.4 P(chien | dog) = tc(chien | dog)/total(dog) = 0.5 / = 0.6

Lecture 1, 7/21/2005Natural Language Processing39 IBM Model 1Example  After 5 iterations: P(le | NULL) = P(chien | NULL) = P(chat | NULL) = P(le | the) = P(chien | the) = P(chat | the) = P(le | dog) = P(chien | dog) = P(le | cat) = P(chat | cat) =

Lecture 1, 7/21/2005Natural Language Processing40 IBM Model 1 Recap  IBM Model 1 allows for an efficient computation of translation probabilities  No notion of fertility, i.e., it’s possible that the same English word is the best translation for all foreign words  No positional information, i.e., depending on the language pair, there might be a tendency that words occurring at the beginning of the English sentence are more likely to align to words at the beginning of the foreign sentence

Lecture 1, 7/21/2005Natural Language Processing41 IBM Model 3  IBM Model 3 offers two additional features compared to IBM Model 1: How likely is an English word e to align to k foreign words (fertility)? Positional information (distortion), how likely is a word in position i to align to a word in position j?

Lecture 1, 7/21/2005Natural Language Processing42 IBM Model 3: Fertility  The best Model 1 alignment could be that a single English word aligns to all foreign words  This is clearly not desirable and we want to constrain the number of words an English word can align to  Fertility models a probability distribution that word e aligns to k words: n(k,e)  Consequence: translation probabilities cannot be computed independently of each other anymore  IBM Model 3 has to work with full alignments, note there are up to (l+1) m different alignments

Lecture 1, 7/21/2005Natural Language Processing43 IBM Model 1 + Model 3  Iterating over all possible alignments is computationally infeasible  Solution: Compute the best alignment with Model 1 and change some of the alignments to generate a set of likely alignments (pegging)  Model 3 takes this restricted set of alignments as input

Lecture 1, 7/21/2005Natural Language Processing44 Pegging  Given an alignment a we can derive additional alignments from it by making small changes: Changing a link (j,i) to (j,i’) Swapping a pair of links (j,i) and (j’,i’) to (j,i’) and (j’,i)  The resulting set of alignments is called the neighborhood of a

Lecture 1, 7/21/2005Natural Language Processing45 IBM Model 3: Distortion  The distortion factor determines how likely it is that an English word in position i aligns to a foreign word in position j, given the lengths of both sentences: d(j | i, l, m)  Note, positions are absolute positions

Lecture 1, 7/21/2005Natural Language Processing46 Deficiency  Problem with IBM Model 3: It assigns probability mass to impossible strings Well formed string: “This is possible” Ill-formed but possible string: “This possible is” Impossible string:  Impossible strings are due to distortion values that generate different words at the same position  Impossible strings can still be filtered out in later stages of the translation process

Lecture 1, 7/21/2005Natural Language Processing47 Limitations of IBM Models  Only 1-to-N word mapping  Handling fertility-zero words (difficult for decoding)  Almost no syntactic information Word classes Relative distortion  Long-distance word movement  Fluency of the output depends entirely on the English language model

Lecture 1, 7/21/2005Natural Language Processing48 Decoding  How to translate new sentences?  A decoder uses the parameters learned on a parallel corpus Translation probabilities Fertilities Distortions  In combination with a language model the decoder generates the most likely translation  Standard algorithms can be used to explore the search space (A*, greedy searching, …)  Similar to the traveling salesman problem

Lecture 1, 7/21/2005Natural Language Processing49 IBM Model 1

Lecture 1, 7/21/2005Natural Language Processing50 IBM Model 1

Lecture 1, 7/21/2005Natural Language Processing51 How does the generative story assign P(F|E) for a Spanish sentence F?  Terminology:  Suppose we had done steps 1 and 2, I.e. we already knew the Spanish length J and the alignment A (and English source E):

Lecture 1, 7/21/2005Natural Language Processing52 Let’s formalize steps 1 and 2  We want P(A|E) of an alignment A (of length J) given an English sentence E  IBM Model 1 makes the (very) simplifying assumption that each alignment is equally likely.  How many possible alignments are there between English sentence of length I and Spanish sentence of length J?  Hint: Each Spanish word must come from one of the English source words (or the NULL word)  (I+1) J  Let’s assume probability of choosing length J is small constant epsilon

Lecture 1, 7/21/2005Natural Language Processing53 Model 1 continued  Prob of choosing a length and then one of the possible alignments:  Combining with step 3:  The total probability of a given foreign sentence F:

Lecture 1, 7/21/2005Natural Language Processing54 Decoding  How do we find the best A?

Lecture 1, 7/21/2005Natural Language Processing55 Training alignment probabilities  Step 1: get a parallel corpus Hansards  Canadian parliamentary proceedings, in French and English  Hong Kong Hansards: English and Chinese  Step 2: sentence alignment  Step 3: use EM (Expectation Maximization) to train word alignments

Lecture 1, 7/21/2005Natural Language Processing56 Step 1: Parallel corpora EnglishGerman Diverging opinions about planned tax reform Unterschiedliche Meinungen zur geplanten Steuerreform The discussion around the envisaged major tax reform continues. Die Diskussion um die vorgesehene grosse Steuerreform dauert an. The FDP economics expert, Graf Lambsdorff, today came out in favor of advancing the enactment of significant parts of the overhaul, currently planned for Der FDP - Wirtschaftsexperte Graf Lambsdorff sprach sich heute dafuer aus, wesentliche Teile der fuer 1999 geplanten Reform vorzuziehen.  Example from DE-News (8/1/1996) Slide from Christof Monz

Lecture 1, 7/21/2005Natural Language Processing57 Step 2: Sentence Alignment The old man is happy. He has fished many times. His wife talks to him. The fish are jumping. The sharks await. Intuition: - use length in words or chars - together with dynamic programming - or use a simpler MT model El viejo está feliz porque ha pescado muchos veces. Su mujer habla con é l. Los tiburones esperan. Slide from Kevin Knight

Lecture 1, 7/21/2005Natural Language Processing58 Sentence Alignment 1. The old man is happy. 2. He has fished many times. 3. His wife talks to him. 4. The fish are jumping. 5. The sharks await. El viejo está feliz porque ha pescado muchos veces. Su mujer habla con él. Los tiburones esperan. Slide from Kevin Knight

Lecture 1, 7/21/2005Natural Language Processing59 Sentence Alignment 1. The old man is happy. 2. He has fished many times. 3. His wife talks to him. 4. The fish are jumping. 5. The sharks await. El viejo está feliz porque ha pescado muchos veces. Su mujer habla con él. Los tiburones esperan. Slide from Kevin Knight

Lecture 1, 7/21/2005Natural Language Processing60 Sentence Alignment 1. The old man is happy. He has fished many times. 2. His wife talks to him. 3. The sharks await. El viejo está feliz porque ha pescado muchos veces. Su mujer habla con él. Los tiburones esperan. Note that unaligned sentences are thrown out, and sentences are merged in n-to-m alignments (n, m > 0). Slide from Kevin Knight

Lecture 1, 7/21/2005Natural Language Processing61 Step 3: word alignments  It turns out we can bootstrap alignments  From a sentence-aligned bilingual corpus  We use is the Expectation-Maximization or EM algorithm

Lecture 1, 7/21/2005Natural Language Processing62 EM for training alignment probs … la maison … la maison bleue … la fleur … … the house … the blue house … the flower … All word alignments equally likely All P(french-word | english-word) equally likely Slide from Kevin Knight

Lecture 1, 7/21/2005Natural Language Processing63 EM for training alignment probs … la maison … la maison bleue … la fleur … … the house … the blue house … the flower … “la” and “the” observed to co-occur frequently, so P(la | the) is increased. Slide from Kevin Knight

Lecture 1, 7/21/2005Natural Language Processing64 EM for training alignment probs … la maison … la maison bleue … la fleur … … the house … the blue house … the flower … “house” co-occurs with both “la” and “maison”, but P(maison | house) can be raised without limit, to 1.0, while P(la | house) is limited because of “the” (pigeonhole principle) Slide from Kevin Knight

Lecture 1, 7/21/2005Natural Language Processing65 EM for training alignment probs … la maison … la maison bleue … la fleur … … the house … the blue house … the flower … settling down after another iteration Slide from Kevin Knight

Lecture 1, 7/21/2005Natural Language Processing66 EM for training alignment probs … la maison … la maison bleue … la fleur … … the house … the blue house … the flower … Inherent hidden structure revealed by EM training! For details, see: Section in the chapter “A Statistical MT Tutorial Workbook” (Knight, 1999). “The Mathematics of Statistical Machine Translation” (Brown et al, 1993) Software: GIZA++ Slide from Kevin Knight

Lecture 1, 7/21/2005Natural Language Processing67 Statistical Machine Translation … la maison … la maison bleue … la fleur … … the house … the blue house … the flower … P(juste | fair) = P(juste | correct) = P(juste | right) = … new French sentence Possible English translations, to be rescored by language model Slide from Kevin Knight

Lecture 1, 7/21/2005Natural Language Processing68 A more complex model: IBM Model 3 Brown et al., 1993 Mary did not slap the green witch Mary not slap slap slap the green witch n(3|slap) Maria no d ió una bofetada a la bruja verde d(j|i) Mary not slap slap slap NULL the green witch P-Null Maria no d ió una bofetada a la verde bruja t(la|the) Generative approach: Probabilities can be learned from raw bilingual text.

Lecture 1, 7/21/2005Natural Language Processing69 How do we evaluate MT? Human tests for fluency  Rating tests: Give the raters a scale (1 to 5) and ask them to rate Or distinct scales for  Clarity, Naturalness, Style Or check for specific problems  Cohesion (Lexical chains, anaphora, ellipsis)  Hand-checking for cohesion.  Well-formedness  5-point scale of syntactic correctness  Comprehensibility tests Noise test Multiple choice questionnaire  Readability tests cloze

Lecture 1, 7/21/2005Natural Language Processing70 How do we evaluate MT? Human tests for fidelity  Adequacy Does it convey the information in the original? Ask raters to rate on a scale  Bilingual raters: give them source and target sentence, ask how much information is preserved  Monolingual raters: give them target + a good human translation  Informativeness Task based: is there enough info to do some task? Give raters multiple-choice questions about content

Lecture 1, 7/21/2005Natural Language Processing71 Evaluating MT: Problems  Asking humans to judge sentences on a 5-point scale for 10 factors takes time and money  We can’t build language engineering systems if we can only evaluate them once every quarter!!!!  We need a metric that we can run every time we change our algorithm.  It would be OK if it wasn’t perfect, but just tended to correlate with the expensive human metrics, which we could still run in quarterly. Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing72 Automatic evaluation  Miller and Beebe-Center (1958)  Assume we have one or more human translations of the source passage  Compare the automatic translation to these human translations Bleu NIST Meteor Precision/Recall

Lecture 1, 7/21/2005Natural Language Processing73 BiLingual Evaluation Understudy (BLEU —Papineni, 2001)  Automatic Technique, but ….  Requires the pre-existence of Human (Reference) Translations  Approach: Produce corpus of high-quality human translations Judge “closeness” numerically (word-error rate) Compare n-gram matches between candidate translation and 1 or more reference translations Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing74 Reference (human) translation: The U.S. island of Guam is maintaining a high state of alert after the Guam airport and its offices both received an from someone calling himself the Saudi Arabian Osama bin Laden and threatening a biological/chemical attack against public places such as the airport. Machine translation: The American [?] international airport and its the office all receives one calls self the sand Arab rich business [?] and so on electronic mail, which sends out ; The threat will be able after public place and so on the airport to start the biochemistry attack, [?] highly alerts after the maintenance. BLEU Evaluation Metric (Papineni et al, ACL-2002) N-gram precision (score is between 0 & 1) –What percentage of machine n-grams can be found in the reference translation? –An n-gram is an sequence of n words –Not allowed to use same portion of reference translation twice (can’t cheat by typing out “the the the the the”) Brevity penalty –Can’t just type out single word “the” (precision 1.0!) *** Amazingly hard to “game” the system (i.e., find a way to change machine output so that BLEU goes up, but quality doesn’t) Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing75 Reference (human) translation: The U.S. island of Guam is maintaining a high state of alert after the Guam airport and its offices both received an from someone calling himself the Saudi Arabian Osama bin Laden and threatening a biological/chemical attack against public places such as the airport. Machine translation: The American [?] international airport and its the office all receives one calls self the sand Arab rich business [?] and so on electronic mail, which sends out ; The threat will be able after public place and so on the airport to start the biochemistry attack, [?] highly alerts after the maintenance. BLEU Evaluation Metric (Papineni et al, ACL-2002) BLEU4 formula (counts n-grams up to length 4) exp (1.0 * log p * log p * log p * log p4 – max(words-in-reference / words-in-machine – 1, 0) p1 = 1-gram precision P2 = 2-gram precision P3 = 3-gram precision P4 = 4-gram precision Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing76 Reference translation 1: The U.S. island of Guam is maintaining a high state of alert after the Guam airport and its offices both received an from someone calling himself the Saudi Arabian Osama bin Laden and threatening a biological/chemical attack against public places such as the airport. Reference translation 3: The US International Airport of Guam and its office has received an from a self-claimed Arabian millionaire named Laden, which threatens to launch a biochemical attack on such public places as airport. Guam authority has been on alert. Reference translation 4: US Guam International Airport and its office received an from Mr. Bin Laden and other rich businessman from Saudi Arabia. They said there would be biochemistry air raid to Guam Airport and other public places. Guam needs to be in high precaution about this matter. Reference translation 2: Guam International Airport and its offices are maintaining a high state of alert after receiving an that was from a person claiming to be the wealthy Saudi Arabian businessman Bin Laden and that threatened to launch a biological and chemical attack on the airport and other public places. Machine translation: The American [?] international airport and its the office all receives one calls self the sand Arab rich business [?] and so on electronic mail, which sends out ; The threat will be able after public place and so on the airport to start the biochemistry attack, [?] highly alerts after the maintenance. Multiple Reference Translations Reference translation 1: The U.S. island of Guam is maintaining a high state of alert after the Guam airport and its offices both received an from someone calling himself the Saudi Arabian Osama bin Laden and threatening a biological/chemical attack against public places such as the airport. Reference translation 3: The US International Airport of Guam and its office has received an from a self-claimed Arabian millionaire named Laden, which threatens to launch a biochemical attack on such public places as airport. Guam authority has been on alert. Reference translation 4: US Guam International Airport and its office received an from Mr. Bin Laden and other rich businessman from Saudi Arabia. They said there would be biochemistry air raid to Guam Airport and other public places. Guam needs to be in high precaution about this matter. Reference translation 2: Guam International Airport and its offices are maintaining a high state of alert after receiving an that was from a person claiming to be the wealthy Saudi Arabian businessman Bin Laden and that threatened to launch a biological and chemical attack on the airport and other public places. Machine translation: The American [?] international airport and its the office all receives one calls self the sand Arab rich business [?] and so on electronic mail, which sends out ; The threat will be able after public place and so on the airport to start the biochemistry attack, [?] highly alerts after the maintenance. Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing77 BLEU in Action 枪手被警方击毙。 (Foreign Original) the gunman was shot to death by the police. (Reference Translation) the gunman was police kill. #1 wounded police jaya of #2 the gunman was shot dead by the police. #3 the gunman arrested by police kill. #4 the gunmen were killed. #5 the gunman was shot to death by the police. #6 gunmen were killed by police ?SUB>0 ?SUB>0 #7 al by the police. #8 the ringer is killed by the police. #9 police killed the gunman. #10 Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing78 BLEU in Action 枪手被警方击毙。 (Foreign Original) the gunman was shot to death by the police. (Reference Translation) the gunman was police kill. #1 wounded police jaya of #2 the gunman was shot dead by the police. #3 the gunman arrested by police kill. #4 the gunmen were killed. #5 the gunman was shot to death by the police. #6 gunmen were killed by police ?SUB>0 ?SUB>0 #7 al by the police. #8 the ringer is killed by the police. #9 police killed the gunman. #10 green = 4-gram match (good!) red = word not matched (bad!) Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing79 Bleu Comparison Chinese-English Translation Example: Candidate 1: It is a guide to action which ensures that the military always obeys the commands of the party. Candidate 2: It is to insure the troops forever hearing the activity guidebook that party direct. Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing80 How Do We Compute Bleu Scores?  Intuition: “What percentage of words in candidate occurred in some human translation?”  Proposal: count up # of candidate translation words (unigrams) # in any reference translation, divide by the total # of words in # candidate translation  But can’t just count total # of overlapping N-grams! Candidate: the the the the the the Reference 1: The cat is on the mat  Solution: A reference word should be considered exhausted after a matching candidate word is identified. Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing81 “Modified n-gram precision”  For each word compute: (1) total number of times it occurs in any single reference translation (2) number of times it occurs in the candidate translation  Instead of using count #2, use the minimum of #2 and #2, I.e. clip the counts at the max for the reference transcription  Now use that modified count.  And divide by number of candidate words. Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing82 Modified Unigram Precision: Candidate #1 Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. It(1) is(1) a(1) guide(1) to(1) action(1) which(1) ensures(1) that(2) the(4) military(1) always(1) obeys(0) the commands(1) of(1) the party(1) What’s the answer???17/18 Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing83 Modified Unigram Precision: Candidate #2 It(1) is(1) to(1) insure(0) the(4) troops(0) forever(1) hearing(0) the activity(0) guidebook(0) that(2) party(1) direct(0) What’s the answer????8/14 Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing84 Modified Bigram Precision: Candidate #1 It is(1) is a(1) a guide(1) guide to(1) to action(1) action which(0) which ensures(0) ensures that(1) that the(1) the military(1) military always(0) always obeys(0) obeys the(0) the commands(0) commands of(0) of the(1) the party(1) What’s the answer???? 10/17 Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing85 Modified Bigram Precision: Candidate #2 Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. It is(1) is to(0) to insure(0) insure the(0) the troops(0) troops forever(0) forever hearing(0) hearing the(0) the activity(0) activity guidebook(0) guidebook that(0) that party(0) party direct(0) What’s the answer????1/13 Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing86 Catching Cheaters Reference 1: The cat is on the mat Reference 2: There is a cat on the mat the(2) the the the(0) the(0) the(0) the(0) What’s the unigram answer?2/7 What’s the bigram answer?0/7 Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing87 Bleu distinguishes human from machine translations Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing88 Bleu problems with sentence length  Candidate: of the  Solution: brevity penalty; prefers candidates translations which are same length as one of the references Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. Problem: modified unigram precision is 2/2, bigram 1/1! Slide from Bonnie Dorr

Lecture 1, 7/21/2005Natural Language Processing89 BLEU Tends to Predict Human Judgments slide from G. Doddington (NIST) (variant of BLEU)

Lecture 1, 7/21/2005Natural Language Processing90 Summary  Intro and a little history  Language Similarities and Divergences  Four main MT Approaches Transfer Interlingua Direct Statistical  Evaluation

Lecture 1, 7/21/2005Natural Language Processing91 Classes  LINGUIST 139M/239M. Human and Machine Translation. (Martin Kay)  CS 224N. Natural Language Processing (Chris Manning)

Lecture 1, 7/21/2005Natural Language Processing92 Intuition of phrase-based translation (Koehn et al. 2003)  Generative story has three steps 1) Group words into phrases 2) Translate each phrase 3) Move the phrases around

Lecture 1, 7/21/2005Natural Language Processing93 Generative story again 1) Group English source words into phrases e 1, e 2, …, e n 2) Translate each English phrase e i into a Spanish phrase f j. The probability of doing this is  (f j |e i ) 3) Then (optionally) reorder each Spanish phrase We do this with a distortion probability A measure of distance between positions of a corresponding phrase in the 2 lgs. “What is the probability that a phrase in position X in the English sentences moves to position Y in the Spanish sentence?”

Lecture 1, 7/21/2005Natural Language Processing94 Distortion probability  The distortion probability is parameterized by a i -b i-1 Where a i is the start position of the foreign (Spanish) phrase generated by the ith English phrase e i. And b i-1 is the end position of the foreign (Spanish) phrase generated by the I-1th English phrase e i-1.  We’ll call the distortion probability d(a i -b i-1 ).  And we’ll have a really stupid model: d(a i -b i-1 ) =  |ai-bi-1| Where  is some small constant.

Lecture 1, 7/21/2005Natural Language Processing95 Final translation model for phrase- based MT  Let’s look at a simple example with no distortion

Lecture 1, 7/21/2005Natural Language Processing96 Phrase-based MT  Language model P(E)  Translation model P(F|E) Model How to train the model  Decoder: finding the sentence E that is most probable

Lecture 1, 7/21/2005Natural Language Processing97 Training P(F|E)  What we mainly need to train is  (f j |e i )  Suppose we had a large bilingual training corpus A bitext In which each English sentence is paired with a Spanish sentence  And suppose we knew exactly which phrase in Spanish was the translation of which phrase in the English  We call this a phrase alignment  If we had this, we could just count-and-divide:

Lecture 1, 7/21/2005Natural Language Processing98 But we don’t have phrase alignments  What we have instead are word alignments:

Lecture 1, 7/21/2005Natural Language Processing99 Getting phrase alignments  To get phrase alignments: 1) We first get word alignments 2) Then we “symmetrize” the word alignments into phrase alignments