A Statistical Approach to Machine Translation ( Brown et al CL ) POSTECH, NLP lab 김 지 협

POSTECH NLP LAB. 2Contents oIntroduction oLanguage Model oTranslation Model oSearching oParameter Estimation oExperiment oPlan

POSTECH NLP LAB. 3Introduction oIn 1949 m Warren Weaver : Statistical methods ( Information theory ) m Not used : Slow computing power & small machine readable texts oThese days m Fast computers & large machine readable corpora m Successful approach to speech recognition m Give them a chance in machine translation oThe job of a Translation : Arts m Don’t hope to reach it m Only the translation of individual sentences m Many translations / a given sentence m The choice among them : a matter of tastes

POSTECH NLP LAB. 4Continued oHow to view a translation in this paper? m Every sentences in one language is a possible translation of any sentence in the other. m Assign to every pair of sentences (S, T) a probability m Pr ( T | S ) : Probability producing T in the target language, given S in the source language ex) Pr(Le matin je me brosse les dents | President Lincoln was a good layers) < Pr(Le president Lincoln etait un bon avocat | President Lincoln was a good layers) oWhat is a translation? m Using Bayes’ theorem

POSTECH NLP LAB. 5Continued oStatistical translation system Source Language Model Decoder Translation Model ST T

POSTECH NLP LAB. 6 Language Model oModeling m A given a word string, m So many histories : n - gram model used m The power of a tri-gam model : Bag Translation Experiment Scheme Experiment (Fig. 2): 38 sentences with fewer than 11 words If we had handled longer sentences a order proper s’: a proper order s : order proper a P(s’) > P(s) arrangement

POSTECH NLP LAB. 7 Translation Model oTerminology (alignment, fertility, distortion) m Alignment (producing) Total possible connections : l x m Set of alignments of (S, T) : 2 lm alignments ex) P. 267, Fig. 1, 2, 3 ( Alignment Examples ) Think about only Fig. 1 in this paper Ss1s2:slSs1s2:sl T t 1 t 2 : t m

POSTECH NLP LAB. 8Continued m Fertility: Number of target words that an source word produce m Distortion: Target word far from the source word that produced it m Notation ( Le chien est battu par Jean|John(6) does beat(3,4) the(1) dog(2) ) Jean battu est chien par Le John dog the beat does

POSTECH NLP LAB. 9Continued o2 - Models m Without loss m Model 1 all connections assigned equal probability translation probability P(t|s) m Model 2 fertility probability P(n|s) distortion probability P(j|i, m) Ss1si:slSs1si:sl T t 1 t j t m a1a1 amam ajaj

POSTECH NLP LAB. 10Searching oAlgorithm m Searching for the sentence S that maximizes Pr(S)Pr(T|S) m Too many sentences to try : use suboptimal search m Use a variant of the Stack Search maintain a list of partial alignment hypotheses initially, one entry : (Jean aime Marie | * ) search proceeds by iterations, extending most promising entries cut some hypothesis, using threshhold ex) (Jean aime Marie | John(1) *), (Jean aime Marie | * loves(2) *), (Jean aime Marie | Mary(3) *) end : a complete alignment on the list that is significantly more promising than any of the incomplete alignment

POSTECH NLP LAB. 11 Parameter Estimation oTraining Corpus m The proceedings of the Canadian parliament (100 million words of English texts and corresponding French texts) m Extract 3 million pairs of sentences : a statistical algorithm based on sentence length ( 99% accurate ) m Language model: use a bi-gram from English texts m Translation model from unaligned pairs of sentences so, we can’t count analogous to the situation in speech recognition using EM algorithm oEstimation Steps m step 1 : translation probability from Model 1 m step 2 : fertility, distortion probability from Model 2

POSTECH NLP LAB. 12Experiment o1st Exp. to estimate parameters for the translation model m 9,000 English vocabulary & 9,000 French vocabulary used m So, we have 81,000,000 parameters m Training corpus: 40,000 pairs of sentences ( 800,000 words in each ) m Result : Fig. 4, 5, 6

POSTECH NLP LAB. 13Continued o2nd Exp. to translate from French to English m Translation Model parameter estimation 1,000 English vocabulary & 1,700 French vocabulary used 17,000,000 parameters from 117,000 pairs of sentences m Language Model parameter estimation bi-gram, 570,000 sentences from English texts (12,000,000 words) not restricted to sentences covered by 1,000 vocabulary m Search 73 new French sentences from elsewhere in the Hansard data m Result 5 categorization (Exact, Alternate, Different, Wrong, Ungrammatical) Fig. 8 Translation Results 776 strokes vs. 1,916 strokes from scratch (60% saved)

POSTECH NLP LAB. 14Plans oSome improvement for Parameter Estimations m Several source words to work together to produce a single target word m Use Tri-gram oMorphologies for French & English

POSTECH NLP LAB. 15