1. Stephan Vogel - Machine Translation1 Machine Translation Word Alignment Stephan Vogel Spring Semester 2011

2. Stephan Vogel - Machine Translation2 Overview lIBM 3: Fertility lIBM 4: Relative Distortion Acknowledgement: These slides are based on slides by Hermann Ney and Franz Josef Och

3. Stephan Vogel - Machine Translation3 Fertility Models lBasic concept: each word in one language can generate multiple words in the other language deseo – I would like übermorgen – the day after tomorrow departed – fuhr ab The same word can generate different number of words -> probability distribution  lAlignment is function -> fertility only on one side lIn my terminology: target words have fertility, i.e. each target word can cover multiple source words lOthers say source word generates multiple target words lSome source words are aligned to NULL word, i.e. NULL word has fertility lMany target words are not aligned, i.e. have fertility 0

4. Stephan Vogel - Machine Translation4 The Generative Story e0e0 e1e1 e2e2 e3e3 e4e4 e5e5 120130 f01 f11 f12 f31 f41 f42 f43 f1f1 f2f2 f3f3 f4f4 f5f5 f6f6 f7f7 fertility generation word generation permutation generation

5. Stephan Vogel - Machine Translation5 Fertility Model Alignment model: Select fertility for each English word: For each English word select a tablet of French words: Select a permutation for the entire sequence of French words: Sum over all realizations:

6. Stephan Vogel - Machine Translation6 Fertility Model: Constraints Fertility bound to alignment: Permutation: French words:

7. Stephan Vogel - Machine Translation7 Fertility Model Decomposition into factors: Apply chain rule to each factor, limit dependencies: Fertility generation (IBM 3,4,5): Word generation (IBM 3,4,5): Permutation generation (only IBM 3): Note: 1/    results from special model for i = 0.

8. Stephan Vogel - Machine Translation8 Fertility Model: Some Issues lPermutation model can not guaranty that p is a permutation -> Words ca be stacked on top of each other -> This leads to deficiency lPosition i = 0 is not a real position -> special alignment and fertility model for the empty word

9. Stephan Vogel - Machine Translation9 Fertility Model: Empty Position lAlignment assumptions for the empty position i = 0 Uniform position distribution for each of the  0 French words generated from e 0 lPlace these French words only after all other words have been placed lAlignment model for the positions aligned to the Empty position: lOne position: lAll positions:

10. Stephan Vogel - Machine Translation10 Fertility Model: Empty Position lFertility model for words generated by e 0, i.e. by empty position lWe assume that each word from f 1 J requires the Empty word with probability [1 – p 0 ] Probability that exactly    from the J words in f 1 J require the Empty word:

11. Stephan Vogel - Machine Translation11 Fertility Model: Generative Process 1. Select a fertility for each position i > 0: 2. Select a fertility for position i = 0: 3. Define the length J for French string: 4. Select a French word for each pair (i  i>=0: 5. Select a position for each pair (i  i>0:

12. Stephan Vogel - Machine Translation12 Fertility Model: Generative Process 1. Select a fertility for each position i > 0: 2. Select a fertility for position i = 0: 3. Define the length J for French string: 4. Select a French word for each pair (i  i>=0: 5. Select a position for each pair (i  i>0:

13. Stephan Vogel - Machine Translation13 Fertility Model: Generative Process 6. Check: if any position was chosen more then once then return FAILURE 7. Select positions in f for the empty position from the remaining vacant positions: Result: Alignment: French words:

14. Stephan Vogel - Machine Translation14 Deficiency lDistortion model for real words is deficient lDistortion model for empty word is non-deficient lDeficiency can be reduced by aligning more words to the empty word lTraining corpus likelihood can be increased by aligning more words with empty word lPlay with p0!

15. Stephan Vogel - Machine Translation15 IBM 4: 1 st Order Distortion Model lIntroduce more detailed dependencies into the alignment (permutation) model lFirst order dependency along e-axis HMM IBM4

16. Stephan Vogel - Machine Translation16 Inverted Alignment lConsider alignments lDependency along I axis: jumps along the J axis lTwo first order models for aligning first word in a set and for aligning remaining words lWe skip the math :-)

17. Stephan Vogel - Machine Translation17 Characteristics of Alignment Models ModelAlignmentFertilityE-stepDeficient IBM1UniformNoExactNo IBM20-orderNoExactNo HMM1-orderNoExactNo IBM30-orderYesApproxYes IBM41-orderYesApproxYes IBM51-orderYesApproxNo

18. Stephan Vogel - Machine Translation18 Consideration: Overfitting lTraining on data has always the danger of overfitting lModel describes training data in too much detail lBut does not perform well on unseen test data lSolution: Smoothing lLexicon: distribute some of the probability mass from seen events to unseen events lfor p( f | e ), do this for each e) lFor unseen e: uniform distribution or ??? lDistortion: interpolate with uniform distribution lFertility: for many languages ‘longer word’ = ‘more content’ lE.g. compounds or agglutinative morphology lTrain a model for fertility given word length and interpolate with lInterpolate fertility estimates based on word frequency: frequent word, use the word model, low frequency word bias towards the length model

19. Stephan Vogel - Machine Translation19 Extension: Using Manual Dictionaries lAdding manual dictionaries lSimple method 1: add as bilingual data lSimple method 2: interpolate manual with trained dictionary lUse constraint GIZA (Gao, Nguyen, Vogel, WMT 2010) lCan put higher weight on word pairs from dictionary (Och, ACL 2000) lNot so simple: “But dictionaries are data too” (Brown et al, HLT 93) lProblem: manual dictionaries do not have inflected form lPossible Solution: lGenerate additional word forms (Vogel and Monson, LREC 04)

20. Stephan Vogel - Machine Translation20 Extension: Using POS lUse POS in distortion model lWe had: lNow we condition of word class of previous aligned target lAvailable in GIZA++ lAutomatic clustering of vocabulary into word classes with mkcls lDefault: 50 classes lUse POS as 2 nd ‘Lexicon’ model (e.g. Zhao et al, ACL 2005) lTrain p( C(f) | C(d ), start with initial model trained with IBM1 just on word classes lAlign sentence pairs using p( C(f) | C(d ) and p( f | e ) lUpdate both distributions from Viterbi path

21. Stephan Vogel - Machine Translation21 And Much More … lAdd fertilities to HMM model lSymmetrize during training: i.e. update lexicon probabilities based on symmetrized alignment lBenefit from shorter sentence pairs lSplit long sentences based on initial alignment and retrain lExtract phrase pairs and add reliable ones to training data lAnd then all the work on discriminative word alignment

22. Stephan Vogel - Machine Translation22 Alignment Results lUnbalanced between wrong and missing -> unbalanced between precision and recall lChinese is harder, many missing links -> low precision lOne direction seems harder: related to which side has more words lAlignment models generate one link per source word AlignmentCorrectWrongMissingPrecisionRecallAER Arabic-English IBM4 S2T202,89872,488134,09773.760.233.7 IBM4 T2S232,840106,441104,15568.669.131.1 Combined244,81489,65292,17873.272.627.1 Chinese-English IBM4 S2T186,620172,865341,18352,9135.457.9 IBM4 T2S299,744151,478228,05966.456.838.8 Combined296,312140.929231,49167.856.138.6

23. Stephan Vogel - Machine Translation23 Unaligned Words AlignmentNULL AlignmentNot Aligned Arabic-English Manual Alignment8.5811.84 IBM4 S2T3.4930.02 IBM4 T2S5.3315.72 Combined5.537.70 Chinese-Engish Manual Alignment7.8011.90 IBM4 S2T5.4623.84 IBM4 T2S6.4134.53 Combined9.8014.64 lNULL Alignment explicit, part of the model; non-aligned happens lThis is serious: alignment model neglects 1/3 of target words lAlignment is very asymmetric, therefore combination

24. Stephan Vogel - Machine Translation24 Alignment Errors for Most Frequent Words (CH-EN)

25. Stephan Vogel - Machine Translation25 Sentence Length Distribution lSentences are often unbalanced lWrong sentence alignment lBad translations lBut also language divergences lMay wanna remove unbalance sentences lSentence length model very weak SL1234567891011121314151617181920 15919344347312116724201 Table: Target sentence length distribution for source sentence length 10

26. Stephan Vogel - Machine Translation26 Summary lWord Alignment Models lAlignment is (mathematically) a function, i.e many source words to 1 target word, but not the other way round lSymmetry by training in both directions lModel IBM1 lword-word probabilities lSimple training with Expectation-Maximization lModel IBM2 lPosition alignment lTraining also with EM lModel HMM lRelative positions (first order model) lTraining with Viterbi or Forward-Backward Algorithm lAlignment errors reflect restrictions in generative alignment models