1. Statistical Machine Translation Alona Fyshe Based on slides from Colin Cherry and Dekang Lin

2. Basic statistics 0 <= P(x) <=1 P(A) Probability that A happens P(A,B) Probabiliy that A and B happen P(A|B) Probability that A happens given that we know B happened

3. Basic statistics Conditional probability

4. Basic Statistics Use definition of conditional probability to derive the chain rule

5. Basic Statistics Bayes Rule

6. Basic Statistics Just remember Definition of cond. prob. Bayes rule Chain rule

7. Goal Translate. Ill use French (F) into English (E) as the running example.

8. Oh, Canada Im Canadian Mandatory French class in school until grade 6 I speak Cereal Box French Gratuit Gagner Chocolat Glaçage Sans gras Sans cholestérol Élevé dans la fibre

9. Oh, Canada

10. Machine Translation Translation is easy for (bilingual) people Process: Read the text in French Understand it Write it down in English

11. Machine Translation Translation is easy for (bilingual) people Process: Read the text in French Understand it Write it down in English

12. Machine Translation Understanding language Writing well formed text Hard tasks for computers The human process is invisible, intangible

13. One approach: Babelfish A rule-based approach to machine translation A 30-year-old feat in Software Eng. Programming knowledge in by hand is difficult and expensive

14. Alternate Approach: Statistics We are trying to model P(E|F) I give you a French sentence You give me back English How are we going to model this? We could use Bayes rule:

15. Alternate Approach: Statistics

16. Why Bayes rule at all? Why not model P(E|F) directly? P(F|E)P(E) decomposition allows us to be sloppy P(E) worries about good English P(F|E) worries about French that matches English The two can be trained independently

17. Crime Scene Analogy F is a crime scene. E is a person who may have committed the crime P(E|F) - look at the scene - who did it? P(E) - who had a motive? (Profiler) P(F|E) - could they have done it? (CSI - transportation, access to weapons, alabi) Some people might have great motives, but no means - you need both!

18. On voit Jon à la télévision good English? P(E)good match to French? P(F|E) Jon appeared in TV. It back twelve saw. In Jon appeared TV. Jon is happy today. Jon appeared on TV. TV appeared on Jon. Jon was not happy. Table borrowed from Jason Eisner

19. On voit Jon à la télévision good English? P(E)good match to French? P(F|E) Jon appeared in TV. It back twelve saw. In Jon appeared TV. Jon is happy today. Jon appeared on TV. TV appeared on Jon. Jon was not happy. Table borrowed from Jason Eisner

20. I speak English good. How are we going to model good English? How do we know these sentences are not good English? Jon appeared in TV. It back twelve saw. In Jon appeared TV. TV appeared on Jon. Je ne parle pas l'anglais.

21. I speak English good. Je ne parle pas l'anglais. These arent English words. It back twelve saw. These are English words, but its jibberish. Jon appeared in TV. appeared in TV isnt proper English

22. I speak English good. Lets say we have a huge collection of documents written in English Like, say, the Internet. It would be a pretty comprehensive list of English words Save for named entities People, places, things Might include some non-English words Speling mitsakes! lol! Could also tell if a phrase is good English

23. Google, is this good English? Jon appeared in TV. Jon appeared 1,800,000 Google results appeared in TV 45,000 Google results appeared on TV 210,000 Google results It back twelve saw. twelve saw 1,100 Google results It back twelve 586 Google results back twelve saw 0 Google results Imperfect counting… why?

24. Google, is this good English? Language is often modeled this way Collect statistics about the frequency of words and phrases N-gram statistics 1-gram = unigram 2-gram = bigram 3-gram = trigram 4-gram = four-gram 5-gram = five-gram

25. Google, is this good English? Seriously, you want to query google for every phrase in the translation? Google created and released a 5-gram data set. Now you can query Google locally (kind of)

26. Language Modeling Whats P(e)? P(English sentence) P(e 1, e 2, e 3 … e i ) Using the chain rule

27. Language Modeling Markov assumption The choice of word e i depends only on the n words before e i Definition of conditional probability

28. Language Modeling

29. Approximate probability using counts Use the n-gram corpus!

30. Language Modeling Use the n-gram corpus! Not surprisingly, given that you love to eat, loving to eat chocolate is more probable (0.177)

31. Language Modeling But what if Then P(e) = 0 Happens even if the sentence is grammatically correct Al Gores pink Hummer was stolen.

32. Language Modeling Smoothing Many techniques Add one smoothing Add one to every count No more zeros, no problems Backoff If P(e 1, e 2, e 3, e 4, e 5 ) = 0 use something related to P(e 1, e 2, e 3, e 4 )

33. Language Modeling Wait… Is this how people generate English sentences? Do you choose your fifth word based on B Admittedly, this is an approximation to process which is both intangible and hard for humans themselves to explain If you disagree, and care to defend yourself, consider a PhD in NLP

34. Back to Translation Anyway, where were we? Oh right… So, weve got P(e), lets talk P(f|e)

35. Where will we get P(F|E)? Cereal boxes in English Same cereal Boxes, in French Machine Learning Magic P(F|E) model

36. Where will we get P(F|E)? Books in English Same books, in French Machine Learning Magic P(F|E) model We call collections stored in two languages parallel corpora or parallel texts Want to update your system? Just add more text!

37. Translated Corpora The Canadian Parliamentary Debates Available in both French and English UN documents Available in Arabic, Chinese, English, French, Russian and Spanish

38. Problem: How are we going to generalize from examples of translations? Ill spend the rest of this lecture telling you: What makes a useful P(F|E) How to obtain the statistics needed for P(F|E) from parallel texts

39. Strategy: Generative Story When modeling P(X|Y): Assume you start with Y Decompose the creation of X from Y into some number of operations Track statistics of individual operations For a new example X,Y: P(X|Y) can be calculated based on the probability of the operations needed to get X from Y

40. What if…? The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux

41. New Information Call this new info a word alignment (A) With A, we can make a good story The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux

42. P(F,A|E) Story null The quick fox jumps over the lazy dog

43. P(F,A|E) Story null The quick fox jumps over the lazy dog f1f2f2 f3f3 …f 10 Simplifying assumption: Choose the length of the French sentence f. All lengths have equal probability

44. P(F,A|E) Story null The quick fox jumps over the lazy dog f1f2f2 f3f3 …f 10 There are (l+1) m = (8+1) 10 possible alignments

45. P(F,A|E) Story null The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux

46. P(F,A|E) Story null The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux

47. Getting P t (f|e) We need numbers for P t (f|e) Example: P t (le|the) Count lines in a large collection of aligned text null The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux null The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux null The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux null The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux null The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux null The quick fox jumps over the lazy dog Le renard rapide saut par - dessus le chien parasseux

48. Where do we get the lines? That sure looked like a lot of monkeys… Remember: some times the information hidden in the text just jumps out at you Well get alignments out of unaligned text by treating the alignment as a hidden variable We infer an A that maxes the prob. of our corpus Generalization of ideas in HMM training: called EM

49. Wheres heaven in Vietnamese? Example borrowed from Jason Eisner

50. English :In the beginning God created the heavens and the earth. Vietnamese :Ban dâu Dúc Chúa Tròi dung nên tròi dât. English :God called the expanse heaven. Vietnamese :Dúc Chúa Tròi dat tên khoang không la tròi. English :… you are this day like the stars of heaven in number. Vietnamese :… các nguoi dông nhu sao trên tròi. Wheres heaven in Vietnamese? Example borrowed from Jason Eisner

51. English :In the beginning God created the heavens and the earth. Vietnamese :Ban dâu Dúc Chúa Tròi dung nên tròi dât. English :God called the expanse heaven. Vietnamese :Dúc Chúa Tròi dat tên khoang không la tròi. English :… you are this day like the stars of heaven in number. Vietnamese :… các nguoi dông nhu sao trên tròi. Wheres heaven in Vietnamese? Example borrowed from Jason Eisner

52. EM: Expectation Maximization Assume a probability distribution (weights) over hidden events Take counts of events based on this distribution Use counts to estimate new parameters Use parameters to re-weight examples. Rinse and repeat

53. Alignment Hypotheses null I like milk Je aime le lait null I like milk Je aime le lait null I like milk Je aime le lait null I like milk Je aime le lait null I like milk Je aime le lait null I like milk Je aime le lait null I like milk Je aime le lait null I like milk Je aime le lait 0.650.250.05 0.01 0.001

54. Weighted Alignments What well do is: Consider every possible alignment Give each alignment a weight - indicating how good it is Count weighted alignments as normal

55. Good grief! We forgot about P(F|E)! No worries, a little more stats gets us what we need:

56. Big Example: Corpus fast car voiture rapide fast rapide 1 2

57. Possible Alignments fast car voiture rapide fast rapide fast car voiture rapide 1a1b2

58. Parameters fast car voiture rapide fast rapide fast car voiture rapide 1a1b2 P(voiture|fast)P(rapide|fast)P(voiture|car)P(rapide|car) 1/2

59. Weight Calculations fast car voiture rapide fast rapide fast car voiture rapide 1a1b2 P(voiture|fast)P(rapide|fast)P(voiture|car)P(rapide|car) 1/2 P(A,F|E)P(A|F,E) 1a1/2*1/2=1/41/4 / 2/4 = 1/2 1b1/2*1/2=1/41/4 / 2/4 = 1/2 21/21/2 / 1/2 = 1

60. Count Lines fast car voiture rapide fast rapide fast car voiture rapide 1a1b2 1/2 1

61. Count Lines fast car voiture rapide fast rapide fast car voiture rapide 1a1b2 1/2 1 #(voiture,fast)#(rapide,fast)#(voiture,car)#(rapide,car) 1/21/2+1 = 3/21/2

62. Count Lines fast car voiture rapide fast rapide fast car voiture rapide 1a1b2 1/2 1 #(voiture,fast)#(rapide,fast)#(voiture,car)#(rapide,car) 1/21/2+1 = 3/21/2 Normalize P(voiture|fast)P(rapide|fast)P(voiture|car)P(rapide|car) 1/43/41/2

63. Parameters fast car voiture rapide fast rapide fast car voiture rapide 1a1b2 P(voiture|fast)P(rapide|fast)P(voiture|car)P(rapide|car) 1/43/41/2

64. Weight Calculations fast car voiture rapide fast rapide fast car voiture rapide 1a1b2 P(voiture|fast)P(rapide|fast)P(voiture|car)P(rapide|car) 1/43/41/2 P(A,F|E)P(A|F,E) 1a1/4*1/2=1/81/8 / 4/8 = 1/4 1b1/2*3/4=3/83/8 / 4/8 = 3/4 23/43/4 / 3/4 = 1

65. Count Lines fast car voiture rapide fast rapide fast car voiture rapide 1a1b2 1/43/41

66. Count Lines fast car voiture rapide fast rapide fast car voiture rapide 1a1b2 1/43/41 #(voiture,fast)#(rapide,fast)#(voiture,car)#(rapide,car) 1/43/4+1 = 7/43/41/4

67. Count Lines fast car voiture rapide fast rapide fast car voiture rapide 1a1b2 1/43/41 #(voiture,fast)#(rapide,fast)#(voiture,car)#(rapide,car) 1/43/4+1 = 7/43/41/4 Normalize P(voiture|fast)P(rapide|fast)P(voiture|car)P(rapide|car) 1/87/83/41/4

68. After many iterations: fast car voiture rapide fast rapide fast car voiture rapide 1a1b2 ~0~11 P(voiture|fast)P(rapide|fast)P(voiture|car)P(rapide|car) 0.0010.999 0.001

69. Seems too easy? What if you have no 1-word sentence? Words in shorter sentences will get more weight - fewer possible alignments Weight is additive throughout the corpus: if a word e shows up frequently with some other word f, P(f|e) will go up Like our heaven example

70. The Final Product Now we have a model for P(F|E) Test it by aligning a corpus! IE: Find argmax A P(A|F,E) Use it for translation: Combine with our n-gram model for P(E) Search space of English sentences for one that maximizes P(E)P(F|E) for a given F

71. Model could be a lot better: Word positions Multiple fs generated by the same e Could take into account who generated your neighbors Could use syntax, parsing Could align phrases

72. Sure, but is it any better? Weve got some good ideas for improving translation How can we quantify the change translation quality?

73. Sure, but is it any better? How to (automatically) measure translation? Original French Dès qu'il fut dehors, Pierre se dirigea vers la rue de Paris, la principale rue du Havre, éclairée, animée, bruyante. Human translation to English As soon as he got out, Pierre made his way to the Rue de Paris, the high-street of Havre, brightly lighted up, lively and noisy. Two machine tranlations back to French: Dès qu'il est sorti, Pierre a fait sa manière à la rue De Paris, la haut-rue de Le Havre, brillamment allumée, animée et bruyante. Dès qu'il en est sorti, Pierre s'est rendu à la Rue de Paris, de la grande rue du Havre, brillamment éclairés, animés et bruyants. Example from http://www.readwriteweb.com/archives/google_translation_systran.php

74. Bleu Score Bleu Bilingual Evaluation Understudy A metric for comparing translations Considers n-grams in common with the target translation Length of target translation Score of 1 is identical, 0 shares no words in common Even human translations dont score 1

75. Google Translate http://translate.google.com/translate_t 25 language pairs In the news (digg.com) http://www.readwriteweb.com/archives/google _translation_systran.php http://www.readwriteweb.com/archives/google _translation_systran.php In competition http://www.nist.gov/speech/tests/mt/doc/mt06 eval_official_results.html http://www.nist.gov/speech/tests/mt/doc/mt06 eval_official_results.html

76. Questions? ?

77. References (Inspiration, Sources of borrowed material) Colin Cherry, MT for NLP, 2005 http://www.cs.ualberta.ca/~colinc/ta/MT650.pdf http://www.cs.ualberta.ca/~colinc/ta/MT650.pdf Knight, K., Automating Knowledge Acquisition for Machine Translation, AI Magazine 18(4), 1997. Knight, K., A Statistical Machine Translation Tutorial Workbook, 1999, http://www.clsp.jhu.edu/ws99/projects/mt/mt-workbook.htmhttp://www.clsp.jhu.edu/ws99/projects/mt/mt-workbook.htm Eisner, J., JHU NLP Course notes: Machine Translation, 2001, http://www.cs.jhu.edu/~jason/465/PDFSlides/lect32-translation.pdf http://www.cs.jhu.edu/~jason/465/PDFSlides/lect32-translation.pdf Olga Kubassova, Probability for NLP, http://www.comp.leeds.ac.uk/olga/ProbabilityTutorial.ppt http://www.comp.leeds.ac.uk/olga/ProbabilityTutorial.ppt

78. Enumerating all alignments There arepossible alignments!

79. Gah! Null (0) Fast (1) car (2) Voiture (1) rapide (2)

80. Lets move these over here… Null (0) Fast (1) car (2) Voiture (1) rapide (2)

81. And now we can do this… Null (0) Fast (1) car (2) Voiture (1) rapide (2)

82. So, it turns out: Requires only operations. Can be used whenever your alignment choice for one word does not affect the probability of the rest of the alignment