Machine Translation: Challenges and Approaches Nizar Habash Associate Research Scientist Center for Computational Learning Systems Columbia University Invited Lecture Introduction to Natural Language Processing Fall 2008

Currently, Google offers translations between the following languages: Arabic Bulgarian Catalan Chinese Croatian Czech Danish Dutch Filipino Finnish French German Greek Hebrew Hindi Indonesian Italian Japanese Korean Latvian Lithuanian Norwegian Polish Portuguese Romanian Russian Serbian Slovak Slovenian Spanish Swedish Ukrainian Vietnamese

Thank you for your attention! Questions?

“BBC found similar support”!!!

Road Map Multilingual Challenges for MT MT Approaches MT Evaluation

Multilingual Challenges Orthographic Variations –Ambiguous spelling كتب الاولاد اشعارا كَتَبَ الأوْلادُ اشعَاراً – Ambiguous word boundaries Lexical Ambiguity –Bank  بنك (financial) vs. ضفة(river) –Eat  essen (human) vs. fressen (animal)

Multilingual Challenges Morphological Variations Affixation vs. Root+Pattern write  writtenكتب  مكتوب kill  killedقتل  مقتول do  doneفعل  مفعول conj noun pluralarticle Tokenization And the cars  and the cars والسيارات  w Al SyArAt Et les voitures  et le voitures

لست هنا I-am-not here am Ihere I am not here not لست هنا Translation Divergences conflation Je ne suis pas ici I not am not here suis Jeicinepas

Translation Divergences head swap and categorial EnglishJohn swam across the river quickly SpanishJuan cruzó rapidamente el río nadando Gloss: John crossed fast the river swimming Arabicاسرع جون عبور النهر سباحة Gloss: sped john crossing the-river swimming Chinese 约翰 快速 地 游 过 这 条 河 Gloss: John quickly (DE) swam cross the (Quantifier) river RussianДжон быстро переплыл реку Gloss: John quickly cross-swam river

Road Map Multilingual Challenges for MT MT Approaches MT Evaluation

MT Approaches MT Pyramid Source word Source syntax Source meaningTarget meaning Target syntax Target word AnalysisGeneration Gisting

MT Approaches Gisting Example Sobre la base de dichas experiencias se estableció en 1988 una metodología. Envelope her basis out speak experiences them settle at 1988 one methodology. On the basis of these experiences, a methodology was arrived at in 1988.

MT Approaches MT Pyramid Source word Source syntax Source meaningTarget meaning Target syntax Target word AnalysisGeneration GistingTransfer

MT Approaches Transfer Example Transfer Lexicon –Map SL structure to TL structure  poner X mantequilla en Y :obj :mod:subj :obj butter X Y :subj:obj X puso mantequilla en YX buttered Y

MT Approaches MT Pyramid Source word Source syntax Source meaningTarget meaning Target syntax Target word AnalysisGeneration GistingTransferInterlingua

MT Approaches Interlingua Example: Lexical Conceptual Structure (Dorr, 1993)

MT Approaches MT Pyramid Source word Source syntax Source meaningTarget meaning Target syntax Target word AnalysisGeneration Interlingua Gisting Transfer

MT Approaches MT Pyramid Source word Source syntax Source meaningTarget meaning Target syntax Target word AnalysisGeneration Interlingual Lexicons Dictionaries/Parallel Corpora Transfer Lexicons

MT Approaches Statistical vs. Rule-based Source word Source syntax Source meaningTarget meaning Target syntax Target word AnalysisGeneration

Statistical MT Noisy Channel Model Portions from

Statistical MT Automatic Word Alignment GIZA++ –A statistical machine translation toolkit used to train word alignments. –Uses Expectation-Maximization with various constraints to bootstrap alignments Slide based on Kevin Knight’s Mary did not slap the green witch Maria no dio una bofetada a la bruja verde

Statistical MT IBM Model (Word-based Model)

Phrase-Based Statistical MT Foreign input segmented in to phrases –“phrase” is any sequence of words Each phrase is probabilistically translated into English –P(to the conference | zur Konferenz) –P(into the meeting | zur Konferenz) Phrases are probabilistically re-ordered See [Koehn et al, 2003] for an intro. This is state-of-the-art! Morgenfliegeichnach Kanadazur Konferenz TomorrowIwill flyto the conferenceIn Canada Slide courtesy of Kevin Knight

Mary did not slap the green witch Maria no dió una bofetada a la bruja verde Word Alignment Induced Phrases (Maria, Mary) (no, did not) (slap, dió una bofetada) (la, the) (bruja, witch) (verde, green) Slide courtesy of Kevin Knight

Mary did not slap the green witch Maria no dió una bofetada a la bruja verde Word Alignment Induced Phrases (Maria, Mary) (no, did not) (slap, dió una bofetada) (la, the) (bruja, witch) (verde, green) (a la, the) (dió una bofetada a, slap the) Slide courtesy of Kevin Knight

Mary did not slap the green witch Maria no dió una bofetada a la bruja verde Word Alignment Induced Phrases (Maria, Mary) (no, did not) (slap, dió una bofetada) (la, the) (bruja, witch) (verde, green) (a la, the) (dió una bofetada a, slap the) (Maria no, Mary did not) (no dió una bofetada, did not slap), (dió una bofetada a la, slap the) (bruja verde, green witch) Slide courtesy of Kevin Knight

Mary did not slap the green witch Maria no dió una bofetada a la bruja verde (Maria, Mary) (no, did not) (slap, dió una bofetada) (la, the) (bruja, witch) (verde, green) (a la, the) (dió una bofetada a, slap the) (Maria no, Mary did not) (no dió una bofetada, did not slap), (dió una bofetada a la, slap the) (bruja verde, green witch) (Maria no dió una bofetada, Mary did not slap) (a la bruja verde, the green witch) … Word Alignment Induced Phrases Slide courtesy of Kevin Knight

Mary did not slap the green witch Maria no dió una bofetada a la bruja verde (Maria, Mary) (no, did not) (slap, dió una bofetada) (la, the) (bruja, witch) (verde, green) (a la, the) (dió una bofetada a, slap the) (Maria no, Mary did not) (no dió una bofetada, did not slap), (dió una bofetada a la, slap the) (bruja verde, green witch) (Maria no dió una bofetada, Mary did not slap) (a la bruja verde, the green witch) … (Maria no dió una bofetada a la bruja verde, Mary did not slap the green witch) Word Alignment Induced Phrases Slide courtesy of Kevin Knight

Advantages of Phrase-Based SMT Many-to-many mappings can handle non- compositional phrases Local context is very useful for disambiguating –“Interest rate”  … –“Interest in”  … The more data, the longer the learned phrases –Sometimes whole sentences Slide courtesy of Kevin Knight

Source word Source syntax Source meaningTarget meaning Target syntax Target word AnalysisGeneration MT Approaches Statistical vs. Rule-based vs. Hybrid

MT Approaches Practical Considerations Resources Availability –Parsers and Generators Input/Output compatability –Translation Lexicons Word-based vs. Transfer/Interlingua –Parallel Corpora Domain of interest Bigger is better Time Availability –Statistical training, resource building

Road Map Multilingual Challenges for MT MT Approaches MT Evaluation

More art than science Wide range of Metrics/Techniques –interface, …, scalability, …, faithfulness,... space/time complexity, … etc. Automatic vs. Human-based –Dumb Machines vs. Slow Humans

Human-based Evaluation Example Accuracy Criteria

Human-based Evaluation Example Fluency Criteria

Automatic Evaluation Example Bleu Metric (Papineni et al 2001) Bleu –BiLingual Evaluation Understudy –Modified n-gram precision with length penalty –Quick, inexpensive and language independent –Correlates highly with human evaluation –Bias against synonyms and inflectional variations

Test Sentence colorless green ideas sleep furiously Gold Standard References all dull jade ideas sleep irately drab emerald concepts sleep furiously colorless immature thoughts nap angrily Automatic Evaluation Example Bleu Metric

Test Sentence colorless green ideas sleep furiously Gold Standard References all dull jade ideas sleep irately drab emerald concepts sleep furiously colorless immature thoughts nap angrily Unigram precision = 4/5 Automatic Evaluation Example Bleu Metric

Test Sentence colorless green ideas sleep furiously Gold Standard References all dull jade ideas sleep irately drab emerald concepts sleep furiously colorless immature thoughts nap angrily Unigram precision = 4 / 5 = 0.8 Bigram precision = 2 / 4 = 0.5 Bleu Score = (a 1 a 2 …a n ) 1/n = (0.8 ╳ 0.5) ½ =  Automatic Evaluation Example Bleu Metric

Automatic Evaluation Example METEOR (Lavie and Agrawal 2007) Metric for Evaluation of Translation with Explicit word Ordering Extended Matching between translation and reference –Porter stems, wordNet synsets Unigram Precision, Recall, parameterized F-measure Reordering Penalty Parameters can be tuned to optimize correlation with human judgments Not biased against “non-statistical” MT systems

Metrics MATR Workshop Workshop in AMTA conference 2008 –Association for Machine Translation in the Americas Evaluating evaluation metrics Compared 39 metrics –7 baselines and 32 new metrics –Various measures of correlation with human judgment –Different conditions: text genre, source language, number of references, etc.

A syntactically-aware evaluation metric – (Liu and Gildea, 2005; Owczarzak et al., 2007; Giménez and Màrquez, 2007) Uses dependency representation –MICA parser (Nasr & Rambow 2006) –77% of all structural bigrams are surface n-grams of size 2,3,4 Includes dependency surface span as a factor in score –long-distance dependencies should receive a greater weight than short distance dependencies Higher degree of grammaticality? Automatic Evaluation Example SEPIA (Habash and ElKholy 2008)

Interested in MT?? Contact me Research courses, projects Languages of interest: –English, Arabic, Hebrew, Chinese, Urdu, Spanish, Russian, …. Topics –Statistical, Hybrid MT Phrase-based MT with linguistic extensions Component improvements or full-system improvements –MT Evaluation –Multilingual computing

Thank You