Machine translation (I) MT overview Ling 571 Fei Xia Week 9: 11/22/05 – 11/29/05

Plan for the rest of the quarter MT: –Part I: MT overview: 11/ /29 –Part II: Word-based SMT: 12/1-12/6 –Next quarter: seminar on MT Starting with a real baseline system Improving the system in various ways Reading and presenting recent papers Project presentation: 12/8

Homework and Quizzes Hw #8: due on 11/23 (tomorrow) Hw #9: due on 12/6 Hw #10: presentation due on 12/8, report due on 12/13 Quiz #3: 11/29 Quiz #4: 12/6

Outline MT in a nutshell Major challenges in MT Major approaches Evaluation of MT systems

MT in a nutshell

Q1: what is the ultimate goal of translation? Translation: source language  target language (S  T) Ultimate goal: find the “perfect” translation for text in S, thus allowing people to “appreciate” the text in S without knowing S: –Accuracy: faithful to S, including meaning, connotation, style, … –Fluency: the translation is as natural as an utterance in T.

Q2: what are the perfect translations? What do “Accuracy” and “Fluency” mean? Ex1: Complement / downplayer (1) A: Your daughter was phenomenal. (2) B: No. She was just so-so. Ex2: Greeting: how’s everything? –Old days: chi1 le5 ma5? (Have {you} eaten?) –1980s -- now: fa1 le5 ma5? (Have (you) gotten rich?) –2000s -- now: li2 le5 ma5? (Have (you) gotten divorced?) The answer: it depends

Q3: Can human always get the perfect translations? Novels: Shakespeare, Cao Xueqin, … - hidden messages: c1c0, c2 c0, c3 c0, c4 c0 c1’ c2’ c3’ c4’ Word play, jokes, puns: –What do prisoners use to call each other? –Cell phones. Concept gaps: double jeopardy, go Greek, fen sui, bei fen, …. Other constraints: lyrics, dubbing, poem.

“Crazy English” by Richard Lederer Let’s face it: English is a crazy language. There is no egg in eggplant or ham in hamburger, neither apple nor pine in pineapple. When a house burns up, it burns down. You fill in a form by filling it out and an alarm clock goes off by going on. When the stars are out, they are visible, but when the lights are out, they are invisible. And why, when I wind up my watch, I start it, but when I wind up this essay, I end it?

How to translate it? “Compound” words: Let’s face it: English is a crazy language. There is no egg in eggplant or ham in hamburger, neither apple nor pine in pineapple. Verb+particle: When a house burns up, it burns down. You fill in a form by filling it out and an alarm clock goes off by going on. Predicate+argument: When the stars are out, they are visible, but when the lights are out, they are invisible. And why, when I wind up my watch, I start it, but when I wind up this essay, I end it?

Q4: Can machines be as good as humans in translation quality? We know there are things that even humans cannot translate “perfectly”. For things that humans can translate, will machines be ever as good as humans in translation quality? –Never say “never”. –Not in the near future.

Q5: what is MT good for? Rough translation: web data Computer-aided human translation Translation for limited domain Cross-lingual IR Machine is better than human in: –Speed: much faster than humans –Memory: can easily memorize millions of word/phrase translations. –Manpower: machines are much cheaper than humans –Fast learner: it takes minutes or hours to build a new system. Erasable memory –Never complain, never get tired, …

Q6: what’s the MT history? (Based on work by John Hutchins) Before the computer: In the mid 1930s, a French- Armenian Georges Artsrouni and a Russian Petr Troyanskii applied for patents for ‘translating machines’. The pioneers ( ): the first public MT demo was given in 1954 (by IBM and Georgetown University). The decade of optimism ( ): ALPAC (Automatic Language Processing Advisory Committee) report in 1966: "there is no immediate or predictable prospect of useful machine translation."

A brief history of MT (cont) The aftermath of the ALPAC report ( ): a virtual end to MT research The 1980s: Interlingua, example-based MT The 1990s: Statistical MT The 2000s: Hybrid MT

Q7: where are we now? Huge potential/need due to the internet, globalization and international politics. Quick development time due to SMT, the availability of parallel data and computers. Translation is reasonable for language pairs with a large amount of resource. Start to include more “minor” languages.

MT in a nutshell What is the ultimate goal of translation? What are the perfect translations? Can human always get perfect translations? Can machines be as good as humans? What are MT good for? What is the MT history? Where are we now?

Outline MT in a nutshell Major challenges in MT Major approaches Evaluation of MT systems

Major challenges in MT

Major challenges Getting the right words: –Choosing the correct root form –Getting the correct inflected form –Inserting “spontaneous” words Putting the words in the correct order: –Word order: SVO vs. SOV, … –Unique constructions: –Divergence

Lexical choice Homonymy/Polysemy: bank, run Concept gap: no corresponding concepts in another language: go Greek, go Dutch, fen sui, lame duck, Chinese idioms, … Coding (Concept  lexeme mapping) differences: –More distinction in one language: e.g., kinship vocabulary. –Different division of conceptual space:

More distinction: the “cousin” example Translations: male/female, older or younger than the speaker, from mother or father’s side, parent’s brother or sister’s child: 堂兄 (tang xiong): father’s brother’s son, who is older 堂弟 (tang di): father’s brother’s son, who is younger 表兄 (biao xiong): – 姨表兄 (yi biao xiong): mother’s sister’s son, who is older – 舅表兄 (jiu biao xiong): mother’s brother’s son, who is older – 姑表兄 (gu biao xiong): father’s sister’s son, who is older 表弟 (biao di): mother’s sibling’s son, who is younger –…–… … 16+ translations  8  4.

More distinction: the “aunt” example Mother’s or father’s side, is a sister or is a brother’s wife: 姑妈： father’s sister 伯母： father’s younger brother’s wife 婶： father’s elder brother’s wife 姨妈： mother’s sister 舅妈： mother’s brother’s wife

A large happy family 三姨姥爷： mother’s mother’s 3 rd sister’s husband. 3 rd among all the sisters, or among all the siblings, or among all the members in the extended family with the same “bei fen”. Same “bei fen” ： same distance to the root node of a family tree.

Sapir-Whorf hypothesis Edward Sapir ( ): American linguist and anthropologist. Benjamin Lee Whorf ( ): Sapir’s student. Linguistic determinism/relativism: the language we use to some extent determines the way in which we view and think about the world around us. Strong determinism: language determines thought, that language and thought are identical. Weak determinism: thought is merely affected by or influenced by our language.

Sapir-Whorf hypothesis (cont) “Snow” –Whorf: the number of words for “snow” the Inuit people have for ‘snow‘  Inuit people treat snow differently than someone who lives in a less snow- dependent environment. –Pullum (1991): Other languages transmit the same ideas using phrases instead of single words. My personal experience: –Preference of one language over the other:

Color coding study Hypothesis: if one language categorizes color differently than another language, then the different groups should perceive it differently also. A 1970 study –Task: give people a sample of 160 colors and ask them to sort it. –People: English speakers (blue-green distinction) and Berinmo speaker (nol-wor distinction) –the Berinmo speakers were better at matching colors across their nol, wor categories than across the English blue and green categories and English speakers were better at matching colors across blue and green than across the Berinmo nol and wor (Sawyer, 1999).

Sapir-Whorf hypothesis (cont) What’s the relation between language, thought, and cultural perception of reality? Does language affect thought? If so, to what degree? The hypothesis is still under much debate.

Major challenges Getting the right words: –Choosing the correct root form –Getting the correct inflected form –Inserting “spontaneous” words Putting the words in the correct order: –Word order: SVO vs. SOV, … –Unique construction: –Structural divergence

Choosing the appropriate inflection Inflection: gender, number, case, tense, … Ex: –Number: Ch-Eng: all the concrete nouns: ch_book  book, books –Gender: Eng-Fr: all the adjectives –Case: Eng-Korean: all the arguments –Tense: Ch-Eng: all the verbs: ch_buy  buy, bought, will buy

Inserting spontaneous words Function words: –Determiners: Ch-Eng: ch_book  a book, the book, the books, books –Prepositions: Ch-Eng: … ch_November  … in November –Relative pronouns: Ch-Eng: … ch_buy ch_book de ch_person  the person who bought /book/ –Possessive pronouns: Ch-Eng: ch_he ch_raise ch_hand  He raised his hand(s) –Conjunction: Eng-Ch: Although S1, S2  ch_although S1, ch_but S2 –…–…

Inserting spontaneous words (cont) Content words: –Dropped argument: Ch-Eng: ch_buy le ma  Has Subj bought Obj? –Chinese First name: Eng-Ch: Jiang …  ch_Jiang ch_Zemin … –Abbreviation, Acronyms: Ch-Eng: ch_12 ch_big  the 12 th National Congress of the CPC (Communist Party of China) – …

Major challenges Getting the right words: –Choosing the correct root form –Getting the correct inflected form –Inserting “spontaneous” words Putting the words in the correct order: –Word order: SVO vs. SOV, … –Unique construction: –Structural divergence

Word order SVO, SOV, VSO, … VP + PP  PP VP VP + AdvP  AdvP + VP Adj + N  N + Adj NP + PP  PP NP NP + S  S NP P + NP  NP + P

“Unique” Constructions Overt wh-movement: Eng-Ch: –Eng: Why do you think that he came yesterday? –Ch: you why think he yesterday come ASP? –Ch: you think he yesterday why come? Ba-construction: Ch-Eng –She ba homework finish ASP  She finished her homework. –He ba wall dig ASP CL hole  He digged a hole in the wall. –She ba orange peel ASP skin  She peeled the orange’s skin.

Translation divergences (based on Bonnie Dorr’s work) Thematic divergence: I like Mary  S: Marta me gusta a mi (‘Mary pleases me’) Promotional divergence: John usually goes home  S: Juan suele ira casa (‘John tends to go home’) Demotional divergence: I like eating  G: Ich esse gern (“I eat likingly) Structural divergence: John entered the house  S: Juan entro en la casa (‘John entered in the house’)

Translation divergences (cont) Conflational divergence: I stabbed John  S: Yo le di punaladas a Juan (‘I gave knife- wounds to John’) Categorial divergence: I am hungry  G: Ich habe Hunger (‘I have hunger’) Lexical divergence: John broke into the room  S: Juan forzo la entrada al cuarto (‘John forced the entry to the room’)

Outline MT in a nutshell Major challenges in MT Major approaches Evaluation of MT systems

How humans do translation? Learn a foreign language: –Memorize word translations –Learn some patterns: –Exercise: Passive activity: read, listen Active activity: write, speak Translation: –Understand the sentence –Clarify or ask for help (optional) –Translate the sentence Training stage Decoding stage Translation lexicon Templates, transfer rules Parsing, semantics analysis? Interactive MT? Word-level? Phrase-level? Generate from meaning? Reinforced learning? Reranking?

What kinds of resources are available to MT? Translation lexicon: –Bilingual dictionary Templates, transfer rules: –Grammar books Parallel data, comparable data Thesaurus, WordNet, FrameNet, … NLP tools: tokenizer, morph analyzer, parser, …  More resources for major languages, less for “minor” languages.

Major approaches Transfer-based Interlingua Example-based (EBMT) Statistical MT (SMT) Hybrid approach

The MT triangle word Word Meaning Transfer-based Phrase-based SMT, EBMT Word-based SMT, EBMT (interlingua) Analysis Synthesis

Transfer-based MT Analysis, transfer, generation: 1.Parse the source sentence 2.Transform the parse tree with transfer rules 3.Translate source words 4.Get the target sentence from the tree Resources required: –Source parser –A translation lexicon –A set of transfer rules An example: Mary bought a book yesterday.

Transfer-based MT (cont) Parsing: linguistically motivated grammar or formal grammar? Transfer: context-free rules? Additional constraints on the rules? Apply at most one rule at each level? How are rules created? Translating words: word-to-word translation? Generation: using LM or other additional knowledge? How to create the needed resources automatically?

Interlingua For n languages, we need n(n-1) MT systems. Interlingua uses a language-independent representation. Conceptually, Interlingua is elegant: we only need n analyzers, and n generators. Resource needed: –A language-independent representation –Sophisticated analyzers –Sophisticated generators

Interlingua (cont) Questions: –Does language-independent meaning representation really exist? If so, what does it look like? –It requires deep analysis: how to get such an analyzer: e.g., semantic analysis –It requires non-trivial generation: How is that done? –It forces disambiguation at various levels: lexical, syntactic, semantic, discourse levels. –It cannot take advantage of similarities between a particular language pair.

Example-based MT Basic idea: translate a sentence by using the closest match in parallel data. First proposed by Nagao (1981). Ex: –Training data: w1 w2 w3 w4  w1’ w2’ w3’ w4’ w5 w6 w7  w5’ w6’ w7’ w8 w9  w8’ w9’ –Test sent: w1 w2 w6 w7 w9  w1’ w2’ w6’ w7’ w9’

EMBT (cont) Types of EBMT: –Lexical (shallow) –Morphological / POS analysis –Parse-tree based (deep) Types of data required by EBMT systems: –Parallel text –Bilingual dictionary –Thesaurus for computing semantic similarity –Syntactic parser, dependency parser, etc.

EBMT (cont) Word alignment: using dictionary and heuristics  exact match Generalization: –Clusters: dates, numbers, colors, shapes, etc. –Clusters can be built by hand or learned automatically. Ex: –Exact match: 12 players met in Paris last Tuesday  12 Spieler trafen sich letzen Dienstag in Paris –Templates: $num players met in $city $time  $num Spieler trafen sich $time in $city

Statistical MT Basic idea: learn all the parameters from parallel data. Major types: –Word-based –Phrase-based Strengths: –Easy to build, and it requires no human knowledge –Good performance when a large amount of training data is available. Weaknesses: –How to express linguistic generalization?

Comparison of resource requirement Transfer- based InterlinguaEBMTSMT dictionary+++ Transfer rules + parser+++ (?) semantic analyzer + parallel data++ othersUniversal representation thesaurus

Hybrid MT Basic idea: combine strengths of different approaches: –Syntax-based: generalization at syntactic level –Interlingua: conceptually elegant –EBMT: memorizing translation of n-grams; generalization at various level. –SMT: fully automatic; using LM; optimizing some objective functions. Types of hybrid HT: –Borrowing concepts/methods: SMT from EBMT: phrase-based SMT; Alignment templates EBMT from SMT: automatically learned translation lexicon Transfer-based from SMT: automatically learned translation lexicon, transfer rules; using LM … –Using two MTs in a pipeline: Using transfer-based MT as a preprocessor of SMT –Using multiple MTs in parallel, then adding a re-ranker.

Outline MT in a nutshell Major challenges in MT Major approaches Evaluation of MT systems

Evaluation Unlike many NLP tasks (e.g., tagging, chunking, parsing, IE, pronoun resolution), there is no single gold standard for MT. Human evaluation: accuracy, fluency, … –Problem: expensive, slow, subjective, non-reusable. Automatic measures: –Edit distance –Word error rate (WER), Position-independent WER (PER) –Simple string accuracy (SSA), Generation string accuracy (GSA) –BLEU

Edit distance The Edit distance (a.k.a. Levenshtein distance) is defined as the minimal cost of transforming str1 into str2, using three operations (substitution, insertion, deletion). Let the operation cost be subCost, insCost, and delCost, respectively. Let |Str1|=m and |Str2|=n, D(i,j) stores the edit distance of converting str1[1..i] to str2[1..j]. D(m,n) is the answer that we are looking for. Use DP and the complexity is O(m*n).

Calculating edit distance D(0, 0) = 0 D(i, 0) = delCost * i D(0, j) = insCost * j D(i+1, j+1) = min( D(i,j) + sub, D(i+1, j) + insCost, D(i, j+1) + delCost) sub = 0 if str1[i+1]=str2[j+1] = subCost otherwise

An example Sys: w1 w2 w3 w4 Ref: w1 w3 w2 All three costs are 1. Edit distance= w1 w3 w2 w1 w2 w3 w4

WER, PER, and SSA WER (word error rate) is edit distance, divided by |Ref|. PER (position-independent WER): same as WER but disregards word ordering SSA (Simple string accuracy) = 1 - WER Previous example: –Sys: w1 w2 w3 w4 –Ref: w1 w3 w2 –Edit distance = 2 –WER=2/3 –PER=1/3 –SSA=1/3

Generation string accuracy (GSA) Example: Ref: w1 w2 w3 w4 Sys: w2 w3 w4 w1 Del=1, Ins=1  SSA=1/2 Move=1, Del=0, Ins=0  GSA=3/4

BLEU Proposal by Papineni et. al. (2002) Most widely used in MT community. BLEU is a weighted average of n-gram precision (p n ) between system output and all references, multiplied by a brevity penalty (BP).

N-gram precision N-gram precision: the percent of n-grams in the system output that are correct. Clipping: –Sys: the the the the the the –Ref: the cat sat on the mat –Unigram precision: –Max_Ref_count: the max number of times a ngram occurs in any single reference translation.

N-gram precision i.e. the percent of n-grams in the system output that are correct (after clipping).

Brevity Penalty For each sent s i in system output, find closest matching reference r i (in terms of length). Longer system output is already penalized by the n-gram precision measure.

An example Sys: The cat was on the mat Ref1: The cat sat on a mat Ref2: There was a cat on the mat Assuming N=3 p 1 =5/6, p 2 =3/5, p 3 =1/4, BP=1  BLEU=0.50 What if N=4?

Summary MT in a nutshell Major challenges in MT –Choose the right words (root form, inflection, spontaneous words) –Put them in right positions (word order, unique constructions, divergences)

Summary (cont) Major approaches –Transfer-based MT –Interlingua –Example-based MT –Statistical MT –Hybrid MT Evaluation of MT systems –Edit distance –WER, PER, SSA, GSA –BLEU