LREC 2008 Marrakech 29 May Caroline Lavecchia, Kamel Smaïli and David Langlois LORIA / Groupe Parole, Vandoeuvre-Lès-Nancy, France Phrase-Based Machine Translation based on Simulated Annealing

LREC 2008 Marrakech 29 May Outline Statistical Machine Translation (SMT) Concept of inter-lingual triggers Our SMT system based on inter-lingual triggers –Word-based approach –Phrase-based approach using Simulated Annealing algorithm (SA) Experiments Conclusion

LREC 2008 Marrakech 29 May T* = argmax T P(T|S) T* = argmax T P(T) * P(S|T) Given a source sentence S, find the best target sentence T * which maximizes the probability P(T|S)  Noisy channel approach Language model Translation Model Introduction Approaches Statistical Machine Translation Introduction:

LREC 2008 Marrakech 29 May Word-based approach –Translation process is done word-by-word –IBM models (Brown et al., 1993) Phrase-based approach (Och et al., 1999), (Yamada and Knight, 2001), (Marcu and Wong, 2002 ) –Better MT system quality –Advantages: Explicitly models lexical units Ex: rat de bibliothèque  bookworm Easily captures local reordering Ex: Tour Eiffel  Eiffel tower Introduction Approaches Statistical Machine Translation Approaches:

LREC 2008 Marrakech 29 May Current translation models are complex Their estimation needs a lot of time and memory A new translation model based on inter-lingual triggers: We propose a new translation model based on a simple concept: the triggers.

LREC 2008 Marrakech 29 May Triggers in statistical language modeling: Concept of inter-lingual triggers A trigger is a set composed of a word and its best correlated words. Triggers are determined by computing Mutual Information (MI) between words on a monolingual corpus. A trigger is a set composed of a word and its best correlated words. Triggers are determined by computing Mutual Information (MI) between words on a monolingual corpus. Gary Kasparov is a chess champion In statistical language modeling, triggers allow to enhance the probability of triggered words given a triggering word. Gary Kasparov is a chess champion In statistical language modeling, triggers allow to enhance the probability of triggered words given a triggering word. Review of triggers Inter-lingual triggers

LREC 2008 Marrakech 29 May Inter-lingual triggers: Review of triggers Inter-lingual triggers An inter-lingual trigger is a set composed of a source unit s and its best correlated target units: t 1, …, t n. Inter-lingual triggers are determined by computing Mutual Information (MI) between units on a bilingual aligned corpus. An inter-lingual trigger is a set composed of a source unit s and its best correlated target units: t 1, …, t n. Inter-lingual triggers are determined by computing Mutual Information (MI) between units on a bilingual aligned corpus. Gary Kasparov is a chess champion Gary Kasparov est un champion d’échecs We hope to find possible translations of s among the set of its triggered target units t 1, …, t n Gary Kasparov is a chess champion Gary Kasparov est un champion d’échecs We hope to find possible translations of s among the set of its triggered target units t 1, …, t n Concept of inter-lingual triggers

1 source word triggers 1 target word. LREC 2008 Marrakech 29 May To-1 triggers: Review of triggers Inter-lingual triggers n-To-m triggers: Gary→ →Kasparov →Gary Kasparov→chess échecs→chess champion→ n source words trigger m target words. Gary Kasparov→ champion d’échecs→chess champion un champion→a champion Kasparov est un→is a chess échecs→chess champion champion→is a chess Source: Gary Kasparov est un champion d’échecs Target: Gary Kasparov is a chess champion Source: Gary Kasparov est un champion d’échecs Target: Gary Kasparov is a chess champion Concept of inter-lingual triggers

LREC 2008 Marrakech 29 May SMT based on inter-lingual triggers How to make good use of inter-lingual triggers in order to estimate a translation model? Word-based translation model using 1-To-1 triggers Phrase-based translation model using n-To-m triggers

LREC 2008 Marrakech 29 May SMT based on inter-lingual triggers For each source word, we keep its k best 1-To-1 triggers. We hope this constitute its potential translations. Translation model –We assign to each inter-lingual trigger a probability calculated as follow: Word-based translation model Word-based translation model using 1-To-1 triggers

Motivations: –Most methods for learning phrase translations require word alignments –All phrase pairs that are consistent with this word alignment are collected  phrases with no linguistic motivation  noisy phrases LREC 2008 Marrakech 29 May Phrase-based translation model using on n-To-m triggers SMT based on inter-lingual triggers Phrase-based translation model

1.Extract phrases from the source corpus 2.Determine potential translations of the source phrases by using n-To-m triggers 3.Start with 1-To-1 triggers to set a baseline MT system 4.Select an optimal subset of n-To-m triggers by Simulated Annealing algorithm LREC 2008 Marrakech 29 May Method for learning phrase translation: SMT based on inter-lingual triggers Phrase-based translation model

Iterative process which selects phrases by grouping words with high Mutual Information. (Zitouni et al., 2003) Only those which improve the perplexity on the source corpus are kept. → pertinent source phrases LREC 2008 Marrakech 29 May Phrase extraction: Method for learning phrase translation Source phrase extraction Determine potential phrase translation Select optimal phrase translations by SA algorithm

LREC 2008 Marrakech 29 May Learning potential phrase translation: A source phrase can be translated by different target sequences of variable sizes. Assumption: each source phrase of l words can be translated by a sequence of j target words where j Є [l-Δl, l+Δl] For each source phrase of length l, potential translations are: sets of n-To-m triggers with n = l and m Є [l-Δl, l+ Δl] Method for learning phrase translation Source phrase extraction Determine potential phrase translation Select optimal phrase translations by SA algorithm

LREC 2008 Marrakech 29 May Example: Method for learning phrase translation Source phrase extraction Determine potential phrase translation Select optimal phrase translations by SA algorithm Potential translations of porter plainte 2-To-12-To-22-To-3 presspress chargescan press charges chargescan pressnot press charges easynot pressyou can press Source phrase: porter plainte ( l=2 ) We assume that porter plainte can be translated by sequences of 1, 2 or 3 target words ( Δl=1 ).

LREC 2008 Marrakech 29 May General case: Method for learning phrase translation Source phrase extraction Determine potential phrase translation Select optimal phrase translations by SA algorithm All source phrases associated with its k potential translations constitute the set of n-To-m triggers. We have to select among n-To-m triggers pertinent translations and discard noisy ones. Our problem: find a optimal subset of phrase translations which leads to the best MT performance Unreasonnable to try all possibilities!!  Proposed method: use Simulated Annealing algorithm

LREC 2008 Marrakech 29 May Terminate search Initial configuration Pertub the configuration Accept new configuration Accept new configuration Update current configuration Adjust temperature Stop no yes no Simulated Annealing: Method for learning phrase translation Source phrase extraction Determine potential phrase translation Select optimal phrase translations by SA algorithm Technique applied to find an optimal solution to a combinatorial problem Initial temperature

LREC 2008 Marrakech 29 May Algorithm applied to SMT: Method for learning phrase translation Source phrase extraction Determine potential phrase translation Select optimal phrase translations by SA algorithm 1.Start with a high temperature T and a baseline word-based MT system using 1-To-1 triggers 2.do a)Perturb the system from state i to state j by randomly adding a subset of n-To-m triggers into the currrent SMT system b)Evaluate the performance of the new system ( E j ) c)If ( E j >E i ) then move from state i to state j Otherwise accepte state j with a probability random(P)<e(E i -E j )/T with P Є[0-1] Until the performance of our SMT system stops increasing 3.Decrease the temperature and go to step 2 until the performance of the system stops increasing

LREC 2008 Marrakech 29 May Text input decoder Text output Translation model Language model Bleu initial Subset of n-to-m trigers Bleu new > Bleu current Initial system Bleu new ≤ Bleu current New system Bleu current  Pertubation of the current system Bleu new 1-To-1 triggers n-To-m triggers

LREC 2008 Marrakech 29 May Subtitle copora: Experiments Corpora Tuning step Evaluation Subtitle parallel corpora built using Dynamic Time Wrapping algorithm (Lavecchia et al., 2007) FrenchEnglish TrainSentences Words Singletons Vocabulary DevSentences 1959 Words TestSentences 756 Words

LREC 2008 Marrakech 29 May SA algorithm parameters: Experiments Corpora Tuning step Evaluation 1-To-1 triggers: all source words associated with its best 50 target words n-To-m triggers: –15860 source phrases –all source phrases associated with its 30 best n-To-1, n-To-2 and n-To-3 inter-lingual triggers Initial temperature: System perturbation: adding 10 potential translations of 10 source phrases

LREC 2008 Marrakech 29 May Initial system: Experiments Corpora Tuning step Evaluation Translation ModeltmlmdwBleu 1-To-1 triggers IBM M3 (2) (1)Trigram model (2)(Brown et al., 1993) Text input Text output Pharaoh decoder Word translation model Language model (1)

LREC 2008 Marrakech 29 May Final system: Experiments Corpora Tuning step Evaluation Translation ModeltmlmdwBleu optimal n-To-m triggers Reference (2) (1)Trigram model (2)(Och, 2002) Text input Text output Pharaoh decoder Phrase translation model Language model (1)

LREC 2008 Marrakech 29 May Evaluation of the final system: Experiments Corpora Tuning step Evaluation Inter-lingual triggersState of the art 1-To-1n-To-m IBM3Reference Dev Test Lead of n-to-m triggers on 1-to-1 triggers not corroborated on the test corpus Explanations: - Over-fitting due to poor amount of data - Corpora of different movie styles Lead of n-to-m triggers on 1-to-1 triggers not corroborated on the test corpus Explanations: - Over-fitting due to poor amount of data - Corpora of different movie styles Impact of over-fitting more important on the state-of-the-art systems.

LREC 2008 Marrakech 29 May Conclusion and future work: A new method for learning phrase translations 1.Extract source phrases 2.Find phrase translations using inter-lingual triggers 3.Select the pertinent ones using SA algorithm advantages: no word alignment + more pertinent phrase translations Experiments on movie subtitle corpora  More robust on sparse data than a state-of-the-art approach  Better translation quality in terms of Bleu score (+7pts dev., +4pts test) A new method for learning phrase translations 1.Extract source phrases 2.Find phrase translations using inter-lingual triggers 3.Select the pertinent ones using SA algorithm advantages: no word alignment + more pertinent phrase translations Experiments on movie subtitle corpora  More robust on sparse data than a state-of-the-art approach  Better translation quality in terms of Bleu score (+7pts dev., +4pts test) Improvement of our system  Classify movies  Integrate linguistic knowledge in the translation process  Considering inter-lingual triggers not only on word surface forms Improvement of our system  Classify movies  Integrate linguistic knowledge in the translation process  Considering inter-lingual triggers not only on word surface forms