1. Multi-Engine Machine Translation
Alon Lavie
Language Technologies Institute Carnegie Mellon University 05 January 2011

2. Multi-Engine MT
January 5, 2011
MEMT

3. MT System Combination
Idea: apply several MT engines to each input in parallel and combine their output translations
Goal: leverage the strengths and diversity of different MT engines to generate an improved translation system
Particularly useful in assimilation scenarios where input is uncontrolled and diverse in domain, genre, style or other characteristics
Can result in significant gains in translation quality
January 5, 2011
MEMT

4. Main Issues and Challenges
Selecting the “best” output among the engines (on a sentence-by-sentence basis), or generating a deeper combination of the translation output from the original engines?
Selecting between engines is easier
Synthetically combining them can potentially produce greater improvements in translation quality
What information is available from each of the MT engines?
How do we determine if a synthetically combined translation is better than the originals?
January 5, 2011
MEMT

5. Approaches
Earliest work on MEMT in early 1990s (PANGLOSS) [pre “ROVER” for Speech Recognition]
Several Main Approaches:
Hypothesis Selection approaches
Lattice Combination and “joint” decoding
Confusion (or Consensus) Networks
Alignment-based Synthetic MEMT
January 5, 2011
MEMT

6. Hypothesis Selection Approaches
Main Idea: construct a classifier that given several translations for the same input sentence selects the “best” translation (on a sentence-by-sentence basis)
Should “beat” a baseline of always picking the system that is best in the aggregate
Main knowledge sources for scoring the individual translations are standard statistical target-language Language Models, overlap features, plus confidence scores for each engine
January 5, 2011
MEMT

7. Lattice-based MEMT
Earliest approach, first tried in CMU’s PANGLOSS in 1994, and still active in some recent work
Main Ideas:
Multiple MT engines each produce a lattice of scored translation fragments, indexed based on source language input
Lattices from all engines are combined into a global comprehensive translation lattice
Joint Decoder finds best translation (or n-best list) from the entries in the lattice
January 5, 2011
MEMT

8. Lattice-based MEMT: Example
El punto de descarge
The drop-off point
se cumplirá en
will comply with
el puente Agua Fria
The cold Bridgewater
The discharge point
will self comply in
the “Agua Fria” bridge
Unload of the point
will take place at
the cold water of bridge
January 5, 2011
MEMT

9. Consensus Network Approach
Main Ideas:
Collapse the collection of linear strings of multiple translations into a minimal consensus network (“sausage” graph) that represents how they align
One translation acts as the “back-bone” and determines the main ordering of words
Search is limited to selecting among aligned alternatives along the back-bone
Decode: find the path through the network that has the best score
January 5, 2011
MEMT

10. Consensus Network Example
January 5, 2011
MEMT

11. CMU’s Alignment-based Synthetic MEMT
Works with any MT engines
Assumes original MT systems are “black-boxes” – no internal information other than the translations themselves
Explores broader search spaces than other MT system combination approaches using more complex and effective features
Achieves state-of-the-art performance in research evaluations over past couple of years
Developed over last five years under research funding from several government grants (DARPA, DoD and NSF)
January 5, 2011
MEMT

12. Alignment-based Synthetic MEMT
Two Stage Approach:
Align: Identify and align equivalent words and phrases across the translations provided by the engines
Decode: search the space of synthetic combinations of words/phrases and select the highest scoring combined translation
Example:
announced afghan authorities on saturday reconstituted four intergovernmental committees
The Afghan authorities on Saturday the formation of the four committees of government
January 5, 2011
MEMT

13. Alignment-based Synthetic MEMT
Two Stage Approach:
Align: Identify and align equivalent words and phrases across the translations provided by the engines
Decode: search the space of synthetic combinations of words/phrases and select the highest scoring combined translation
Example:
announced afghan authorities on saturday reconstituted four intergovernmental committees
The Afghan authorities on Saturday the formation of the four committees of government
MEMT: the afghan authorities announced on Saturday the formation of four intergovernmental committees
January 5, 2011
MEMT

14. The String Alignment Matcher
Developed as a component in our METEOR Automated MT Evaluation system
Originally word-based, extended to phrasal matches
Finds maximal alignment match with minimal “crossing branches”
Allows alignment of:
Identical words
Morphological variants of words (using stemming)
Synonymous words (based on WordNet synsets)
Single and multi-word Paraphrases (based on statistically-learned paraphrase tables)
Implementation: efficient search algorithm for best scoring weighed string match
January 5, 2011
MEMT

15. The String Alignment Matcher Examples:
January 5, 2011
MEMT

16. The MEMT Decoder Algorithm
Algorithm builds collections of partial hypotheses of increasing length
Partial hypotheses are extended by selecting the “next available” word from one of the original systems
Extending a partial hypothesis with a word marks the word as “used” and marks its aligned words as also “used”
Partial hypotheses are scored and ranked
Pruning and re-combination
Hypothesis can end if any original system proposes an end of sentence as next word
January 5, 2011
MEMT

17. Decoding Example
January 5, 2011
MEMT

18. Decoding Example
January 5, 2011
MEMT

19. Decoding Example
January 5, 2011
MEMT

20. Decoding Example
January 5, 2011
MEMT

21. Scoring MEMT Hypotheses
Features:
Language Model score based on filtered large-scale target language LM
N-gram support features with n-grams matches from the original systems (unigrams to 4-grams)
Length
Scoring:
Weighed Log-linear feature combination tuned on development set
January 5, 2011
MEMT

22. N-gram Match Support Features
January 5, 2011
MEMT

23. Recent Performance Results NIST-2009 and WMT-2009
January 5, 2011
MEMT

24. Recent Performance Results WMT-2010
January 5, 2011
MEMT

25. Recent Performance Results WMT-2010
January 5, 2011
MEMT

26. UIMA-based MEMT MEMT engine set up as a remote server:
Communication over socket connections
Sentence-by-sentence translation
Java “wrapper” turns the MEMT service into a UIMA- style annotator component
UIMA supports easy integration of the MEMT component into various processing “workflows”:
Input is a “document” annotated with multiple translations
Output is the same “document” with an additional MEMT annotation
January 5, 2011
MEMT

27. Current and Future Work
Incorporation of multi-word paraphrase matches into the decoding algorithm
Improved search-space exploration
Additional scoring features
Second-pass MBR-decoding over n-best lists
Multi-Engine Human Translation
January 5, 2011
MEMT

28. Summary
Sentence-level MEMT approach with unique properties and state-of-the-art performance results:
5-30% improvement in scores in multiple evaluations
State-of-the-art results in NIST-09, WMT-09 and WMT-10 competitive evaluations
Easy to run on both research and COTS systems
UIMA-based architecture design for effective integration in large distributed systems/projects
GALE Integrated Operational Demonstration (IOD) experience has been very positive
Can serve as a model for integration framework under other projects
January 5, 2011
MEMT

29. METEOR
METEOR = Metric for Evaluation of Translation with Explicit Ordering [Lavie and Denkowski, 2009]
Main ideas:
Automated MT evaluation scoring system for assessing and analyzing the quality of MT systems
Generates both sentence-level and aggregate scores and assessments
Based on our proprietary effective string matching algorithm
Matching takes into account translation variability via word inflection variations, synonymy and paraphrasing matches
Parameters of metric components are tunable to maximize the score correlations with human judgments for each language
METEOR has been shown to consistently outperform BLEU in correlation with human judgments of translation quality
Top performing evaluation scoring system in NIST 2008 and open evaluations (out of ~40 submissions)
January 5, 2011
METEOR

30. METEOR Analysis Tools
METEOR v1.2 comes with a suite of new analysis and visualization tools called METEOR-XRAY
January 5, 2011
METEOR

31. Safaba Translation Solutions
Technology spin-off start-up company from CMU/LTI
Co-founded by Alon Lavie and Bob Olszewski
Mission: Develop and deliver advanced translation automation software solutions for the commercial translation business sector
Target Clients: Language Services Providers (LSPs) and their enterprise clients
Major Service:
Software-as-a-Service customized MT Technology: Develop specialized highly-scalable software for delivering high-quality client- customized Machine Translation (MT) as a remote service
Integration into Human Translation Workflows: Customized-MT is integrated with Translation Memory and embedded into standard human translation automation tools to improve translator productivity and consistency
Details:
Contact: Alon Lavie