Multi-Engine Machine Translation Alon Lavie Language Technologies Institute Carnegie Mellon University 05 January 2011

Multi-Engine MT January 5, 2011 MEMT

MT System Combination Idea: apply several MT engines to each input in parallel and combine their output translations Goal: leverage the strengths and diversity of different MT engines to generate an improved translation system Particularly useful in assimilation scenarios where input is uncontrolled and diverse in domain, genre, style or other characteristics Can result in significant gains in translation quality January 5, 2011 MEMT

Main Issues and Challenges Selecting the “best” output among the engines (on a sentence-by-sentence basis), or generating a deeper combination of the translation output from the original engines? Selecting between engines is easier Synthetically combining them can potentially produce greater improvements in translation quality What information is available from each of the MT engines? How do we determine if a synthetically combined translation is better than the originals? January 5, 2011 MEMT

Approaches Earliest work on MEMT in early 1990s (PANGLOSS) [pre “ROVER” for Speech Recognition] Several Main Approaches: Hypothesis Selection approaches Lattice Combination and “joint” decoding Confusion (or Consensus) Networks Alignment-based Synthetic MEMT January 5, 2011 MEMT

Hypothesis Selection Approaches Main Idea: construct a classifier that given several translations for the same input sentence selects the “best” translation (on a sentence-by-sentence basis) Should “beat” a baseline of always picking the system that is best in the aggregate Main knowledge sources for scoring the individual translations are standard statistical target-language Language Models, overlap features, plus confidence scores for each engine January 5, 2011 MEMT

Lattice-based MEMT Earliest approach, first tried in CMU’s PANGLOSS in 1994, and still active in some recent work Main Ideas: Multiple MT engines each produce a lattice of scored translation fragments, indexed based on source language input Lattices from all engines are combined into a global comprehensive translation lattice Joint Decoder finds best translation (or n-best list) from the entries in the lattice January 5, 2011 MEMT

Lattice-based MEMT: Example El punto de descarge The drop-off point se cumplirá en will comply with el puente Agua Fria The cold Bridgewater The discharge point will self comply in the “Agua Fria” bridge Unload of the point will take place at the cold water of bridge January 5, 2011 MEMT

Consensus Network Approach Main Ideas: Collapse the collection of linear strings of multiple translations into a minimal consensus network (“sausage” graph) that represents how they align One translation acts as the “back-bone” and determines the main ordering of words Search is limited to selecting among aligned alternatives along the back-bone Decode: find the path through the network that has the best score January 5, 2011 MEMT

Consensus Network Example January 5, 2011 MEMT

CMU’s Alignment-based Synthetic MEMT Works with any MT engines Assumes original MT systems are “black-boxes” – no internal information other than the translations themselves Explores broader search spaces than other MT system combination approaches using more complex and effective features Achieves state-of-the-art performance in research evaluations over past couple of years Developed over last five years under research funding from several government grants (DARPA, DoD and NSF) January 5, 2011 MEMT

Alignment-based Synthetic MEMT Two Stage Approach: Align: Identify and align equivalent words and phrases across the translations provided by the engines Decode: search the space of synthetic combinations of words/phrases and select the highest scoring combined translation Example: announced afghan authorities on saturday reconstituted four intergovernmental committees The Afghan authorities on Saturday the formation of the four committees of government January 5, 2011 MEMT

Alignment-based Synthetic MEMT Two Stage Approach: Align: Identify and align equivalent words and phrases across the translations provided by the engines Decode: search the space of synthetic combinations of words/phrases and select the highest scoring combined translation Example: announced afghan authorities on saturday reconstituted four intergovernmental committees The Afghan authorities on Saturday the formation of the four committees of government MEMT: the afghan authorities announced on Saturday the formation of four intergovernmental committees January 5, 2011 MEMT

The String Alignment Matcher Developed as a component in our METEOR Automated MT Evaluation system Originally word-based, extended to phrasal matches Finds maximal alignment match with minimal “crossing branches” Allows alignment of: Identical words Morphological variants of words (using stemming) Synonymous words (based on WordNet synsets) Single and multi-word Paraphrases (based on statistically-learned paraphrase tables) Implementation: efficient search algorithm for best scoring weighed string match January 5, 2011 MEMT

The String Alignment Matcher Examples: January 5, 2011 MEMT

The MEMT Decoder Algorithm Algorithm builds collections of partial hypotheses of increasing length Partial hypotheses are extended by selecting the “next available” word from one of the original systems Extending a partial hypothesis with a word marks the word as “used” and marks its aligned words as also “used” Partial hypotheses are scored and ranked Pruning and re-combination Hypothesis can end if any original system proposes an end of sentence as next word January 5, 2011 MEMT

Decoding Example January 5, 2011 MEMT

Decoding Example January 5, 2011 MEMT

Decoding Example January 5, 2011 MEMT

Decoding Example January 5, 2011 MEMT

Scoring MEMT Hypotheses Features: Language Model score based on filtered large-scale target language LM N-gram support features with n-grams matches from the original systems (unigrams to 4-grams) Length Scoring: Weighed Log-linear feature combination tuned on development set January 5, 2011 MEMT

N-gram Match Support Features January 5, 2011 MEMT

Recent Performance Results NIST-2009 and WMT-2009 January 5, 2011 MEMT

Recent Performance Results WMT-2010 January 5, 2011 MEMT

Recent Performance Results WMT-2010 January 5, 2011 MEMT

UIMA-based MEMT MEMT engine set up as a remote server: Communication over socket connections Sentence-by-sentence translation Java “wrapper” turns the MEMT service into a UIMA- style annotator component UIMA supports easy integration of the MEMT component into various processing “workflows”: Input is a “document” annotated with multiple translations Output is the same “document” with an additional MEMT annotation January 5, 2011 MEMT

Current and Future Work Incorporation of multi-word paraphrase matches into the decoding algorithm Improved search-space exploration Additional scoring features Second-pass MBR-decoding over n-best lists Multi-Engine Human Translation January 5, 2011 MEMT

Summary Sentence-level MEMT approach with unique properties and state-of-the-art performance results: 5-30% improvement in scores in multiple evaluations State-of-the-art results in NIST-09, WMT-09 and WMT-10 competitive evaluations Easy to run on both research and COTS systems UIMA-based architecture design for effective integration in large distributed systems/projects GALE Integrated Operational Demonstration (IOD) experience has been very positive Can serve as a model for integration framework under other projects January 5, 2011 MEMT

METEOR METEOR = Metric for Evaluation of Translation with Explicit Ordering [Lavie and Denkowski, 2009] Main ideas: Automated MT evaluation scoring system for assessing and analyzing the quality of MT systems Generates both sentence-level and aggregate scores and assessments Based on our proprietary effective string matching algorithm Matching takes into account translation variability via word inflection variations, synonymy and paraphrasing matches Parameters of metric components are tunable to maximize the score correlations with human judgments for each language METEOR has been shown to consistently outperform BLEU in correlation with human judgments of translation quality Top performing evaluation scoring system in NIST 2008 and 2010 open evaluations (out of ~40 submissions) January 5, 2011 METEOR

METEOR Analysis Tools METEOR v1.2 comes with a suite of new analysis and visualization tools called METEOR-XRAY January 5, 2011 METEOR

Safaba Translation Solutions Technology spin-off start-up company from CMU/LTI Co-founded by Alon Lavie and Bob Olszewski Mission: Develop and deliver advanced translation automation software solutions for the commercial translation business sector Target Clients: Language Services Providers (LSPs) and their enterprise clients Major Service: Software-as-a-Service customized MT Technology: Develop specialized highly-scalable software for delivering high-quality client- customized Machine Translation (MT) as a remote service Integration into Human Translation Workflows: Customized-MT is integrated with Translation Memory and embedded into standard human translation automation tools to improve translator productivity and consistency Details: http://www.safaba.com Contact: Alon Lavie <alavie@safaba.com>