1. Monoligual Semantic Text Alignment and its Applications in Machine Translation
Alon Lavie
March 29, 2012

2. Monolingual Semantic Text Alignment
Input: two (semantically-comparable) text strings
Output: Most likely word (or multi-word) alignment correspondence between the two strings
Example:
Applications in MT:
Automated MT Evaluation Metrics (METEOR)
Automated MT Error Analysis (METEOR)
MT System Combination (Multi-Engine MT)
Automated MT Post-Editing
MT System Optimization
March 29, 2012
Monolingual Text Alignment

3. Monolingual Semantic Text Alignment
Monolingual Text Aligner originally developed as an integral component of our METEOR MT Evaluation Metric
Input: Sentence-level MT-generated translation and a human-generated reference translation
Goal: Assess and assign a quality score to the MT-generated translation based on how similar (in meaning) it is to the reference
Address translation variability by explicitly identifying different types of semantically-equivalent correspondences:
Exact matches
Morphological variants
Synonyms
Paraphrases
Explicit alignment provides information on differences in word-ordering and other characteristics of correspondence beyond short n-gram matches
March 29, 2012
Monolingual Text Alignment

4. Monolingual Semantic Text Alignment
Semantic-similarity Information Sources:
Porter Stemmers (morphological variants)
WordNet (synonyms)
Paraphrase Tables (statistically-derived, filtered)
Alignment Algorithm:
Identify all matches of all types
Bipartite Match: each word (or multi-word) can match at most once
Single-pass approximate search for the maximal alignment with minimal crossing branches
March 29, 2012
Monolingual Text Alignment

5. Monolingual Text Alignment
METEOR
METEOR = Metric for Evaluation of Translation with Explicit Ordering [Lavie and Denkowski, 2010] [Denkowski and Lavie, 2011]
Main ideas:
Combine Recall and Precision as weighted score components
Look only at unigram Precision and Recall
Align MT output with each reference individually and take score of best pairing
Matching takes into account translation variability via word inflection variations, synonymy and paraphrasing matches
Addresses fluency via a direct penalty for word order: how fragmented is the matching of the MT output with the reference?
Parameters of metric components are tunable to maximize the score correlations with human judgments for each language
METEOR has been shown to consistently outperform BLEU and other metrics in correlation with human judgments
March 29, 2012
Monolingual Text Alignment

6. Monolingual Text Alignment
METEOR Scoring
March 29, 2012
Monolingual Text Alignment

7. Monolingual Text Alignment
METEOR Scoring
March 29, 2012
Monolingual Text Alignment

8. Monolingual Text Alignment
METEOR Scoring
March 29, 2012
Monolingual Text Alignment

9. METEOR Parameter Tuning
METEOR has several “free” parameters that can be optimized to maximize correlation with different notions of human judgments
Alpha, Beta and Gamma control overall metric behavior
Tunable weights for different types of matches
Latest version (1.3) distinguishes and tunes separate weights for content and function words
Optimized for Adequacy, Fluency, A+F, Rankings, and Post-Editing effort for English on available development data
Optimized independently for different target languages
Limited number of parameters means that optimization can be done by full exhaustive search of the parameter space
March 29, 2012
Monolingual Text Alignment

10. METEOR Parameter Tuning
March 29, 2012
Monolingual Text Alignment

11. Monolingual Text Alignment
METEOR Analysis Tools
METEOR v1.3 comes with a suite of analysis and visualization tools called METEOR-XRAY
March 29, 2012
Monolingual Text Alignment

12. Monolingual Text Alignment
METEOR Analysis Tools
March 29, 2012
Monolingual Text Alignment

13. MT System Combination
Idea: apply several MT engines to each input in parallel and combine their output translations
Goal: leverage the strengths and diversity of different MT engines to generate an improved translation system
Particularly useful in assimilation scenarios where input is uncontrolled and diverse in domain, genre, style or other characteristics
Can result in significant gains in translation quality 13

14. CMU’s Alignment-based Multi-Engine System Combination
Works with any MT engines
Assumes original MT systems are “black-boxes” – no internal information other than the translations themselves
Explores broader search spaces than other MT system combination approaches using linguistically-based and statistical features
Achieves state-of-the-art performance in competitive research evaluations of recent years
Developed over last seven years under research funding from several government grants (DARPA, DoD and NSF) 14

15. Alignment-based MEMT Two Stage Approach: Example:
Align: Identify and align equivalent words and phrases across the translations provided by the engines
Decode: search the space of synthetic combinations of words/phrases and select the highest scoring combined translation
Example:
announced afghan authorities on saturday reconstituted four intergovernmental committees
The Afghan authorities on Saturday the formation of the four committees of government 15

16. Alignment-based MEMT Two Stage Approach: Example:
Align: Identify and align equivalent words and phrases across the translations provided by the engines
Decode: search the space of synthetic combinations of words/phrases and select the highest scoring combined translation
Example:
announced afghan authorities on saturday reconstituted four intergovernmental committees
The Afghan authorities on Saturday the formation of the four committees of government
MEMT: the afghan authorities announced on Saturday the formation of four intergovernmental committees 16

17. The MEMT Decoder Algorithm
Search-space of system combination hypotheses implicitly defined by the initial alignment stage, and partially explored
Search-space is controlled by linguistic similarity features
Algorithm builds collections of partial hypotheses of increasing length
Partial hypotheses are extended by selecting the “next available” word from one of the original systems
Extending a partial hypothesis with a word marks the word as “used” and marks its aligned words as also “used”
Partial hypotheses are scored and ranked
Pruning and re-combination for efficiency
Hypothesis can end if any original system proposes an end of sentence as next word 17

18. Recent Performance Results NIST-2009 and WMT-200918

19. Recent Performance Results WMT-201119

20. Multi-Engine Human Translation
Translation crowdsourcing using untrained bilingual labor is error-prone
Idea: combine the output of multiple human translators into a consensus translation using MT system combination methods
[Ambati 2011]
March 29, 2012
Monolingual Text Alignment

21. Smoothing MERT in SMT [Cettolo, Bertoldi and Federico 2011]
Interesting application of MT system combination to overcome instability of MERT optimization in SMT
Perform MERT multiple times
Use the CMU MEMT system to combine the different instances of the same MT system 21

22. MT Automated Post-Editing
Idea: Target-language-side adaptation and refinement method
Adapt and specialize the output of a single “baseline” cross-lingual MT system to specific data characteristics: genre, domain, dialect, etc.
Particularly useful in situations where each specialization has only limited amounts of data available
Approach: Train a secondary “translation” system from baseline MT system target-language output to the specialized target output
Monolingual text alignment is a highly-effective method for aligning the monolingual “parallel” training data
Used extensively by Safaba Translation Solutions for client-specific MT adaptation
Goal: minimize the amount of human post-editing required on client-specific MT output
March 29, 2012
Monolingual Text Alignment

23. Safaba Two-Stage MT Approach
Baseline MT
Target
Language
Text
Client-Adapted
Target
Language
Text
Source
Language
Text
Baseline
MT Engine
Automated
Post-Editing (APE)
Engine
Safaba or
Partner-developed
(SMT or RBMT)
Safaba-developed
(Moses-based)
Client + General
Data
Client
Data

24. MT System Optimization
Goals:
Optimize MT systems to minimize important types of translation errors and maximize utility of generated MT (i.e. who did what to whom)
Minimize the amount of human post-editing effort required for correcting the MT output
Approach:
Develop the capabilities to automatically identify and quantify different types of MT errors (important content words, Named Entities, etc.)
Use the above capability to analyze in depth the error characteristics of different types of MT systems (phrase-based, Hierarchical, Syntax- based, etc.) at segment-level granularity
Develop an automated evaluation metric that can be used as an effective optimization function in advanced MT tuning optimizers such as PRO and MIRA
Existing metrics such as HTER and METEOR 1.3 are starting points
Monolingual Semantic Text Alignment is a critical first step

25. Post-Editing Pilot Study
Pilot study conducted with colleagues at Kent State IAL
Goals:
Develop post-editing analysis infrastructure and identify procedures and issues
Conduct a preliminary human post-editing study and analyze post-editing behavior, issues and outcomes
Pilot Setup:
English-to-Spanish
Two MT systems (Microsoft and Safaba)
Five short texts (~300 words), computer software documentation domain
20 trials: 5 texts x 2 MT systems x 2 post-edit translations
Seven bilingual translator post-editors each post-editing three MT docs
Collected all post-editing output + timing information + complete keystroke logger information
Expert translator generated segment-level human assessments on a 4-point scale [1= no edit; 2 = minor edit; 3 = major edit; 4 = complete rewrite]
Analysis currently underway
March 29, 2012
Monolingual Text Alignment

26. Questions