1. Evaluating Translation Memory Software Francie Gow MA Translation, University of Ottawa Translator, Translation Bureau, Government of Canada www.chandos.ca/thesis.html

2. Motivation for Research Volume of translation work increasing Machine translation not yet ready to meet new demand in a significant way Translators increasingly turning to translation memory (TM) software to increase productivity Available tools are difficult to compare

3. What is Translation Memory? Translation support software Allows users to recycle repetitive translation material Compares segments of new text with source material in the database If matches are found, retrieves corresponding target text and inserts it into new document

4. Automatic Search and Retrieval: Two Approaches Sentence-based approach –Example: TRADOS Character-string-within-a-bitext-based approach (CSB-based approach) –Example: MultiTrans

10. Two Approaches to TM Evaluation Primarily objective approach –Edit distance Primarily subjective approach –Human rating systems

11. Edit Distance Definition: smallest number of insertions, deletions, and substitutions required to change one string […] into another –National Institute of Standards and Technology

12. Edit Distance Advantages –Programmable –Once algorithm is developed, evaluation is fast and inexpensive Disadvantages –Loose approximation of usefulness –Definition of edit distance vague and variable –Assumes model translation

13. Human Rating Systems Example of a rating system (Sato, 1990) –(A) exact match –(B) “the example provides enough information about the translation of the whole input” –(C) “the example provides information about the translation of the whole input” –(F) “the example provides almost no information about the translation of the whole input”

14. Human Rating Systems Advantage –More valid than computer-generated results Disadvantages –Time consuming –Applicable to sentences, but not to mixed- language output of MultiTrans –Human is influenced by proposals

15. An evaluation system should be: reliable valid efficiently applicable –EAGLES Evaluation of Natural Language Processing Systems: Final Report

16. Measuring Usefulness Usefulness is a function of –Validity –Time Gain –Time Loss

17. New Evaluation Methodology Construction of a corpus in both tools Analysis and mark-up of new texts Processing of new texts in both tools Application of scores Analysis of scores

18. Conclusions Resulting methodology produces valid and reliable results Not as efficiently applicable as an edit distance algorithm, but highly customizable to a variety of translation contexts Applicable to any combination of tools, determines which is best for a given job