MT Evaluation: What to do and what not to do Eduard Hovy Information Sciences Institute University of Southern California

My background: MT Eval is pain Psychological damage: DARPA MT program Killer competition: TREC IR and QA contests Painstaking details: ISLE collaboration

Best and worst Best: you describe your needs, your environment, and give some sample text… …and the Evaluation system gives you a report of all the systems’ performances! Worst: a repeat of the DARPA experience: expensive, stressful, and inconclusive.

The worst thing an evaluator can do Apply some arbitrary measure: …without understanding what it measures, …without knowing when it is applicable, …without understanding the accuracy or reliability of the metric. What do we need? …a systematic outline of the metrics and their appropriateness and their scope.

Do-It-Yourself Evaluations Procedure: 1. characterize your translation purposes and operational processes, 2. moving downward, locate their points in the taxonomies, 3. find their evaluation metrics, 4. apply them to your MT system(s), De/Recomposable scores: 5. recombine scores, by moving back up the taxonomy.

Work required The ISLE taxonomy is there, more or less. BUT: the actual metrics have not been characterized systematically yet: –inter-tester agreement, –robustness over different domains and genres, –expense and difficulty, –independence/relationship to other metrics. What’s needed: a project of evaluation evaluations.