1. Towards Application of User-Tailored Machine Translation in Localization Andrejs Vasiļjevs, Raivis Skadiņš, Inguna Skadiņa TILDE JEC 2011, Luxembourg October 14, 2011

2. MT in Localization Work  The localization process is generally related to the translation and cultural adaptation of software, video games, and websites, and less frequently to any written translation  Translation Memory is commonly used  Although there are number of MT use-cases in practical localization process, it is not yet widely adapted

3. Previous Work  Evaluation with keyboard-monitoring program and Choice Network Analysis for measuring the effort involved in post-editing MT output (O´Brien, 2005)  Productivity tests have been performed in translation and localization industry settings at Microsoft (Schmidtke, 2008): SMT system of Microsoft Research trained on MS tech domain for 3 languages for Office Online 2007 localization task: Spanish, French and German By applying MT to all new words on average 5-10% productivity was gained

4. Previous Work – Adobe  In Adobe two experiments were performed (Flournoy and Duran, 2009): Small test set of 800-2000 words was machine translated and post-edited Then, based on the positive results, about 200,000 words of new text were localized The rule-based MT was used for translation into Russian (PROMT) and SMT for Spanish and French (Language Weaver) Productivity increase between 22% and 51%

5. Previous Work – Autodesk  Evaluation of Autodesk Moses SMT system (Plitt and Masselot, 2010): Translation from English to French, Italian, German and Spanish with three translators for each language pair To measure translation time special workbench was designed to capture keyboard and pause times for each sentence MT allowed translators to improve their throughput on average by 74% Varying increase in productivity: from 20% to 131% Optimum throughput has been reached for sentences of around 25 words in length

6. LetsMT! project  User-driven cloud-based MT factory, based on open-source MT tools  Services for data collection, MT generation, customization and running of variety of user-tailored MT systems  Application in localization among the key usage scenarios  Strong synergy with FP7 project ACCURAT to advance data- driven machine translation for under-resourced languages and domains

7. Partnership with Complementing Competencies  Tilde (Project Coordinator) - Latvia  University of Edinburgh - UK  University of Zagreb - Croatia  Copenhagen University - Denmark  Uppsala University - Sweden  Moravia – Czech Republic  SemLab – Netherlands Industry experience Research excellence Language expertise

8. LetsMT! Main Features  Online collaborative platform for MT building from user-provided data  Repository of parallel and monolingual corpora for MT generation  Automated training of SMT systems from specified collections of data  Users can specify particular training data collections and build customised MT engines from these collections  Users can also use LetsMT! platform for tailoring MT system to their needs from their non-public data

9. User requirement analysis  21 interviews have been conducted with localization/translation agencies.  Only 7 of the respondents replied that they use fully automatic MT systems in their translation practice and only one LSP organization employs MT as the primary translation method.

10. User survey

13. LetsMT! architecture  Multitier architecture User interface (webpage UI, web service API) Application Logic Resource Repository (stores MT training data and trained models) High-performance Computing Cluster (executes all computationally heavy tasks: SMT training, MT service, Processing and aligning of training data etc.)

14. LetsMT! architecture  Webpage interface where users can See, upload and manage corpora See, train and manage user tailored SMT system Translate texts and files  Web service API Integration with CAT tools Integration in web pages and web browsers

15. LetsMT! architecture  Resource Repository Stores SMT training data Supports different formats – TMX, XLIFF, PDF, DOC, plain text Converts to unified format Performs format conversions and alignment

16. Based on Moses SMT toolkit  Training tasks are managed with Moses Experiment Management System  Training tasks are executed in HPC cluster (Oracle Grid Engine)  Hosted in Web Services infrastructure which provides easy access to on demand computing resources  New Moses features: incremental training, distributed language models, interpolated language models for domain adaptation randomized language models to train using huge corpora translation of formatted texts, running Moses decoder in a server mode

21. Bilingual corpora for the English- Latvian system Bilingual corpusParallel units Localization TM~1.29 M DGT-TM~1.06 M OPUS EMEA~0.97 M Fiction~0.66 M Dictionary data~0.51 M Web corpus~0.9 M Total5.37 M

22. Latvian monolingual corpora Monolingual corpusWords Latvian side of parallel corpus 60 M News (web)250 M Fiction9 M Total, Latvian319 M

23.  Tilde English-Latvian MT  Tools: Moses, Giza++, SRILM, Tilde Latvian morphological analyzer  Factored phrase-based SMT system: Surface form  Surface form, Morphology tag 2 Language models: (1) 5-gram surface form (2) 7-gram morphology tag The MT System

24.  Development and evaluation data Development - 1000 sentences Evaluation – 500 sentences Balanced BLEU score: 35.0 The MT system TopicPercentage General information about European Union12% Specifications, instructions and manuals12% Popular scientific and educational12% Official and legal documents12% News and magazine articles24% Information technology18% Letters5% Fiction5%

25. Localization workflow at Tilde Evaluate original / assign Translator and Editor Import into SDL Trados Analyze against TMs Translate using translation suggestions for TMs Evaluate translation quality / Edit (optional) Fix errors Hand over to customer

26. MT Integration into Localization Workflow Evaluate original / assign Translator and Editor Analyze against TMs Translate using translation suggestions for TMs and MT Evaluate translation quality / Edit Fix errors Ready translation MT translate new sentences

27. Evaluation of Productivity  Key interest of localization industry is to increase productivity of translation process while maintaining required quality level  Productivity was measured as the translation output of an average translator in words per hour  5 translators participated in evaluation including both experienced and new translators

28. Evaluation of Quality  Performed by human editors as part of their regular QA process  Result of translation process was evaluated, editors did not know was or was not MT applied to assist translator  Comparison to reference is not part of this evaluation  Tilde standard QA assessment form was used covering the following text quality areas: Accuracy Spelling and grammar Style Terminology

29. Error Score 

30. QA Evaluation Form Error CategoryWeightAmount of errorsNegative points 1. Accuracy 1.1. Understanding of the source text3 0 1.2. Understanding the functionality of the product3 0 1.3. Comprehensibility3 0 1.4. Omissions/Unnecessary additions2 0 1.5. Translated/Untranslated1 0 1.6. Left-overs1 0 Total 0 2. Language quality 2.1. Grammar2 0 2.2. Punctuation1 0 2.3. Spelling1 0 Total 0 3. Style 3.1. Word order, word-for-word translation1 0 3.2. Vocabulary and style choice1 0 3.3. Style Guide adherence2 0 3.4. Country standards1 0 Total 0 4. Terminology 4.1. Glossary adherence2 0 4.2. Consistency2 0 Total 0 Additional plus points for style (if applicable)0 Grand Total 0 Negative points per 1000 words 0 Quality: Resulting Evaluation

31. QA Grades Error Score (sum of weighted errors) Resulting Quality Evaluation 0…9Superior 10…29Good 30…49Mediocre 50…69Poor >70Very poor Tilde Localization QA assessment applied in the evaluation

32. Evaluation data ► 54 documents in IT domain ► 950-1050 adjusted words in each document ► Each document was split in half: ► the first part was translated using suggestions from TM only ► the second half was translated using suggestions from both TM and MT

33. Integration of MT in SDL Trados

34. Evaluation Results ► Average translation productivity: ► Baseline with TM only: 550 w/h ► With TM and MT: 731 w/h  32.9% productivity increase ► High variability in individual performance with productivity increase by 64% to decrease by 5%. ► Increase of error score from 20.2 to 28.6 points but still at the level “GOOD” (<30 points)

35. Average error scores by error types Error ClassBaseline scenario MT scenario Accuracy69 Language quality610 Style34 Terminology57 Biggest quality degradation in language quality due to increase of grammatical errors.

36. Conclusions  The results of our experiment clearly demonstrate that it is feasible to integrate the current state of the art SMT systems for highly inflected languages into the localization process.  Quality of translation decreases in all error categories still degradation is not critical and the result is acceptable for production purposes.  Significant performance differences for individual translators hints to the role of human factors such as training, work procedures, personal inclinations etc.  Extended experiments are planned (I) involving more translators, (II) translating texts in different domains and (III) in other language pairs.

37. Application in localization workflow

38. Thank you! Let's MT! The research within the LetsMT! project leading to these results has received funding from the ICT Policy Support Programme (ICT PSP), Theme 5 – Multilingual web, grant agreement no 250456