Evaluation of the Statistical Machine Translation Service for Croatian-English Marija Brkić Department of Informatics, University of Rijeka Tomislav Vičić Freelance teacher of economics and translator Sanja Seljan Department of Information Sciences Faculty of Humanities and Social Sciences, University of Zagreb

I.Machine translation  Syntactic transfer  Example-based translation  Statistically-based translation II.Evaluation  Manual  Automatic III.Experimental study  Google Translate Service (Croatian → English)  Comparison and analysis  Manual evaluation IV.Conclusion 2/18 Outline

3/18 I. Machine translation basics Speeding up translation processSpeeding up translation process Limited human componentLimited human component Multilingual access to written materialMultilingual access to written material Limited capabilitiesLimited capabilities Help in discovering general idea behind textHelp in discovering general idea behind text For limited use onlyFor limited use only

4/18 Approaches Word-for-wordWord-for-word Syntactic transfer*Syntactic transfer* InterlinguaInterlingua Controlled languageControlled language Example based*Example based* Statistically based*Statistically based* Various combinationsVarious combinations

5/18 Syntactic transfer Involves some linguistic rulesInvolves some linguistic rules Analyzes sources and translates using intermediary linguistic representationsAnalyzes sources and translates using intermediary linguistic representations Usage still limited for particular purposes (i.e. scientific, marketing, etc.)Usage still limited for particular purposes (i.e. scientific, marketing, etc.) Examples: Systran, Eurotra, MetalExamples: Systran, Eurotra, Metal

6/18 Example-based Uses blocks of words (example sentences)Uses blocks of words (example sentences) Utilizes analogy principleUtilizes analogy principle Needs to be fed with infoNeeds to be fed with info System “learns” during augmenting stageSystem “learns” during augmenting stage Suitable for structurally completely different languagesSuitable for structurally completely different languages Example: translation memoriesExample: translation memories

7/18 Statistically-based (a.k.a. SMT) Utilizes statistical modelsUtilizes statistical models  Parameters derived from bilingual corpora Phrases as n-grams (n is number of terms in a phrase)Phrases as n-grams (n is number of terms in a phrase) Requires vast quantities of matched bilingual textsRequires vast quantities of matched bilingual texts Outputs most likely match inputsOutputs most likely match inputs Does not apply linguistic rulesDoes not apply linguistic rules Attempts to match language patternsAttempts to match language patterns 

ProblemsProblems  Modeling / Learning / Decoding ApproachesApproaches  Word-based / Phrase-based / Syntax-based ExampleExample  Google Translate Service 8/18 Statistically based ( cont. ) 

ManualManual  Human bilingual or monolingual evaluators score outputs according to fluency (grammar) and adequacy (preservation of information)  Time-consuming, expensive and very subjective Automatic (BLEU, METEOR, etc.)Automatic (BLEU, METEOR, etc.)  Reference translations  Goal  higher degree of correlation with human judgements 9/18 II. Evaluation

10/18 III. Experimental study Croatian – EnglishCroatian – English  “Very odd couple”  A lot of systematic, idiosyncratic and lexical differences Three types of texts:Three types of texts:  Corpus linguistics, annotation and research methods  Enterprises and Government's reform plan  Washing machine manual

Google translate web UI

12/18 SMT Offers Croatian as source and target languageOffers Croatian as source and target language Statistically basedStatistically based  Monolingual target language texts  Aligned texts (human translations) Fluency and adequacy highly depend on available corporaFluency and adequacy highly depend on available corpora

Reference translations vs. candidate translationsReference translations vs. candidate translations Levels of analysis:Levels of analysis:  Lexical (misuse of words, zerotones)  Morphological (wrong word forms)  Syntactic (word order)  Semantic (preservation of original message)  Usage of punctuation marks 13/18 Task

14/18 Manual Evaluation Procedure 6 bilingual evaluators and 21 sentences (machine translation and reference translation)6 bilingual evaluators and 21 sentences (machine translation and reference translation) 1 – 5 scale1 – 5 scale Fluency Adequacy 1 incomprehensible none 2 disfluent English little meaning 3 non- native English much meaning 4 good English most meaning 5 flawless English all meaning

HypothesesHypotheses  There is no significant difference in assigning score 3 according to both criteria (fluency and adequacy).  There are no significant differences in assigning score 3 to fluency and adequacy per each evaluator. 15/18 Chi-square Test (  2 )

 T here is no significant difference in assigning score 3 according to both criteria.  There are no significant differences in assigning score 3 to fluency and adequacy per each evaluator.  There are significant differences in assigning score 3 to fluency and adequacy for half of the evaluators. 16/18 Results st2nd3rd4th5th6th Fluency Adequacy Fluency and adequacy per average judgements

Usage:Usage:  Basic information transfer  Personal use only Improvements:Improvements:  Integration with language dependent modules  Human post-editing  Greater number of evaluators needed 17/18 IV. Conclusion

Thank you on your attention!