Joke Daems PhD student Lieve Macken, Sonia Vandepitte, Robert Hartsuiker Comparing HT and PE using advanced research tools

Overview Experimental setup Process – Speed – Translation units – Cognitive load – Gaze behavior – External resources Product – Acceptability – Adequacy – Influence of MT on PE quality – Overall quality: Influence of external resources

Overview Experimental setup Process – Speed – Translation units – Cognitive load – Gaze behavior – External resources Product – Acceptability – Adequacy – Influence of MT on PE quality – Overall quality: Influence of external resources

Experimental setup Text selection Participants Session Tools Data collected

Text selection 15 newspaper articles (+/- 150 words) from Newsela.com Readability metrics Translation problems MT quality  8 texts selected

Participants 10 master’s students of translation (pass general translation exam) No previous experience post-editing Reward: 2 coupons of 50 euros each

Session overview Session 1: – Survey – LexTALE test (proficiency) – Warmup task – 2 texts HT, 2 texts PE Session 2: – Warmup task – 2 texts PE, 2 texts HT – Retrospection – Survey

Experimental setup: Design ParticipantP1P3P5P7P9P2P4P6P8P10 Session1task1PE_1PE_8PE_7PE_6PE_5HT_1HT_8HT_7HT_6HT_5 task2PE_2PE_1PE_8PE_7PE_6HT_2HT_1HT_8HT_7HT_6 task3HT_3HT_2HT_1HT_8HT_7PE_3PE_2PE_1PE_8PE_7 task4HT_4HT_3HT_2HT_1HT_8PE_4PE_3PE_2PE_1PE_8 Session2task5HT_5HT_4HT_3HT_2HT_1PE_5PE_4PE_3PE_2PE_1 task6HT_6HT_5HT_4HT_3HT_2PE_6PE_5PE_4PE_3PE_2 task7PE_7PE_6PE_5PE_4PE_3HT_7HT_6HT_5HT_4HT_3 task8PE_8PE_7PE_6PE_5PE_4HT_8HT_7HT_6HT_5HT_4

Tools EyeLink (eye-tracking) Casmacat (keystrokes + compatibility EyeLink) Inputlog (keystrokes + logging of external resources)

Data 80 sessions – 40 HT, 40 PE – 10 for each text – 8 for each translator Keystroke logging (Casmacat + Inputlog) & eye-tracking Pre- & post surveys Annotation of problems

Overview Experimental setup Process – Speed – Translation units – Cognitive load – Gaze behavior – External resources Product – Acceptability – Adequacy – Influence of MT on PE quality – Overall quality: Influence of external resources

Translation speed  ms (± standard error) more in HT

Production units  0,1 PU (± 0,019 se) more per ST token in HT  1295ms (± 227 se) more per PU in HT

Cognitive load Post-task survey: Which translation method was most tiring? – 5 respondents: HT and PE equally tiring – 1 respondent: PE most tiring – 4 respondents: HT most tiring

Cognitive load What makes HT so tiring? – “Having to start from nothing” – “Insecurity, I need to double-check everything because there’s no basic structure as with PE” – “Looking up synonyms, finding the correct words” What makes PE so tiring? – “Letting go of the MT output” – “Noticing errors in the MT output” – “Making sure the structure is still okay after fixing one part of the sentence”

Gaze behavior: duration  Longer fixations on TT (cognitive load?)  Difference greatest for HT (harder to understand MT without ST? No effect found for MT quality )

Gaze behaviour: # fixations  More fixations on TT  Difference greatest for PE (MT output? No effect found for MT quality )

Gaze behaviour: # fixations ST and TT  ST: 2,3 (± 0,4 se) more fixations per ST token in HT  TT: 1,3 (± 0,5 se) fewer fixations per ST token in HT

External resources

 ms (± standard error) more time in HT No significant effect of MT quality

Overview Experimental setup Process – Speed – Translation units – Cognitive load – Gaze behavior – External resources Product – Acceptability – Adequacy – Influence of MT on PE quality – Overall quality: Influence of external resources

Quality: acceptability Task in itself no significant predictor Participant * Task: R²=0,44, p<0,001

Quality: adequacy Task in itself no significant predictor Participant * Task no significant predictor Post-editing adequacy error score: negatively influenced by MT acceptability error score

PE adequacy ~ MT acceptability  Having to solve many acceptability issues might make students more aware of adequacy issues as well.

Overall quality: influence of external resources  Strategies used when translating not always successful when post-editing

Conclusions PE faster than HT More & longer production units in HT HT perceived as more tiring Longer fixations on TT (more so in HT) More fixations on TT (more so in PE) HT: more fixations ST, PE: more fixations TT More time in external resources in HT Acceptability more dependent on participant than task PE adequacy influenced by MT acceptability Consulting external resources more effective in HT

In sum PE is faster than HT, cognitively less demanding, and requires less usage of external resources to obtain a product of comparable quality. But, students need to be trained to better use external resources when post-editing.

Future research More fine-grained analysis: – Segment level – Comparison per participant – Comparison per text – Problem-solving strategies – Difficult passages Repeat with professional translators

Thank you for your time Questions? Suggestions?