IUED Institute of Translation and Interpreting When is Less Actually More? A Usability Comparison of Two CAT Interfaces

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Overview Background: Cognitive friction, usability The study: Research questions, participants, methods Results Implications and future directions

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Background Cognitive friction: “the resistance encountered by a human intellect when it engages with a complex system of rules that change as the problem changes” (Cooper 2004, 19) Translators may be exposing themselves to unnecessary cognitive friction by the way they use their tools

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Background Cognitive friction Apparent overload of cognitive resources by the amount of information presented on crowded screen Apparent information underload of cognitive resources on a lean user interface

Availability of (adaptive) MT results When is Less Actually More? A Usability Comparison of Two CAT Interfaces Background New forms of cognitive friction may arise as a result of recent developments in modern CAT tools. Sub-segment matching Availability of (adaptive) MT results Suggestions change over the course of the translation (within segments, across segments) Potential for cognitive friction (Ehrensberger-Dow & O’Brien 2015; O’Brien et al. 2017; Teixeira & O’Brien 2017).

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces The study Typical translation task Focus on productivity Eye-tracking Keystroke-logging Post-task questionnaire

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces The study - Participants 8 MA students (2nd or 4th semester) Tools skills (self-rated) No previous experience Beginner skills Basic skills SDL Trados Studio 2017 2 6 Lilt 1

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces The study – Texts and resources Translation of 2 short comparable texts (see following slide) Language direction: English  German Translation Memory (> 6000 Translation units) Terminology (55 entries) Machine Translation (Lilt, SDL Language Cloud) Access to external resources (Online dictionaries etc.)

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces The study – Texts Text type: Academic job announcement from linguistlist.org Basic familiarity with the type of text (classroom exercises) Comparable level of difficulty Text 1 Text 2 Length 208 words 239 words Number of unique words 136 words 145 words Vocabulary density .654 .607 Average sentence length 34.7 words 34.1 words Readability (Gunning-Fog) 9.7 7.7

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results – What are we looking at? Focus on 3 segments per text Stage of translation at beginning, middle and end of the text Segment length: 2 Short segments (segments 1 and 3) 8 to 10 words/segment each 1 long segment (segment 2) 36 words/segment

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results – What are we looking at? Time on segment Number of Keystrokes Number of Deletions Number of pauses (> 3s)

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results Time on segments (in seconds) Segment 1 Segment 2 Segment 3 SDL Trados Studio 2017 62 252 Lilt 43 217 36 Percentage of Time on segment (Lilt/SDL Trados) 69% 86% 58%

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results Number of keystrokes Segment 1 Segment 2 Segment 3 SDL Trados Studio 2017 85 341 89 Lilt 35 303 72 Percentage of keystrokes (Lilt/SDL Trados) 41% 89% 81%

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results Number of deletions Segment 1 Segment 2 Segment 3 SDL Trados Studio 2017 8 31 6 Lilt 2 20 Percentage of deletions (Lilt/SDL Trados) 25% 65% 100%

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results Number of pauses (>3s) Segment 1 Segment 2 Segment 3 SDL Trados Studio 2017 6.3 17.7 6.7 Lilt 5.3 15.6 3.9 Percentage of number of pauses (Lilt/SDL Trados) 84% 88% 58%

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results Number of brief pauses (>3s and <5s) Segment 1 Segment 2 Segment 3 SDL Trados Studio 2017 2.3 7.8 3.3 Lilt 2.2 6.5 1.4 Percentage of deletions (Lilt/SDL) 96% 83% 42%

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results Number of extended pauses (>5s) Segment 1 Segment 2 Segment 3 SDL Trados Studio 2017 4 9.8 3.3 Lilt 3 9.1 1.4 Percentage of deletions (Lilt/SDL) 75% 93% 42%

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results Heatmap SDL Trados Studio 2017

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results Heatmap Lilt

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results Key Performance Indicators SDL Trados Studio 2017

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results Video Clip showing representative Eye-Tracking data for SDL Trados Studio 2017

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results Video Clip showing representative Eye-Tracking data for Lilt

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results – Summary (SDL Trados Studio 2017) Lower overall productivity with SDL Trados Studio 2017 (across all categories measured) Participants take time to get used to SDL Trados Studio 2017 interface (number of deletions decreases during task) Users do not focus on “traditional” Translation Memory results window in SDL Trados Studio 2017

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Results – Summary (Lilt) Higher overall productivity with Lilt (across all categories measured) The longer the segment, the smaller the difference in productivity (time on segment) Changing of MT suggestions is perceived to be disruptive Fewer pauses in the 3rd segment (result of Adaptive MT?)

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Implications and future directions Is there a difference in quality? Participants seem to trust MT suggestions for shorter segments more than for longer segments. Perceived or real difference in quality? How can this effect translator’s training?

When is Less Actually More When is Less Actually More? A Usability Comparison of Two CAT Interfaces Literature Cooper, Alan (2004). The Inmates are Running the Asylum: Why Hi-Tech Products Drive Us Crazy and How to Restore the Sanity. Indianapolis: Sams Publishing. Ehrensberger-Dow, Maureen & Sharon O’Brien (2015). Ergonomics of the translation workplace: Potential for cognitive friction. Special Issue of Translation Spaces, 4 (1): 98-118. O’Brien, Sharon, Maureen Ehrensberger-Dow, Marcel Hasler, & Megan Connolly (2017). Irritating CAT tool features that matter to translators. Hermes Journal of Language and Communication in Business 57. Teixeira, Carlos S. & Sharon O'Brien (2017/forthcoming). Investigating the cognitive ergonomic aspects of translation tools in a workplace setting. Special Issue of Translation Spaces 6 (1).