“Retrospective vs. concurrent think-aloud protocols: usability testing of an online library catalogue.” Presented by: Aram Saponjyan & Elie Boutros

Overview: The article discusses the think aloud techniques that are used as part of usability tests. The two main think aloud approaches, retrospective and concurrent, are compared through the test of an online library catalogue. The three main points of comparison are : the detected usability problems, the overall task performance and the participants experience.

Think Aloud Protocol A method of usability evaluation. A method that allows researchers to understand the thought process of testers as they use a given product or device. It is a great method for software designers to interact with potential users and to improve their designs based on the user feedback.

RTA vs. CTA In RTA, participants are asked to perform a set of tests silently(while being video taped) and then verbalize their experience at the end of the testing session while watching themselves on tape. In CTA, participants are asked to explain their thoughts as they are testing the product. A facilitator is always present to remind them to “think aloud” in case they remain silent.

Both the retrospective and the concurrent techniques are used for usability tests of websites,GUIs, and database front ends. Both techniques are valid, useful, and widely adopted in usability tests. Both methods yield nearly unbiased software evaluations since participants do not have to recall their thoughts long after performing the tasks.

Advantages of CTA CTA tends to involve less biased thoughts since users are asked to verbalize their thinking process during task performance. “CTA is more faithful representative of a strictly task oriented usability test”. More observed problems are revealed during task completion as opposed to the RTA which depends heavily on the user’s verbalizations which take place after task completion.

Disadvantages of CTA Users might potentially feel uncomfortable verbalizing their thoughts while performing the task at hand.( especially if they are not doing so in their native spoken language) participants have an extra burden in speaking their thoughts while performing the tasks as opposed to the RTA where users have more time to verbalize the problems after task completion.

Effects of CTA disadvantages on test results. This burden did not slow down the process of task completion. However, the success rate of task completion was affected. CTA participants were less successful in completing their tasks than those who used RTA.

Advantages of RTA Participants are not burdened with the extra task of verbalizing their thoughts as they test. This will make it easier for non-native English speakers since they will have more time to think and translate their thoughts from their native language into English. Another benefit of RTA is the potential decrease in reactivity since participant can execute a task at their own pace and are not rushed in a way that can affect their normal software usage. This will make it more likely for them to not perform better nor worse than usual.

Disadvantages of RTA RTA might not be as precise in the user experience description as CTA since users are asked to describe their experience after finishing their tasks. This extra time might introduce biased judgment since participants might forget specific things they had experienced during their task performance. Overall session time is longer in RTA than it is in CTA since users of RTA not only perform their tasks but also watch these in retrospect.

Test Object. The online library catalogue was chosen to be tested because it combines the characteristics of a search engine and a website which makes it complexes enough for novice users. The participants were a group of 40 university students gathered by the mean of announcements and printed forms. The participants were of age 18 to 24 and were asked to participate in return for a financial reward.

Tasks The tasks were all equally difficult and independent in order to prevent participants from getting stuck. They were defined to cover the catalogue’s main search functions. Those search functions included the simple search, advance search, sort results and filter results.

Questionnaires Two different questionnaires were given to the participants. One at the beginning of the test session and the other at the end. The 1 st one had questions on the demographic details of the participants such as age, gender and education. The 2 nd one had questions aiming towards finding out how participants felt about participating in the experiment.

Processing of the data Total number of usability problems detected in each condition was examined. After that, a distinction was made according to the way the usability problems had surfaced in the data: through observation of the behavioral data through verbalization by the participant a combination of observation and verbalization.

Problem Types Layout problems: The participant fails to spot a particular element within a screen of the catalogue; Terminology problems: The participant does not comprehend part(s) of the terminology used in the catalogue; Data entry problems: The participant does not know how to conduct a search (i.e. enter a search term, use dropdown windows, or start the actual searching); Comprehensiveness problems: The catalogue lacks information necessary to use it effectively; Feedback problems: The catalogue fails to give relevant feedback on searches conducted.

Results 93% of all comments made by CTA participants corresponded to an observable problem in their task execution, compared to 54% of the comments of the RTA participants

Of the 72 problems that were detected, 47% were reported in both conditions, 31% were detected exclusively in the CTA condition, and another 22% were detected exclusively in the RTA condition.

This table shows that 89% of all the problem detections involved problems that were experienced by participants in both conditions.

What this tables show? The CTA participants had to verbalize and work at the same time, which gave them less time to comment on problems that were not acute. While the CTA method reveals more problems that can be observed during task performance, the RTA method depends more on the participants’ verbalizations. Verbal protocols in this study do not so much serve to reveal problems but rather to verbally support the problems that are otherwise observable.

Task performance Does double workload in CTA has an effect on the participants’ task performance? Indicators:  the successful completion of the seven tasks  the time it took the participants to complete them Result: No significant differences were found.

Participant experiences Questions:  experiences with concurrent or retrospective thinking aloud  method of working  presence of the facilitator and the recording equipment Result: No significant differences as to how the participants in both conditions experienced CAT & RAT.

CTA participants found the test situation less disturbing than the RTA participants. Explanation:  RTA participants are given more time to fill in the questionnaire.  Presence of the facilitator during the first part of the RTA test (silent task performance) is less functional than in a CTA design, and that it may be confronting for participants to see their actions back on video.  The CTA participants had to actively perform tasks and think aloud, which considerably reduced the amount of attention they could spare for noticing the facilitator and the recording equipment.

Conclusion Both methods are comparable in terms of quantitative output, they differed significantly as to how this output was established. RTA method proved to be more effective in revealing problems that were not observable, but could only be detected by means of verbalization. RTA participants tended to give explanations and suggestions, while CTA participants more often limited themselves to giving descriptions of their actions. Very limited contribution of the participant’s verbalizations to the outcome (in terms of user problems detected) of the usability test.

Conclusion The task of concurrently verbalizing thoughts caused the participants to make more errors in the process of task performing and to be less successful in completing the seven tasks. Less successful performance of CTA method lies in the participant’s workload: the difficulty of the tasks given to the participants may have been a crucial factor in this study. A strong, and new argument in favor of RTA protocols is that they may be less susceptible to the influence of task difficulty, both in terms of reactivity and in terms of completeness of the verbalizations.