Usability Assessment Methods beyond Testing Chapter 7 Evaluating without users

Observation n The goal is to become virtually invisible to users so that they will use the system and you observation how they use the system n Observing users one often finds that they are using the software in ways that would not have been expected

Questionnaires n One cannot always take user comments at face value. –Little problems could be overblown –Data about people’s actual behavior should have precedence over people’s claims of what they think they do n E.g., in a study, 26% of users commented on a command even though they had previously stated that they did not know it.

Questionnaires n Users’ predictions of new features and their rating of the features after trying them, was only 28% n In a study users were given a questionnaire about the difficulty of the instructions of a mobile phone. –When tested on the instructions, only 50% got it right. n Always pilot test the questions before hand n Only ask a question if you want to know the answer (if the replies will make a difference to the system)

Interviews n They have the advantage of being more flexible –The interviewer can rephrase a question –Or ask follow-up questions n Interviewers should ask open-ended questions (not yes or no questions) n Ask users to recall critical incidents in their USE OF THE SYSTEM

Focus Groups n Are used to assess user needs and feelings before and after system deployment n A group of at least six is run by a moderator who maintains focus n The moderator prepares a script for what issues need to brought up

Focus Groups n Only after having seen a prototype of the new system do focus groups change from being skeptical to having a feeling they would like to get new features

Logging Actual Use n Logging means having the computer collect statistics about the detailed use of the system n Frequency with which each user has used each feature n Frequency of errors –In a study, 85% of errors came from 9 common error messages

Logging Actual Use n In another study 10% of the help screens that were accessed the most accounted for 92% of the requests. n Logging is done by instrumenting low-level system software. Try to use middle-level logs. E.g., save Java button presses. n It’s possible to log complete transcripts of user sessions

Logging Actual Use n Another use of logging –Study the users’ use of a user interface to find usability problems that were over looked when observing users

User Feedback n One will tend to hear MOSTLY from dissatisfied users. So, the feedback may not be representative of the majority n You could allow users a quick or “gripe” function n Check with custom support n Use a user internet board for feedback

Combining Usability Methods n Heuristic evaluation and user testing –Heuristic methods will not waste users time –Both methods may find different usability problems.

Cognitive Walkthroughs n A formalized way of imagining people’s thoughts and actions when they use an interface for the first time.

Cognitive Walkthroughs n Anticipate problems before testing your users n Envision the users’ route on their way to complete a task n Walk through this route and identify problems they might encounter –Confusing labels, options, likely errors –Then fix the found problems