1. Usability Testing & Think Aloud
2/20/14
HCC 729

2. In-class Homework Critiques
What happened after the UARs?
Pair up and share your HEs with another group (5 minutes each)
Identify good examples, questions

3. Sharing example HEs
Good examples
Questions / unsure?

4. UAR Awards Best written UAR Funniest usability problem
Interesting edge case

5. Discussion Inspirations or UI news? Readings
For next time: I’ll pick selected examples (to save time)

6. Recap from last week HTAs & personas feedback coming soon (Saturday)
Heuristic evaluation process

7. Phases of Heuristic Evaluation
0) Pre-evaluation training (optional)
Give evaluators needed domain knowledge & information on the scenario
Evaluate the interface to find usability problems
Record the problem
Aggregate problems
Assign severity rating
5) Find a solution complexity rating

8. Find which heuristic is violated
Simple & Natural Dialog
Speak User’s Language
Minimize User’s Memory Load
Consistency
Feedback
Clearly Marked Exits
Shortcuts
Good Error Messages
Prevent Errors
Help & Documentation
Nielsen’s 10
Heuristics

9. Marie Silverstrim

10. Today: Working with Real Users

11. When should you user test?
Early and often

12. Categories of user testing
Formative user testing
Usually happens early in the design process
Identify problems for the next round of design
Can be done with non-functional prototypes
Summative user testing
Done later in the development process
Can be part of QA

13. Assumptions of user testing
Even an expert can’t find all the usability challenges
Expertise can blind us problems
Domain experts, technical experts
Understanding the system image can help us to find bugs
People focus on aesthetics / other issues

14. Use it to validate/asses an interface
Think Aloud Method
Usually used as a summative evaluation technique:
Use it to validate/asses an interface
Could be used as a Formative one:
Use Think Aloud to discover process

15. Goals for Today Things you will learn:
How to conduct a Think Aloud
When to conduct a Think Aloud
What data can & should be captured
Preparation: picking users & tasks
Think Alouds are useful to discover how users think about doing a task
Interface evaluation, problem solving, learning, etc.
Follows detailed usability testing

16. How to conduct a Think Aloud Evaluation

17. What is the Think Aloud Method?
“[Think-Aloud]...may be the single most valuable usability engineering method” [Nielsen, p. 195]
An empirical technique where you ask users to “think aloud” as they work on a task you believe is interesting or important, using the interface you are interested in improving

18. Think Aloud Metrics Think Aloud is about observing
how users think about tasks
where users have problems with a system
What kind of information do you get?
Qualitative Insights (from a few users)
What kind of information do you NOT get?
Performance Measurement (time, accuracy)
This comes from a usability test

19. Think Aloud Process Preparation: Design a test plan
Preparation: Choose what to observe
Preparation: Prepare for the user test
Run the user test
Aggregate, organize, and analyze your notes

20. Think Aloud Process Preparation: Design a test plan
Preparation: Choose what to observe
Preparation: Prepare for the user test
Run the user test
Aggregate, organize, and analyze your notes

21. Preparation: Design a Test Plan
What problem is your system trying to solve?
Who are your target users?
What scenarios are you evaluating?
What are your usability goals?
What is your hypothesis?

22. Think Aloud Process Preparation: Design a test plan
Preparation: Choose what to observe
Preparation: Prepare for the user test
Run the user test
Aggregate, organize, and analyze your notes

23. Choose Tasks to Observe
Choose what to observe
Don’t pick obvious or impossible tasks
Should be more than just “try out the interface”
Pick tasks that are: representative, critical, frequent
Often better to integrate small tasks as a sequence

24. Example Task Sheet Company ABC’s photo website
Task 1: A friend mentions ABC’s website. You Google it & reach its front page. Without leaving this page, think aloud about your thoughts. What would you click next?
Task 2: You’re preparing a talk about Paris & need some photos for illustrations. Use ABC’s website to find a photo from Paris. Remember to think aloud as you work.
Task 3: You’ve found good photos from Paris, and you’d like to bookmark them for future use. How would you bookmark a photo that you are viewing now? Remember to think aloud as you work.
Task 4: Your friends have asked for your Rome photos from last year. Publish one photo from Rome using ABC’s website. The photos are on your desktop in the “Rome” folder. Remember to think aloud as you work.

25. Notes about the Task Sheet
Give each user multiple tasks to perform
Break complex tasks into small parts
Not all tasks need to start in the beginning
Clearly structured tasks help the user focus on evalution
Make the tasks independent from each other so you can skip tasks and move around
i.e. make the tasks independent
Always remind them to think aloud

26. Think Aloud Process Preparation: Design a test plan
Preparation: Choose what to observe
Preparation: Prepare for the user test
Run the user test
Aggregate, organize, and analyze your notes

27. #3 Piloting your user test
It is always good to “pilot” your user test before you recruit real users
Practice your think aloud
Learn the script
Test out the tasks, see how long it takes

28. Think Aloud Process Preparation: Design a test plan
Preparation: Choose what to observe
Preparation: Prepare for the user test
Run the user test
Aggregate, organize, and analyze your notes

29. Run a Think Aloud: Warm-up
Introduce yourself
Describe Think Aloud:
May initially be awkward
It’s important that they talk
You may prompt them
Best to provide examples
Promise confidentiality
Get permission to record

30. Run a Think Aloud: Tasks
Introduce your system
Don’t rationalize or defend system
Remind them this isn’t your system
Describe the tasks
Use your Task Sheet
User can read these aloud
Ask for questions

31. Run a Think Aloud: Tasks
Keep the user talking
A few phrase you might use
“Please keep talking”
“Tell me what you are thinking”
“Tell me what you’re trying to do"
“Can you tell me more?”
“What is your opinion?”

32. Run a Think Aloud: Tasks
You are only an observer
Don’t help do the task
Unless they are really struggling and taking too long
Do respond to & note questions
Take good notes
Record all problems, not just what you were looking for
Helpful: watch, A/V recordings
Must: take notes

33. Notes about your notes
Keep track of notes for each task (for each user)
Keep track of the actions and the behaviors you saw for each action
Include important quotes
Find the fastest note taking medium that isn’t distracting

34. User Test Notes Example: Company ABC’s photo website
User 1: Task 2 (search for photos)
Scrolled through several pages of photos
Found Paris photos but didn’t use search button
User 1: Task 3 (bookmark photos)
Successfully clicked on bookmark button with no delay
Used copy and paste to enter bookmark data
Confused by the fact that fields aren’t necessarily in the same order as they would normally be entered
User 2: Task 2 (search for photos)
“so...it’s not one of the categories...so I’ll search for it”
“I want to search for the phrase Paris, but don’t see a way to do it other than paging” (finds photos by clicking Next button)

35. Run a Think Aloud: Tasks
When it’s over, it’s helpful to ask overall impressions
What were the best three things about it?
What were the worst three things about it?
Thank the user
Provide any compensation

36. Think Aloud Process Preparation: Design a test plan
Preparation: Choose what to observe
Preparation: Prepare for the user test
Run the user test
Aggregate, organize, and analyze your notes

37. Organize your notes Aggregate, organize, and analyze your notes
This is not just about transcribing what you recorded.
You need to analyze both what they say & do.
Establish criteria for “critical incidents”
Record these critical incidents on UAR forms

38. Critical Incidents First introduced by Flanagan, 1954
“To be critical, an incident must occur in a situation where the purpose or intent of the act seems fairly clear to the observer and where its consequences are sufficiently definite to leave little doubt concerning its effects.”
In other words: describes a discrete event
Observable behavior leads to specific consequence
Can be good or bad!

39. Bad Critical Incidents
Possible criteria:
User articulates goal but doesn’t attain goal within [3 minutes]
User articulates goal, tries several things (or same thing) before giving up
User articulates goal but tries multiple ways before finding a solution
User accomplishes task but in a suboptimal way
User expresses surprise or hesitation
User expresses some negative sentiment, either about interface or about their skills

40. Good Critical Incidents
Possible criteria:
User expressed some positive affect or says something is really easy
User expresses happy surprise
Some previous analysis has predicted a usability problem, but user has no difficulty with that aspect of the system

41. Use the data
Make sure your reflections are supported directly by the data
Don’t include your own hunches (there are other opportunities to do this)

42. Record your evidence using UARs!

43. Critical Incident UARs
1. Summarize user test notes
2. List critical incidents: both good & bad
3. Complete UARs: refer back to raw data
4. Try to judge why each difficulty occurred

44. User Test Report What did Think Aloud tell you? Rethink your design
Does the UI work the way you thought it would?
Do the users take the approaches you expected?
Is something missing from your UI?
Rethink your design
Based on UAR rankings, what fixes will you work on?
Redesign is often more than just fixing bugs

45. Live demo
Need two volunteers

46. Pro Tips

47. Notes Search “snow boots”! Jump to bottom What do widths mean?
Repeat size
Colors
Heel styles – what were they?
Walking through hierarchy
Didn’t see “womens”
Hard to see pictures
Positive – said what was missing
Backtracking
Out of stock

48. Notes on think aloud Letting her know when to stop
Hard not to call out
Not clear when to end
Hard to keep track of both

49. Pro tips from our demo?

50. Think Aloud Introspection
Think Alouds can disrupt a user’s thought process
When people are trying to figure something out
When a problem requires “insight”
If you notice your participant looking confused, now might not be exactly the time to push them to “Keep Talking”

51. Think Alouds: Avoid “Why”
“Why” questions also lead to introspection
“Why did you do that?”
“What does that mean?”
During Think Aloud, best not to ask “why”
Introspection can change your process
If you ask a question, people will always give an answer…
Remember: TA is about observing raw process, a contextual inquiry is about understanding & interpreting context

52. For next week
Assignment
Readings

53. Reading None! Focus on your think aloud evaluation.
Don’t wait until the last minute!

54. Assignment
Perform a think aloud with 3 people (4 if in a group) who represents someone from your user analysis
Complete CI UARs based on what you saw
Fill out the top part for all users first
Aggregate across all users
Complete the bottom half
Have them do 5 tasks (ideally from your TA)
Everyone: Write 200 words about what you learned

55. In-class exercise
Work in pairs (with someone who didn’t do a UAR for your site) and conduct a think aloud for the task you mapped out in your HTA.
Take notes on the back of the UAR sheet
Fill out a CI UAR together