1. User Testing 101

2. Recruiting Users
Find people with the same experience level as the typical user
Don’t get people who are familiar with the product or your views on it.
Be careful about “friends and family” testing
Public places like libraries, dining halls, coffee shops can be good places to find people who wouldn’t mind helping for a few minutes.
Some companies have user testing labs that they set up and they handle recruiting users.
In academia, we often post fliers or set up agreements with local organizations.
A small budget to give out gift certificates or something can help.

3. User Instructions Tell users:
You are testing a piece of software, not them.
It’s ok for them to stop at any time.
How do you handle cases where people do leave?
Demonstrate equipment that users will need to use (unless the equipment is what you are testing)
(so for example, if you are developing for a Oculus and your user has never seen it, it’s entirely appropriate to show how to move around and select objects)

4. Tasks These should similar to the tasks you developed in M1.
For user testing –
Rephrase as “you”
The user influence on task goals should remain.
Note: the user details will come back in forms of testing without a user, which are coming up soon.

5. Good Task Examples….
Say what the user wants to do but do not say how he/she would do it
no assumptions made about the interface
can be used to compare design alternatives in a fair way
2. Are very specific
says exactly what the user wants to do
specifies actual items the user would somehow want to input

6. Good Task Examples…. 3. Describe a complete job
forces designer to consider how interface features work together
contrasts how information input / output flows through the dialog
where does information come from?
where does it go?
what has to happen next?
Do not
create a list of simple things the system should do
present a sub-goal independent of other sub-goals

7. Good Task Examples… 4. Say who the users are
name names, if possible
says what they know
Why?
design success strongly influenced by what users know
can go back and ask them questions later
reflects real interests of real users
helps you find tasks that illustrate functionality in that person’s real work context

8. Good Task Examples… Are evaluated
Circulate descriptions to users, and rewrite if needed
ask users for
omissions
corrections
clarifications
suggestions
As a set, identifies a broad coverage of users and task types
the typical ‘expected’ user, typical routine tasks
the occasional but important user, infrequent but important tasks
the unusual user unexpected or odd tasks

9. Photo Sharing
Jennifer’s daughter (Anna) has just gone to her first high school prom and Jennifer took a bunch of pictures using her phone. Later in the evening, she gets a notification asking if she wants to share some of the pictures with family. Jennifer wants to share a good picture of Anna by herself, one of Anna with her date, and one with the group of couples that went to the dance together. Jennifer is a fairly comfortable smart phone user, she searches for and downloads different apps frequently. However, she is also concerned about security; photos can often be labelled with meta data that can reveal information she doesn’t want to be public. She’s fine with this information going to family, but doesn’t want to share it more broadly. Jennifer has previously downloaded and used the photo sharing app.

10. Photo Sharing
GOAL: focus on the picture selection, sharing, and security. You have recently installed a new photosharing app in the hopes of staying in better contact with family. This evening, your daughter Anna has just gone to her first high school prom and you took a bunch of pictures. When you check your phone, there’s a notification asking if you would like to share some of these pictures. You’d like to share one of Anna by herself, one of Anna with her date, and one of the group of couples that went to the dance together with your family. However, you’d like to make sure that no one else has access to these photos. Here’s the notification you see…

11. Photo Sharing
GOAL: focus on the picture selection, sharing, and security. You have recently installed a new photosharing app in the hopes of staying in better contact with family. This evening, your daughter Anna has just gone to her first high school prom and you took a bunch of pictures. When you check your phone, there’s a notification asking if you would like to share some of these pictures. You’d like to share one of Anna by herself, one of Anna with her date, and one of the group of couples that went to the dance together with your family. However, you’d like to make sure that no one else has access to these photos. Here’s the notification you see…
So we’ve set the goal here of focusing on picture selection and we’re sort of forcing the user to notice the notification and try to act on it. Once we have that going, we may want to also ensure that people see the notification and understand what it means.

12. Photo Sharing
GOAL: focus on the picture selection, sharing, and security. You have recently installed a new photosharing app in the hopes of staying in better contact with family. This evening, your daughter Anna has just gone to her first high school prom and you took a bunch of pictures. When you check your phone, there’s a notification asking if you would like to share some of these pictures. You’d like to share one of Anna by herself, one of Anna with her date, and one of the group of couples that went to the dance together with your family. However, you’d like to make sure that no one else has access to these photos. Here’s the notification you see…
There’s a clear end goal here.

13. Photo Sharing
GOAL: focus on the picture selection, sharing, and security. You have recently installed a new photosharing app in the hopes of staying in better contact with family. This evening, your daughter Anna has just gone to her first high school prom and you took a bunch of pictures. When you check your phone, there’s a notification asking if you would like to share some of these pictures. You’d like to share one of Anna by herself, one of Anna with her date, and one of the group of couples that went to the dance together with your family. However, you’d like to make sure that no one else has access to these photos. Here’s the notification you see…
And, we’re keeping the security concerns.

14. Think Aloud Protocol Ask users to “think aloud” as they are working.
Explain why – rich information source for you
You may need to model it once for them
You may also want to get them to practice once with an unrelated task
Are pairs problematic?
If problematic, try retrospective.

15. As an observer Capture the user’s behavior What they do
But especially the thoughts behind it

16. Listen for… I’m looking for…. I’m guessing that… I’m confused…
Oh, so this is the…

17. Dealing with silence… Can you tell me what you’re thinking?
Can you tell me why you clicked on x?

18. No Help You *cannot* provide help. Tell users.
When users have questions, they should ask them anyway – you can note the question and answer it at the end.
In some cases, you can intercede. But. Know in advance when you’ll step in.
For example, users have to be making no progress for 3 minutes for the experimenter to help.

19. Evaluating Results
You should find lots of problems – what do you go after?
Importance – is this a nit, a minor hurdle, or a complete showstopper in terms of users completing tasks.
Difficulty – is this an easy fix or a major rewrite
(note major rewrite can to come into play when there’s a digital prototype, not on paper. That’s the point of the low-fi – you have to be willing to chuck it).

20. Stages Phase 1 – Can people tolerate this intervention at all?
Do they get sick (yes/no)?
Phase 2 – small scale lab studies
Dosing requirements
Does it work
Phase 3 – large multi-center trials
Larger numbers, sometimes less tightly controlled.
Paper prototyping – can people complete the tasks at all?
(yes/no) If not, fix
Digital Prototype – which tasks are the most problematic per cost to change?
Timing, errors, etc.
Deployed Beta –
Bug reports, complaints…real world scenarios, more variable data.