1. Gathering Usability Data
Observing users & subjective data

2. Directing Sessions Issues: Are you in same room or not?
Single person session or pairs of people
Objective data -- stay detached
Fall 2006
PSYCH / CS 6750

3. Collecting Data Data gathering Note-taking Audio and video tape
Instrumented user interface
Post-experiment questions and interviews
Fall 2006
PSYCH / CS 6750

4. Collecting Data Identifying errors can be difficult
Qualitative techniques
Think-aloud - can be very helpful
Post-hoc verbal protocol - review video
Critical incident logging - positive & negative
Structured interviews - good questions
“What did you like best/least?”
“How would you change..?”
Fall 2006
PSYCH / CS 6750

5. Observing Users Not as easy as you think
One of the best ways to gather feedback about your interface
Watch, listen and learn as a person interacts with your system
Fall 2006
PSYCH / CS 6750

6. Observation Direct Indirect In same room Can be intrusive
Users aware of your presence
Only see it one time
May use 1-way mirror to reduce intrusiveness
Indirect
Video recording
Reduces intrusiveness, but doesn’t eliminate it
Cameras focused on screen, face & keyboard
Gives archival record, but can spend a lot of time reviewing it
Fall 2006
PSYCH / CS 6750

7. Location Observations may be
In lab - Maybe a specially built usability lab
Easier to control
Can have user complete set of tasks
In field
Watch their everyday actions
More realistic
Harder to control other factors
Fall 2006
PSYCH / CS 6750

8. Challenge
In simple observation, you observe actions but don’t know what’s going on in their head
Often utilize some form of verbal protocol where users describe their thoughts
Fall 2006
PSYCH / CS 6750

9. Verbal Protocol One technique: Think-aloud
User describes verbally what s/he is thinking and doing
What they believe is happening
Why they take an action
What they are trying to do
Fall 2006
PSYCH / CS 6750

10. Think Aloud Very widely used, useful technique
Allows you to understand user’s thought processes better
Potential problems:
Can be awkward for participant
Thinking aloud can modify way user performs task
Fall 2006
PSYCH / CS 6750

11. Teams
Another technique: Co-discovery learning (Constructive interation)
Join pairs of participants to work together
Use think aloud
Perhaps have one person be semi-expert (coach) and one be novice
More natural (like conversation) so removes some awkwardness of individual think aloud
Fall 2006
PSYCH / CS 6750

12. Alternative
What if thinking aloud during session will be too disruptive?
Can use post-event protocol
User performs session, then watches video afterwards and describes what s/he was thinking
Sometimes difficult to recall
Opens up door of interpretation
Fall 2006
PSYCH / CS 6750

13. Historical Record
In observing users, how do you capture events in the session for later analysis? ?
Fall 2006
PSYCH / CS 6750

14. Capturing a Session 1. Paper & pencil Can be slow May miss things
Is definitely cheap and easy
Task Task Task …
Time 10:00
10:03
10:08
10:22 S e S e
Fall 2006
PSYCH / CS 6750

15. Capturing a Session 2. Recording (audio and/or video)
Good for talk-aloud
Hard to tie to interface
Multiple cameras probably needed
Good, rich record of session
Can be intrusive
Can be painful to transcribe and analyze
Fall 2006
PSYCH / CS 6750

16. Capturing a Session 3. Software logging
Modify software to log user actions
Can give time-stamped key press or mouse event
Two problems:
Too low-level, want higher level events
Massive amount of data, need analysis tools
Fall 2006
PSYCH / CS 6750

17. Issues What if user gets stuck on a task? You can ask
“What are you trying to do..?”
“What made you think..?”
“How would you like to perform..?”
“What would make this easier to accomplish..?”
Maybe offer hints
Can provide design ideas
Fall 2006
PSYCH / CS 6750

18. Subjective Data
Satisfaction is an important factor in performance over time
Learning what people prefer is valuable data to gather
Fall 2006
PSYCH / CS 6750

19. Methods Ways of gathering subjective data (Focus on first two)
Questionnaires
Interviews
Booths (eg, trade show)
Call-in product hot-line
Field support workers
(Focus on first two)
Fall 2006
PSYCH / CS 6750

20. Questionnaires Preparation is expensive, but administration is cheap
Oral vs. written
Oral advs: Can ask follow-up questions
Oral disadvs: Costly, time-consuming
Forms can provide better quantitative data
Fall 2006
PSYCH / CS 6750

21. Questionnaires Issues Only as good as questions you ask
Establish purpose of questionnaire
Don’t ask things that you will not use
Who is your audience?
How do you deliver and collect questionnaire?
Fall 2006
PSYCH / CS 6750

22. Questionnaire Topic
Can gather demographic data and data about the interface being studied
Demographic data:
Age, gender
Task expertise
Motivation
Frequency of use
Education/literacy
Fall 2006
PSYCH / CS 6750

23. Interface Data Can gather data about screen graphic design terminology
capabilities
learning
overall impression
...
Fall 2006
PSYCH / CS 6750

24. Question Format Closed format Answer restricted to a set of choices
Typically very quantifiable
Variety of styles
Fall 2006
PSYCH / CS 6750

25. Closed Format Likert Scale Typical scale uses 5, 7 or 9 choices
Above that is hard to discern
Doing an odd number gives the neutral choice in the middle
Characters on screen
hard to read easy to read
Fall 2006
PSYCH / CS 6750

26. Other Styles Rank from 1 - Very helpful
2 - Ambivalent
3 - Not helpful
0 - Unused
Which word processing systems do you use?
LaTeX
Word
___ Tutorial
___ On-line help
___ Documentation
FrameMaker
WordPerfect
Fall 2006
PSYCH / CS 6750

27. Closed Format Advantages Disadvantages Clarify alternatives
Easily quantifiable
Eliminate useless answer
Disadvantages
Must cover whole range
All should be equally likely
Don’t get interesting, “different” reactions
Fall 2006
PSYCH / CS 6750

28. Open Format Asks for unprompted opinions
Good for general, subjective information, but difficult to analyze rigorously
May help with design ideas
“Can you suggest improvements to this interface?”
Fall 2006
PSYCH / CS 6750

29. Questionnaire Issues Question specificity Language Clarity
“Do you have a computer?”
Language
Beware of terminology, jargon
Clarity
Leading questions
Can be phrased either positive or negative
Fall 2006
PSYCH / CS 6750

30. Questionnaire Issues Prestige bias Embarrassing questions
People answer a certain way because they want you to think that way about them
Embarrassing questions
Hypothetical questions
“Halo effect”
When estimate of one feature affects estimate of another (eg, intelligence/looks)
Fall 2006
PSYCH / CS 6750

31. Deployment Steps Discuss questions among team
Administer verbally/written to a few people (pilot). Verbally query about thoughts on questions
Administer final test
Fall 2006
PSYCH / CS 6750

32. Interviews
Get user’s viewpoint directly, but certainly a subjective view
Advantages:
Can vary level of detail as issue arises
Good for more exploratory type questions which may lead to helpful, constructive suggestions
Fall 2006
PSYCH / CS 6750

33. Interviews Disadvantages Subjective view
Interviewer can bias the interview
User may not appropriately characterize usage
Time-consuming
Fall 2006
PSYCH / CS 6750

34. Interview Process How to Can be done in groups
Plan a set of questions (provides for some consistency)
Don’t ask leading questions
“Did you think the use of an icon there was really good?”
Can be done in groups
Get consensus, get lively discussion going
Fall 2006
PSYCH / CS 6750

35. Data Analysis Simple analysis Determine
Determine the means (time, # of errors, etc.) and compare with goal values (coming up…)
Determine
Why did the problems occur?
What were their causes?
Fall 2006
PSYCH / CS 6750

36. Experimental Results
How does one know if an experiment’s results mean anything or confirm any beliefs?
Example: 20 people participated, 11 preferred interface A, 9 preferred interface B
What do you conclude?
Fall 2006
PSYCH / CS 6750

37. Hypothesis Testing
In experiment, we set up a “null hypothesis” to check
Basically, it says that what occurred was simply because of chance
For example, any participant has an equal chance of preferring interface A over interface B
Fall 2006
PSYCH / CS 6750

38. Hypothesis Testing
If probability result happened by chance is low, then your results are said to be “significant”
Statistical measures of significance levels
0.05 often used
Less than 5% possibility it occurred by chance
Fall 2006
PSYCH / CS 6750

39. Presentation Techniques
Middle 50%
low
high
Age
Mean 20
Time in secs.
Fall 2006
PSYCH / CS 6750

40. Upcoming
Audio
Web
Fall 2006
PSYCH / CS 6750

41. Using the Results How do you use the results of your evaluation?
How can you make your design better with this knowledge?
Fall 2006
PSYCH / CS 6750