1. Interviews and Questionnaires
a.k.a. How to talk to your users

2. Agenda Questions Interviewing techniques Questionnaire design
Evaluation Plan discussions

3. First: a little exercise
Interview each other about instant messaging use
4 or 5 questions, 3 minutes
Person 1: the silent type
Person 2: the going off track type

4. Today’s focus is asking people about stuff…
Interviews
Questionnaires
Assessment of that data

5. Interviews & Questionnaires
Subjective view of participants
Quantitative – very structured
Questionnaires
often quantitative, but not entirely
Structured Interviews
Strict set of questions, deviation would compromise study
Qualitative – less or no structure
Semi-structured interviews
Some deviation encouraged
Unstructured interviews
Little guide, very explorative

6. Interviews Potentially lots of detail Can vary questions as needed
Inexpensive
Time consuming to perform and analyze
Some interpretation required
Subject to interviewer biases

7. Questionnaires Expensive to create …but cheap to administer
Easier to get quantifiable results
Can gather info from many more people
Protects participant identity
Only as good as the questions asked

8. Structured Interviews
More similar to questionnaires
Require a lot of training for any hope at inter-interviewer reliability
But that means that they tend to give much more repeatable results

9. Unstructured Interviews
Have a plan, but keep interview open to different directions
Get participant to open up and express themselves in their terms and at own pace
Create interpretations with users
Be sure to use their terminology
Take lots of time, but learn a lot as well

10. Semi-Structured Interviews
Predetermine data of interest - know why you are asking questions - don’t waste time
Plan for effective question types
How do you perform task x?
Why do you perform task x?
Under what conditions do you perform task x?
What do you do before you perform…?
What information do you need to…?
Whom do you need to communicate with to …?
What do you use to…?
What happens after you…?
See Gordon & Gill, 1992; Graesser, Lang, & Elofson, 1987

11. Asking Questions Understand your goals
Consider the ordering of the questions
Avoid complex/long/multiple questions
Avoid jargon; talk in participant’s language
Be careful of stereotypes, biases

12. Clarity is important
Questions must be clear, succinct, and unambiguous
How much time have you spent reading news on the Web recently?
Some
A lot
Every day
Rarely
Etc.
None
0 to 5 hours
6 to 10 hours
11 to 20 hours
More than 20 hours

13. Avoid question bias
Leading questions unnecessarily force certain answers.
Do you think parking on campus can be made easier?
What is your overall impression of…
1.Superb
2.Excellent
3.Great
4.Not so Great

14. Be aware of connotations
Do you agree with the NFL owner’s decision to oppose the referee’s pay request?
Do you agree with the NFL owner’s decision in regards to the referee’s pay demand?
Do you agree with the NFL owner’s decision in regards to the referee’s suggested pay?

15. Leading questions
People want to do well, give you what you are looking for
Be aware of your own expectations before creating questions and while interviewing
Use value neutral terms
What do you like about this system?
Vs.
Tell me what you thought about this system.

16. Avoid hypotheticals Avoid gathering information on uninformed opinions
Subjects should not be asked to consider something they’ve never thought about (or know or understand)
Would a device aimed to make cooking easier help you?

17. Handle personal info carefully
Ask questions subjects would not mind answering honestly.
What is your age?
What is your waist size?
If subjects are uncomfortable, you will lose their trust
Ask only what you really need to know

18. What’s wrong with this picture?
How much easier is it to use this client than Outlook?
I see you choose to use your keyboard shortcuts more than the mouse. Is that faster for you?
Your choice of red is different than any other user we saw. Why did you do that?

19. Planning your interview:
Introduction
Warmup
Main session
Cool-off
Closing
Record everything exactly in your participants’ language
(don’t forget to test your recording equipment)

20. The warmup or “grand tour” question
The first question helps set the tone for the interview
Familiarize the participant to talking
Encourage the participant that their true opinion does matter
Question should be
Easy to answer
But not answered easily
More than just a “yes” or “no” response
Examples:
Tell me about the work you do?
What made you buy the computer?

21. Prompts
“Nudge” a participant in a direction, or to get additional response
Silent: remain silent until they say more
Echo: repeat back and then ask “then what happens” etc.
Make agreeing sounds: you say “uh huh” and the other person continues
Tell Me More: could you tell me more about that?
Clarifying: summarize and ask for confirmation or clarification, often leads to new discussion

22. Contents of a survey General/Background info Objective questions
Demographic data
Also functions as a “warm up”
Correlate responses between groups
Objective questions
Open-ended/subjective

23. Background examples Demographic data: Age, gender Task expertise
i.e. Have you ever worked in a restaurant?
Motivation
Frequency of use
How often do you…
Education/literacy
What training have you had in …?

24. Closed Format Restricting set of choices Quantifiable Disadvantages
Must cover whole range
All should be equally likely
Don’t get interesting, “different” reactions
Advantages
Clarify alternatives
Easily quantifiable
Eliminate useless answer

25. Many forms of response Dichotomous Multiple Choice Multiple Response
Rank/Match
Likert
Rating

26. Questionnaire Styles Rank from 1 - Very helpful
2 - Ambivalent
3 - Not helpful
0 - Unused
Which word processing systems do you use?
LaTeX
Word
___ Tutorial
___ On-line help
___ Documentation
FrameMaker
WordPerfect

27. Likert-type scale Typical scale uses 5, 7 or 9 choices
Above that is hard to discern
Doing an odd number gives the neutral choice in the middle
You may not want to give a neutral option
Characters on screen were:
hard to read easy to read

28. What’s wrong with this picture?
1. What is your age? _______________
2. How long have you used the internet?
<1 year
1-3 years
3-5 years
>5 years
3. How do you get information about courses?
Web site
Flyers
Registration booklet
Advisor
Other students
4. How useful is the Internet in getting information about courses?
___________________________________________________________
-- people might be uncomfortable with specific age
-- overlap in frst set of checkboxes
-- no none or other in number 4
-- not clear how many boxes to check
-- what does “useful” mean?
-- not much space for useful answer

29. On line questionnaires
or internet
Change checkboxes into dropdowns, etc
Take advantage of the technology – check input
Ensure its as accessible as paper (browser and client compatibility)
Ensure confidentiality – how is this different from paper?

30. Free Web Survey Tools Survey Monkey Survey Share phpESP
Survey Share
phpESP
Open Source surveys using PHP.

31. Analyzing your quantitative data
“Code” open ended responses or interview questions to make quantitative
Categorize all responses
Look for trends in the data
Count, average, tabulate
Make charts, etc
Run statistical analysis
Use lo-fi methods (post-its, affinity diagrams, etc)

32. Analyzing qualitative data
Find interesting cases, responses
Look for patterns of responses
Use post-its, affinity diagrams, etc.
Look for any useful suggestions, improvements, explanations that help you improve your design
Gather illustrative quotes from users that demonstrate your conclusions

33. Evaluation discussion
Someone else should be able to pick up your plan and execute it.
Be as SPECIFIC as possible
What criteria are important?
What tasks EXACTLY?
What data? How will you record?
What questions will you ask?