1. Interviews and Questionnaires
a.k.a. How to talk to your users

2. Agenda Questions Interviewing techniques Questionnaire design
Evaluation Plan discussions

3. Reminder: Part 3 Evaluation Plan Prototypes
We’ll talk about your first draft today
Prototypes
Can have more than one
Can be low-fi – drawings, storyboards, scenarios
Or hi-fi – Flash, VB, Java
Or physical – foam, playdough, duct tape 
BUILD PROTOTYPES THAT ALLOW YOU TO TEST WHAT YOU NEED TO TEST AND DO YOUR EVALUATION RIGHT

4. Making an evaluation plan
What criteria are important?
What resources available?
evaluators, prototype, subjects
Required authenticity of system

5. Today’s focus is asking people about stuff…
Interviews
Questionnaires
Assessment of that data

6. Guidelines for both Consider the ordering of the questions
Avoid complex/long/multiple questions
Avoid jargon; talk in participant’s language
Be careful of stereotypes, biases

7. Clarity is important
Questions must be clear, succinct, and unambiguous
How much time have you spent reading news on the Web recently?
Some
A lot
Every day
Rarely
Etc.
None
0 to 5 hours
6 to 10 hours
11 to 20 hours
More than 20 hours

8. Avoid question bias
Leading questions unnecessarily force a certain answers.
Do you think parking on campus can be made easier?
What is your overall impression of…
1.Superb
2.Excellent
3.Great
4.Not so Great

9. Be aware of connotations
Do you agree with the NFL owner’s decision to oppose the referee’s pay request?
Do you agree with the NFL owner’s decision in regards to the referee’s pay demand?
Do you agree with the NFL owner’s decision in regards to the referee’s suggested pay?

10. Handle personal info carefully
Ask questions subjects would not mind answering honestly.
What is your waist size?
All men wear a 32!!!
If subjects are uncomfortable, you will lose their trust
Ask only what you really need to know

11. Avoid hypotheticals Avoid gathering information on uninformed opinions
Subjects should not be asked to consider something they’ve never thought about (or know or understand)
Would a device aimed to make cooking easier help you?

12. Interview design Summative or formative Quantitative or qualitative
Usually inexpensive way to get detailed information

13. Interviews
Get user’s viewpoint directly, but certainly a subjective view
Advantages:
Can vary level of detail as issue arises
Good for more exploratory type questions which may lead to helpful, constructive suggestions

14. Interviews Disadvantages Subjective view
Interviewer(s) can bias the interview
Must interpret responses
User may not appropriately characterize usage
Time-consuming
Hard to quantify

15. Preparing for your interviews
Get buy-in from your participants
Develop a rapport beforehand if possible
Meet on their ground (if possible)
Have your informed consent ready and available
Test your recording equipment

16. A good interview follows a plan:
Introduction
Warmup
Main session
Cool-off
Closing
Record everything exactly in your participants’ languages

17. Types of interviews Structured Unstructured Semi-structured Group
Efficient
Require training specific to interview needs
Unstructured
Inefficient
No new specific training (outside of the training that one needs to be a good unstructured interviewer)
Semi-structured
Good balance
Often appropriate
Group
Focus groups

18. Unstructured Interviews
Have a question plan, but keep interview open to different directions
Be specific
Create interpretations with users
Be sure to use their terminology
At end, query “What should I have asked?”
Record interviews

19. Structured Interviews
More similar to questionnaires
Require a lot of training for any hope at inter-interviewer reliability
But that means that they tend to give much more repeatable results

20. Semi-Structured Interviews
Predetermine data of interest - know why you are asking questions - don’t waste time
Plan for effective question types
How do you perform task x?
Why do you perform task x?
Under what conditions do you perform task x?
What do you do before you perform…?
What information do you need to…?
Whom do you need to communicate with to …?
What do you use to…?
What happens after you…?
See Gordon & Gill, 1992; Graesser, Lang, & Elofson, 1987

21. What’s wrong with this picture?
How much easier is it to use this client than Outlook?
I see you choose to use your keyboard shortcuts more than the mouse. Is that faster for you?
Your choice of red is different than any other user we saw. Why did you do that?

22. Analyzing interview data
Depends on the type of interview
Structured interviews can be analyzed like questionnaires
Unstructured interviews generate data like that from participant observation
It is best to analyze unstructured interviews as soon as possible to identify topics and themes from the data

23. Analyzing interview data
Find interesting cases, responses
Look for patterns of responses
Use post-its, affinity diagrams, etc.
“Code” responses – categorize all responses of a certain question
Look for any useful suggestions, improvements, explanations that help you improve your design

24. Questionnaire design Summative or formative
Quantitative or qualitative
Usually inexpensive way to get lots of information from many users (but not necessarily detailed)

25. When to use a questionnaire?
When resources (time & money) are limited
Expensive to create, but cheap to administer
When you want quantitative results
When it is necessary to protect participant privacy
When corroborating other findings

26. Questionnaire issues Only as good as questions you ask
Establish purpose of questionnaire
What questions are you trying to answer?
Don’t ask things that you will not use
Who is your audience?
How do you deliver and collect questionnaire?

27. Contents of a survey General/Background info Objective questions
name, experience
Objective questions
Open-ended/subjective

28. Collecting demographic data
Usually done at the beginning of the questionnaire
Helps to “warm up” participant
Used to correlate responses within and between participant groups
Only ask for information that you think will be important and necessary

29. Background examples Demographic data: Age, gender Task expertise
i.e. Have you ever worked in a restaurant?
Motivation
Frequency of use
How often do you…
Education/literacy
What training have you had in …?

30. Interface Data Can gather data about screen graphic design terminology
capabilities
learning
overall impression
...

31. Closed Format Restricting set of choices Quantifiable Disadvantages
Must cover whole range
All should be equally likely
Don’t get interesting, “different” reactions
Advantages
Clarify alternatives
Easily quantifiable
Eliminate useless answer

32. Many forms of response Dichotomous Multiple Choice Multiple Response
Rank/Match
Likert
Semantic Differential
Rating

33. Questionnaire Styles Rank from 1 - Very helpful
2 - Ambivalent
3 - Not helpful
0 - Unused
Which word processing systems do you use?
LaTeX
Word
___ Tutorial
___ On-line help
___ Documentation
FrameMaker
WordPerfect

34. Likert-type scale Typical scale uses 5, 7 or 9 choices
Above that is hard to discern
Doing an odd number gives the neutral choice in the middle
You may not want to give a neutral option
Characters on screen were:
hard to read easy to read

35. What’s wrong with this picture?
2. What grade are you in? _______________
3. How long have you used the internet?
<1 year
1-3 years
3-5 years
>5 years
4. How do you get information about courses?
Web site
Flyers
Registration booklet
Advisor
Other students
5. How useful is the Internet in getting information about courses?
___________________________________________________________
-- people might be uncomfortable with specific age
-- overlap in frst set of checkboxes
-- no none or other in number 4
-- not clear how many boxes to check
-- what does “useful” mean?
-- not much space for useful answer

36. Advice on survey design
Take your own survey first
Know what answers you are trying to elicit
Too long, and you’ll be sorry

37. Some more advice..
Make your questions clear and specific – you can’t follow up at the time of the inquiry
Include a none/NA/no opinion option
When using a range, make sure it is appropriate and non-overlapping
Provide clear instructions on completion
Balance white space and compacting questionnaire

38. On line questionnaires
or internet
Change checkboxes into dropdowns, etc
Take advantage of the technology – check input
Ensure its as accessible as paper (browser and client compatibility)
Ensure confidentiality – how is this different from paper?

39. Free Web Survey Tools Zoomerang Survey Monkey phpESP
Survey Monkey
phpESP
Open Source surveys using PHP.

40. Analyzing your data Analyze open ended responses similar to interviews
Enter your data into something you can manipulate
Excel
SPSS
Look for trends in the data
Count, average, tabulate
Make charts, etc
Run statistical analysis
Use lo-fi methods (post-its, affinity diagrams, etc)

41. Using interviews and questionnaires later in the design process
With a (semi-) working protoype
I felt I could accomplish all of the tasks I was given easily
I would be willing to use this software/device/kiosk/etc to perform these types of tasks in the future
What are some tasks for which you could imagine using this software/device/kiosk/etc?

42. Using interviews and questionnaires later in the design process
With non-working protoypes (scenarios, storyboards, foam, playdough, etc)
I could imagine using a device of this size (form factor, weight, etc) in those situations
Describe how you would carry this device in your daily life.
I would feel comfortable using this software in my home.

43. Evaluation discussion
Someone else should be able to pick up your plan and execute it.
Be as SPECIFIC as possible
What criteria are important?
What tasks EXACTLY?
What data? How will you record?
What questions will you ask?

44. Let’s brainstorm questions
IM voice chat feature