1. Asking Users and Experts
Changsung Moon
Jaeyoung Lee

2. The Main Aims
Discuss when it is appropriate to use different types of interviews and questionnaires.
Teach you the basics of questionnaire design.
Describe how to do interviews, heuristic evaluation, and walkthroughs.
Describe how to collect, analyze, and present data.
Discuss the strengths and limitations of these techniques.

3. The Types of one-on-one Interviews
Unstructured (or open-ended)
Questions or topics are open-ended, interviewee is free to answer in any manner, and topics do not have to be covered in any particular order.
Ex) Suggest ways of collecting the interview data.
Structured
Most controlled type.
Interview may consist of closed-ended questions and interviewee must choose from the options provided.
Ex) Would you like to receive news from CNN?
Yes No Don’t know
Semi-structured
Combination of structured and unstructured
Ex) Would you listen to the news from CNN?
If yes, why? If no, why not?

4. The Types of one-on-one Interviews
Unstructured (or open-ended)
Questions or topics are open-ended, interviewee is free to answer in any manner, and topics do not have to be covered in any particular order.
Ex) Suggest ways of collecting the interview data.
Structured
Most controlled type.
Interview may consist of closed-ended questions and interviewee must choose from the options provided.
Ex) Would you like to receive news from CNN?
Yes No Don’t know
Semi-structured
Combination of structured and unstructured
Ex) Would you listen to the news from CNN?
If yes, why? If no, why not?

5. The Types of one-on-one Interviews
Unstructured (or open-ended)
Questions or topics are open-ended, interviewee is free to answer in any manner, and topics do not have to be covered in any particular order.
Ex) Suggest ways of collecting the interview data.
Structured
Most controlled type.
Interview may consist of closed-ended questions and interviewee must choose from the options provided.
Ex) Would you like to receive news from CNN?
Yes No Don’t know
Semi-structured
Combination of structured and unstructured
Ex) Would you listen to the news from CNN?
If yes, why? If no, why not?

6. The Types of one-on-one Interviews
Unstructured (or open-ended)
Questions or topics are open-ended, interviewee is free to answer in any manner, and topics do not have to be covered in any particular order.
Ex) Suggest ways of collecting the interview data.
Structured
Most controlled type.
Interview may consist of closed-ended questions and interviewee must choose from the options provided.
Ex) Would you like to receive news from CNN?
Yes No Don’t know
Semi-structured
Combination of structured and unstructured
Ex) Would you listen to the news from CNN?
If yes, why? If no, why not?

8. Guidelines for Developing interview questions (Things to avoid)
Long questions
They are difficult to remember.
Compound sentences
Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?”
Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record)
Using jargon
The interviewee may not understand.
Leading questions
Ex) Wrong: “Most of our users prefer the new look and feel of our site over the old one. How do you feel?”
Right: “How do you feel about the visual appearance of this website?”
Unconscious biases
Ex) Wrong: “Safety experts recommend using a travel agent instead of booking your travel online. Do you feel safe using travel websites?”
Right: “Do you feel that booking travel online is more or less confidential than booking through a travel agent?”

9. Guidelines for Developing interview questions (Things to avoid)
Long questions
They are difficult to remember.
Compound sentences
Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?”
Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record)
Using jargon
The interviewee may not understand.
Leading questions
Ex) Wrong: “Most of our users prefer the new look and feel of our site over the old one. How do you feel?”
Right: “How do you feel about the visual appearance of this website?”
Unconscious biases
Ex) Wrong: “Safety experts recommend using a travel agent instead of booking your travel online. Do you feel safe using travel websites?”
Right: “Do you feel that booking travel online is more or less confidential than booking through a travel agent?”

10. Guidelines for Developing interview questions (Things to avoid)
Long questions
They are difficult to remember.
Compound sentences
Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?”
Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record)
Using jargon
The interviewee may not understand.
Leading questions
Ex) Wrong: “Most of our users prefer the new look and feel of our site over the old one. How do you feel?”
Right: “How do you feel about the visual appearance of this website?”
Unconscious biases
Ex) Wrong: “Safety experts recommend using a travel agent instead of booking your travel online. Do you feel safe using travel websites?”
Right: “Do you feel that booking travel online is more or less confidential than booking through a travel agent?”

11. Guidelines for Developing interview questions (Things to avoid)
Long questions
They are difficult to remember.
Compound sentences
Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?”
Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record)
Using jargon
The interviewee may not understand.
Leading questions
Ex) Wrong: “Most of our users prefer the new look and feel of our site over the old one. How do you feel?”
Right: “How do you feel about the visual appearance of this website?”
Unconscious biases
Ex) Wrong: “Safety experts recommend using a travel agent instead of booking your travel online. Do you feel safe using travel websites?”
Right: “Do you feel that booking travel online is more or less confidential than booking through a travel agent?”

12. Guidelines for Developing interview questions (Things to avoid)
Long questions
They are difficult to remember.
Compound sentences
Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?”
Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record)
Using jargon
The interviewee may not understand.
Leading questions
Ex) Wrong: “Most of our users prefer the new look and feel of our site over the old one. How do you feel?”
Right: “How do you feel about the visual appearance of this website?”
Unconscious biases
Ex) Wrong: “Safety experts recommend using a travel agent instead of booking your travel online. Do you feel safe using travel websites?”
Right: “Do you feel that booking travel online is more or less confidential than booking through a travel agent?”

13. Guidelines for Developing interview questions (Things to avoid)
Long questions
They are difficult to remember.
Compound sentences
Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?”
Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record)
Using jargon
The interviewee may not understand.
Leading questions
Ex) Wrong: “Most of our users prefer the new look and feel of our site over the old one. How do you feel?”
Right: “How do you feel about the visual appearance of this website?”
Unconscious biases
Ex) Wrong: “Safety experts recommend using a travel agent instead of booking your travel online. Do you feel safe using travel websites?”
Right: “Do you feel that booking travel online is more or less confidential than booking through a travel agent?”

14. Timeline for an Interview

15. Group Interviews ‘focus group’: one form of group interview
Normally 3 to 10 people are involved
Participants are selected to provide a representative sample of typical users;
they share certain characteristics
Benefit is that it allows diverse or sensitive issues that would otherwise be missed

16. Questionnaires Questions can be closed or open (similar to interview)
Closed questions are easier to analyze
can be administered to large populations
Paper, and the web used for dissemination
Electronic questionnaires
Data goes into a database and is easy to analyze
Online questionnaires
Sampling can be a problem when the size of a population is unknown

17. Advice for Designing a Questionnaire
Make questions clear and specific
When possible, ask closed questions and offer a range of answers
Consider including a “no-opinion” option
General questions should precede specific ones
Avoid complex multiple questions
The range of answers is appropriate and does not overlap
Ex) 15-20, (X) , (O)
The ordering of scales should be intuitive and consistent
Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high agreement
Avoid jargon
Prove clear instructions on how to complete the questionnaire
Keep the questionnaire as compact as possible

18. Advice for Designing a Questionnaire
Make questions clear and specific
When possible, ask closed questions and offer a range of answers
Consider including a “no-opinion” option
General questions should precede specific ones
Avoid complex multiple questions
The range of answers is appropriate and does not overlap
Ex) 15-20, (X) , (O)
The ordering of scales should be intuitive and consistent
Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high agreement
Avoid jargon
Prove clear instructions on how to complete the questionnaire
Keep the questionnaire as compact as possible

19. Advice for Designing a Questionnaire
Make questions clear and specific
When possible, ask closed questions and offer a range of answers
Consider including a “no-opinion” option
General questions should precede specific ones
Avoid complex multiple questions
The range of answers is appropriate and does not overlap
Ex) 15-20, (X) , (O)
The ordering of scales should be intuitive and consistent
Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high agreement
Avoid jargon
Prove clear instructions on how to complete the questionnaire
Keep the questionnaire as compact as possible

20. Advice for Designing a Questionnaire
Make questions clear and specific
When possible, ask closed questions and offer a range of answers
Consider including a “no-opinion” option
General questions should precede specific ones
Avoid complex multiple questions
The range of answers is appropriate and does not overlap
Ex) 15-20, (X) , (O)
The ordering of scales should be intuitive and consistent
Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high agreement
Avoid jargon
Prove clear instructions on how to complete the questionnaire
Keep the questionnaire as compact as possible

21. Advice for Designing a Questionnaire
Make questions clear and specific
When possible, ask closed questions and offer a range of answers
Consider including a “no-opinion” option
General questions should precede specific ones
Avoid complex multiple questions
The range of answers is appropriate and does not overlap
Ex) 15-20, (X) , (O)
The ordering of scales should be intuitive and consistent
Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high agreement
Avoid jargon
Prove clear instructions on how to complete the questionnaire
Keep the questionnaire as compact as possible

22. Advice for Designing a Questionnaire
Make questions clear and specific
When possible, ask closed questions and offer a range of answers
Consider including a “no-opinion” option
General questions should precede specific ones
Avoid complex multiple questions
The range of answers is appropriate and does not overlap
Ex) 15-20, (X) , (O)
The ordering of scales should be intuitive and consistent
Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high agreement
Avoid jargon
Prove clear instructions on how to complete the questionnaire
Keep the questionnaire as compact as possible

23. Question and Response Format
Different types of questions require different types of responses.
Check boxes and Ranges
Ex) under over 65
Likert Scales
Ex) The use of color is excellent: (where 1 represents strongly agree and 5 represents strongly disagree)
Semantic differential scales
Ex) Exciting Boring

24. What to Do with the Data Identify any trends or patterns
Using a spreadsheet like Excel can help in initial analysis
Often simple statistics are needed
Bar charts can be used to display data graphically
More advanced statistics can be used
show whether there is a relationship between question and responses

25. ASKING EXPERTS

26. Why Asking Experts? Sometimes users are not easily accessible.
Involving users is too expensive
Evaluating a interface design takes so long time.
Skillful experts can capture many of the usability problems by themselves.
We can employ just few experts.

27. Two Different Technologies for Asking Experts
Inspections
Heuristic Evaluation
Walkthrough
Cognitive Walkthrough
Pluralistic Walkthrough

28. Heuristic Evaluation

29. Heuristic Evaluation
A discount method for quick, cheap, and easy evaluation of the user interface.
Developed by Jacob Nielsen in the early 1990s based on heuristics
distilled from an empirical analysis of 249 usability problems

30. Heuristic Evaluation -10 Heuristics
Visibility of system status
Match between system and the real world
User control and freedom
Consistency and standards
Help users recognize, diagnose, and recover from errors
Error prevention
Recognition rather than recall
Flexibility and efficiency of use
Aesthetic and minimalist design
Help and documentation
“Are users kept informed about what is going on?”
“Is appropriate feed back provided within reasonable time about a user’s action?”

31. Heuristic Evaluation Revised for current technology
HOMERUN for the Web : These heuristics are more useful for evaluating commercial websites.
High-quality content
Often updated
Minimal download time
Ease of use
Relevant to users’ needs
Unique to the online medium
Netcentric corporate culture
Heuristics still needed for new technologies
mobile devices, wearable interfaces, virtual worlds, etc.

32. How many Experts? Nielsen found that about 5 evaluations found 75% of
the problems
Above that you get more, but at decreasing efficiency

33. Three Stages for Doing Heuristic Evaluation
Briefing session to tell experts what to do
Evaluation period of 1-2 hours in which:
Each expert works separately
Take one pass to get a feel for the product
Take a second pass to focus on specific features
Debriefing session in which experts work together to prioritize problems

34. Advantages and Problems
Best experts have knowledge of application domain & users
Few ethical & practical issues to consider
Can be difficult & expensive to find experts
Biggest problems
important problems may get missed
many trivial problems are often identified

35. Walkthroughs
Cognitive Walkthroughs
Pluralistic Walkthroughs

36. Cognitive Walkthroughs
“Cognitive walkthroughs involve simulating a user’s problem-solving process at each step in the human-computer dialog, checking to see if the user’s goals and memory for actions can be assumed to lead to the next correct action.” (Nielsen and Mack, 1994)

37. Four Steps of Cognitive Walkthroughs
Analysis Tasks
Evaluate each task
Compile the record of critical information
Suggest revised design

38. Steps involved in Cognitive Walkthroughs – First Step
The characteristics of typical users are identified and documented.
Sample tasks are developed that focus on the aspects of the design to be evaluated.
A description or prototype of the interface is also produced, along with a clear sequence of the actions needed for the users to complete the task.

39. Steps involved in Cognitive Walkthroughs – Second Step
Designers and one or more expert evaluators then come together to do the analysis
The evaluators walk through the action sequences for each task, placing it within the context of a typical scenario.

40. Steps involved in Cognitive Walkthroughs – Second Step
While experts are evaluating each task, they try to answer the following questions
Will the correct action be sufficiently evident to the user?
Will the user notice that the correct action is available?
Will the user associate and interpret the response from the action correctly?

41. Steps involved in Cognitive Walkthroughs – Third Step
As the walkthrough is being done, a record of critical information is compiled in which:
The assumptions about what would cause problems and why are recorded.
Notes about side issues and design changes are made.
A summary of the results is compiled.

42. Steps involved in Cognitive Walkthroughs – Fourth Step
The design is then revised to fix the problems presented.

43. Pluralistic Walkthroughs

44. What is Pluralistic Walkthrough?
“Another type of walkthrough in which users, developers and usability experts work together to step through a scenario, discussing usability issues associated with dialog elements involved in the scenario steps”
(Nielsen and Mack, 1994)

45. Steps of Doing Pluralistic Walkthroughs
Scenarios are developed in the form of a series of hard-copy screens
The scenarios are presented to the panel of evaluators.
The panelists discuss the actions that they suggested for that round of the review.
Then the panel moves on to the next round of screens.

46. What is benefit of Pluralistic Walkthrough?
Pluralistic walkthroughs have the advantage of providing a diverse range of skills and perspectives.
The result of pluralistic walkthroughs includes a strong focus on users’ tasks.
The approach also lends itself well to participatory design practices by involving a multidisciplinary team in which users play a key role.

47. Thank you~ Any question?