1. Human-Computer Interaction: User Study Examples
CSCE 315 – Programming Studio, Fall 2017
Project 3, Lecture 3

2. Review: The User Study Questions to Address
How to get participants?
Where to gather data?
What data to gather?
How to gather that data?
How to analyze that data?

3. How to Gather Data? Data Gathering Processes
Interview
Pre-questionnaire
Interactive System Use
Observation!
Recording: photos, audio, video
Logging
Answers found/products produced
Think-aloud
Post-questionnaire
Some material from Kerne’s slides

4. First Click Testing
Determines what a user would click first on the interface
Use to determine how well an interface lets someone navigate through it
Why important?
Clicking the first right link leads to success 87% of the time (compared to only 46% on a wrong first click)
Give the user the task (e.g., set up auto bill pay from chase.com)
Track each click (could be manual or automated)
Record time taken to make click
Can gather subjective information on how easy the user was able to do the task
Usability.gov

5. System Usability Scale
5-point Likert scale (strongly disagree to strongly agree)
10 questions
5 are “positive”,5 are “negative”
Given to users after they’ve used the system, before any debriefing
From 1986, has become an industry standard
Each question gives score 0 to 4. For the negative questions, reverse order.
Sum them up, multiply by 2.5 and you have a scale from 0 to 100, with 100 being best
A “quick-and-dirty” approach to usability testing
Not ideal, but widely used/accepted, so easier to compare
Just gives a score on usability – no diagnosis for how to improve
That’s why, we wont use it in out Project 3 to improve UX

6. System Usability Scale Questions
I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system.
I found the various functions in this system were well integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly.
I found the system very cumbersome to use.
I felt very confident using the system.
I needed to learn a lot of things before I could get going with this system.

7. Test by Experts Heuristic Evaluation Expert Reviews
Usability Experts review your UI and compare it against accepted usability principles
Nielsen’s Heuristics are most commonly used set of heuristics
Pros: Relatively inexpensive, can be applied early in the process
Cons: Requires knowledge and experience that are hard to get
Expert Reviews
A less formal version of Heuristic Evaluation
Requires the same level of expertise, but a different view

8. Interview Contextual Interview User interview
Observer watches/listens as user works in user’s environment instead of the lab
Collects observations, typically does not include measurements
Observation is followed/interspersed with interview
You find out such things as: Task duration, Internet speed at site, preference between keyboard and mouse usage etc.
User interview
Interview is more in-depth discussion (30-60 minutes)
Often record interview for later review
Not directly in the context of using the product

9. Interviews
Unstructured - are not directed by a script. Rich but not replicable.
Structured - are tightly scripted, often like a questionnaire. Replicable but may lack richness.
Semi-structured - guided by a script but interesting issues can be explored in more depth. Can provide a good balance between richness and replicability.
Focus groups – a group interview

10. Interview questions Two types: Closed questions are easier to analyze
‘closed questions’ have a predetermined answer format, e.g.. ‘yes’ or ‘no’
‘open questions’ do not have a predetermined format
Closed questions are easier to analyze
Avoid:
Long questions
Compound sentences - split them into two
Jargon and language that the interviewee may not understand
Leading questions that make assumptions e.g.. why do you like …?
Unconscious biases e.g.. gender stereotypes

11. Focus Group
More efficient than one-on-one interview in terms of getting qualitative data from many people
Less efficient, but allows deeper exploration than a survey
Capture information (recording or notes)
Moderator has to be able to progress through questions and handle group dynamics
Poor moderation can lead to people not being forthright, getting into group-think, getting drowned out

12. Online surveys Low cost Broad audience Identify early on:
The purpose of the survey
Where you will get respondents
Software to use
Who will analyze data, and now
Questions should be created to measure or get feedback on what you intend
Can mix open-ended and closed questions
Allows mix of quantitative and qualitative evaluation

13. Task Analysis Early stage user study Find how users go about some task
Observe users in action to determine goals/scope of a project
Find how users go about some task
What their goals are, what they try to achieve
What they actually do to achieve the goals
What experiences (personal, social, cultural) they bring to the tasks
How users are influenced by physical environment
How users’ previous knowledge and experience influence:
Their thought process
Their workflow

14. Eye Tracking
Use an eye tracker to determine where people are looking on a screen during a task
Can measure several items:
Where they are looking
How long they are looking
Focus/order of examination
Where they don’t look
Can infer where people are reading/studying vs. glancing
Focus does not mean comprehension

15. Eye Tracking (2)
Heat Maps
Saccade Pathways

16. References Some slide material taken from information at:
tools/methods/index.html
Some qualitative slides taken from Andruid Kerne’s slides:
Noted in footers
Material is copyrighted
Can be used and cited freely for non-commercial education and scholarship without profit provided that credit is given to the author and this notice is included
Some topics taken from:
Interaction Design, 4th edition, by Rogers, Sharp, and Preece, Wiley, 2014.
See chapters 7-8 in particular