Conducting a User Study Human-Computer Interaction

Overview What is a study? What is a study? Empirically testing a hypothesis Empirically testing a hypothesis Evaluate interfaces Evaluate interfaces Why run a study? Why run a study? Determine ‘truth’ Determine ‘truth’ Evaluate if a statement is true Evaluate if a statement is true

Example Overview Ex. The heavier a person weighs, the higher their blood pressure Ex. The heavier a person weighs, the higher their blood pressure Many ways to do this: Many ways to do this: Look at data from a doctor’s office Look at data from a doctor’s office Descriptive design: What’s the pros and cons? Descriptive design: What’s the pros and cons? Get a group of people to get weighed and measure their BP Get a group of people to get weighed and measure their BP Analytic design: What’s the pros and cons? Analytic design: What’s the pros and cons? Ideally? Ideally? Ideal solution: have everyone in the world get weighed and BP Ideal solution: have everyone in the world get weighed and BP Participants are a sample of the population Participants are a sample of the population You should immediately question this! You should immediately question this! Restrict population Restrict population

Study Components Design Design Hypothesis Hypothesis Population Population Task Task Metrics Metrics Procedure Procedure Data Analysis Data Analysis Conclusions Conclusions Confounds/Biases Confounds/Biases

Study Design How are we going to evaluate the interface? How are we going to evaluate the interface? Hypothesis Hypothesis What statement do you want to evaluate? What statement do you want to evaluate? Population Population Who? Who? Metrics Metrics How will you measure? How will you measure?

Hypothesis Statement that you want to evaluate Statement that you want to evaluate Ex. A mouse is faster than a keyboard for numeric entry Ex. A mouse is faster than a keyboard for numeric entry Create a hypothesis Create a hypothesis Ex. Participants using a keyboard to enter a string of numbers will take less time than participants using a mouse. Ex. Participants using a keyboard to enter a string of numbers will take less time than participants using a mouse. Identify Independent and Dependent Variables Identify Independent and Dependent Variables Independent Variable – the variable that is being manipulated by the experimenter (interaction method) Independent Variable – the variable that is being manipulated by the experimenter (interaction method) Dependent Variable – the variable that is caused by the independent variable. (time) Dependent Variable – the variable that is caused by the independent variable. (time)

Hypothesis Testing Hypothesis: Hypothesis: People who use a mouse and keyboard will be faster to fill out a form than keyboard alone. People who use a mouse and keyboard will be faster to fill out a form than keyboard alone. US Court system: Innocent until proven guilty US Court system: Innocent until proven guilty NULL Hypothesis: Assume people who use a mouse and keyboard will fill out a form than keyboard alone in the same amount of time NULL Hypothesis: Assume people who use a mouse and keyboard will fill out a form than keyboard alone in the same amount of time Your job to prove that the NULL hypothesis isn’t true! Your job to prove that the NULL hypothesis isn’t true! Alternate Hypothesis 1: People who use a mouse and keyboard will fill out a form either faster or slower than keyboard alone. Alternate Hypothesis 1: People who use a mouse and keyboard will fill out a form either faster or slower than keyboard alone. Alternate Hypothesis 2: People who use a mouse and keyboard will fill out a form faster than keyboard alone. Alternate Hypothesis 2: People who use a mouse and keyboard will fill out a form faster than keyboard alone.

Population The people going through your study The people going through your study Anonymity Anonymity Type - Two general approaches Type - Two general approaches Have lots of people from the general public Have lots of people from the general public Results are generalizable Results are generalizable Logistically difficult Logistically difficult People will always surprise you with their variance People will always surprise you with their variance Select a niche population Select a niche population Results more constrained Results more constrained Lower variance Lower variance Logistically easier Logistically easier Number Number The more, the better The more, the better How many is enough? How many is enough? Logistics Logistics Recruiting (n>20 is pretty good) Recruiting (n>20 is pretty good)

Two Group Design Design Study Design Study Groups of participants are called conditions Groups of participants are called conditions How many participants? How many participants? Do the groups need the same # of participants? Do the groups need the same # of participants? Task Task What is the task? What is the task? What are considerations for task? What are considerations for task?

Design External validity – do your results mean anything? External validity – do your results mean anything? Results should be similar to other similar studies Results should be similar to other similar studies Use accepted questionnaires, methods Use accepted questionnaires, methods Power – how much meaning do your results have? Power – how much meaning do your results have? The more people the more you can say that the participants are a sample of the population The more people the more you can say that the participants are a sample of the population Pilot your study Pilot your study Generalization – how much do your results apply to the true state of things Generalization – how much do your results apply to the true state of things

Design People who use a mouse and keyboard will be faster to fill out a form than keyboard alone. People who use a mouse and keyboard will be faster to fill out a form than keyboard alone. Let’s create a study design Let’s create a study design Hypothesis Hypothesis Population Population Procedure Procedure Two types: Two types: Between Subjects Between Subjects Within Subjects Within Subjects

Procedure Formally have all participants sign up for a time slot (if individual testing is needed) Formally have all participants sign up for a time slot (if individual testing is needed) Informed Consent (let’s look at one) Informed Consent (let’s look at one) Execute study Execute study Questionnaires/Debriefing (let’s look at one) Questionnaires/Debriefing (let’s look at one)

IRB Let’s look at a completed one Let’s look at a completed one You MUST turn one in before you complete a study to the TA You MUST turn one in before you complete a study to the TA Must have OKed before running study Must have OKed before running study

Biases Hypothesis Guessing Hypothesis Guessing Participants guess what you are trying hypothesis Participants guess what you are trying hypothesis Learning Bias Learning Bias User’s get better as they become more familiar with the task User’s get better as they become more familiar with the task Experimenter Bias Experimenter Bias Subconscious bias of data and evaluation to find what you want to find Subconscious bias of data and evaluation to find what you want to find Systematic Bias Systematic Bias Bias resulting from a flaw integral to the system Bias resulting from a flaw integral to the system E.g. An incorrectly calibrated thermostat E.g. An incorrectly calibrated thermostat List of biases List of biases

Thought Experiment You are creating a new interface for Windows. You are creating a new interface for Windows. You are having your friends test your interface, what are their biases? You are having your friends test your interface, what are their biases? You are having your family test your interface, what are their biases? You are having your family test your interface, what are their biases? You are going to go through the Gainesville phonebook and call people to test your interface, what are their biases? You are going to go through the Gainesville phonebook and call people to test your interface, what are their biases?

Confounds Confounding factors – factors that affect outcomes, but are not related to the study Confounding factors – factors that affect outcomes, but are not related to the study Population confounds Population confounds Who you get? Who you get? How you get them? How you get them? How you reimburse them? How you reimburse them? How do you know groups are equivalent? How do you know groups are equivalent? Design confounds Design confounds Unequal treatment of conditions Unequal treatment of conditions Learning Learning Time spent Time spent

Metrics What you are measuring What you are measuring Types of metrics Types of metrics Objective Objective Time to complete task Time to complete task Errors Errors Ordinal/Continuous Ordinal/Continuous Subjective Subjective Satisfaction Satisfaction Pros/Cons of each type? Pros/Cons of each type?

Analysis Most of what we do involves: Most of what we do involves: Normal Distributed Results Normal Distributed Results Independent Testing Independent Testing Homogenous Population Homogenous Population Recall, we are testing the hypothesis by trying to prove the NULL hypothesis false Recall, we are testing the hypothesis by trying to prove the NULL hypothesis false

Raw Data Keyboard times Keyboard times What does mean mean? What does mean mean? What does variance and standard deviation mean? What does variance and standard deviation mean? E.g. 3.4, 4.4, 5.2, 4.8, 10.1, 1.1, 2.2 E.g. 3.4, 4.4, 5.2, 4.8, 10.1, 1.1, 2.2 Mean = 4.46 Mean = 4.46 Variance = 7.14 (Excel’s VARP) Variance = 7.14 (Excel’s VARP) Standard deviation = 2.67 (sqrt variance) Standard deviation = 2.67 (sqrt variance) What do the different statistical data tell us? What do the different statistical data tell us? User study.xlsx User study.xlsx

What does Raw Data Mean?

Roll of Chance How do we know how much is the ‘truth’ and how much is ‘chance’? How do we know how much is the ‘truth’ and how much is ‘chance’? How much confidence do we have in our answer? How much confidence do we have in our answer?

Hypothesis We assumed the means are “equal” We assumed the means are “equal” But are they? But are they? Or is the difference due to chance? Or is the difference due to chance? Ex. A μ 0 = 4, μ 1 = 4.1 Ex. A μ 0 = 4, μ 1 = 4.1 Ex. B μ 0 = 4, μ 1 = 6 Ex. B μ 0 = 4, μ 1 = 6

T - test T – test – statistical test used to determine whether two observed means are statistically different T – test – statistical test used to determine whether two observed means are statistically different

T-test Distributions Distributions

T – test (rule of thumb) Good values of t > 1.96 (rule of thumb) Good values of t > 1.96 Look at what contributes to t Look at what contributes to t htm htm

F statistic, p values F statistic – assesses the extent to which the means of the experimental conditions differ more than would be expected by chance F statistic – assesses the extent to which the means of the experimental conditions differ more than would be expected by chance t is related to F statistic t is related to F statistic Look up a table, get the p value. Compare to α Look up a table, get the p value. Compare to α α value – probability of making a Type I error (rejecting null hypothesis when really true) α value – probability of making a Type I error (rejecting null hypothesis when really true) p value – statistical likelihood of an observed pattern of data, calculated on the basis of the sampling distribution of the statistic. (% chance it was due to chance) p value – statistical likelihood of an observed pattern of data, calculated on the basis of the sampling distribution of the statistic. (% chance it was due to chance)

T and alpha values

Small PatternLarge Pattern t – test with unequal variance p – value t – test with unequal variance p - value PVE – RSE vs. VFHE – RSE ** *** PVE – RSE vs. HE – RSE ** * VFHE – RSE vs. HE – RSE

Significance What does it mean to be significant? What does it mean to be significant? You have some confidence it was not due to chance. You have some confidence it was not due to chance. But difference between statistical significance and meaningful significance But difference between statistical significance and meaningful significance Always know: Always know: samples (n) samples (n) p value p value variance/standard deviation variance/standard deviation means means