1. CS 594: Empirical Methods in HCC Experimental Research in HCI (Part 1)
Dr. Debaleena Chattopadhyay
Department of Computer Science
debaleena.com
hci.cs.uic.edu

2. Agenda Overview Sampling Significance Testing Study Design
Parametric Statistics
Correlation
Regression
T-test
ANOVA
Multilevel Linear Modeling

3. Overview
Experimental research is used in HCI to answer questions of causality.
To show how the manipulation of one variable of interest has a direct causal influence on another variable of interest
Exp. Research in HCI builds upon the tradition of psychology, sociology, cognitive science, information science, and broadly social science.
Exp. Research in HCI can be theoretically driven or engineering driven.

4. Advantages and Limitations of Exp. Research
Internal validity
Demonstrate a strong causal connection
Provides a systematic process to test theoretical propositions and advance theory
Disadvantages
Requires well-defined, testable hypotheses, and a small set of well-controlled variables.
Risk of low external and ecological validity
A poorly executed experiment may have the veneer of “scientific validity” because of the methodological rigor, but ultimately provides little more than well-measured noise.
External Validity. External validity refers to whether the study results can be generalized to the population that you are studying across setting and time.
Ecological Validity. Ecological validity tells us whether or not the findings can be generalized to real-world settings.

5. Sampling

6. Sampling (cont….) Non-probability sampling Probability sampling
Snowball sampling
Quota sampling
Probability sampling
Random selection
To err is human, to randomly err is statistically divine.

7. Hypothesis Formulation
Precise
Meaningful
Testable
Falsifiable
A hypothesis both defi nes the variables involved and the relationship
between them , and can take many forms: A causes B; A is larger, faster, or
more enjoyable than B; etc.

8. Study Design Independent and Dependent variable Operational definition
Manipulation check and check for operational confounds
Reliability and Validity of the dependent variable
Rules for quantifying
Scope and boundaries of what is to be measured
Face validity, concurrent validity, predictive validity
Use standardized measures whenever you can
Consider sensitivity and practicality

9. Significance Testing
The first step in null hypothesis significance testing is to formulate the original research hypothesis as a null hypothesis and an alternative hypothesis.
The null hypothesis (H0 ) is set up as a falsifiable statement that predicts no difference between experimental conditions.
The alternative hypothesis (often written as HA or H1) captures departures from the null hypothesis.

10. Building a model out of data

11. Example model
The standard deviation of sample means is known as the standard error of the mean (SE).
sampling variation: that is, samples will vary because they contain different
members of the population;
A sampling distribution is simply the frequency distribution of
sample means5 from the same population.

12. Parametric data

13. Skewness and Kurtosis Positively skewed Negatively skewed Platykurtic
Leptokurtic
Kurtosis cut-off 3

14. Significance Testing (cont…)
Type 1 error = Pr(reject H 0 | H 0 true). (also significance level or alpha)
p value describes the probability of obtaining the observed data, or more extreme data, if the null hypothesis were true. Pr(observed data| H 0 true)
p < .05 means the chances of a Type I error occurring are less than 5 %.

15. Type 1 and Type 2 error

16. Type 1 and Type 2 error (cont…)

17. Type 1 and Type 2 error (cont…)

18. Study Design (cont…) Randomized Experiments
Between Subject Experiments
Within Subject Experiments
Mixed or Factorial Designs
How to balance to avoid order effect?
Complete counterbalancing; >= n! participants needed
Use Latin Square Designs or Balanced Latin Square Designs

19. Study Design (cont…)

20. One-tailed vs. Two-tailed
Assumption: DV follows a normal distribution
A statistical
model that tests a directional hypothesis is called a one-tailed test, whereas one testing a
non-directional hypothesis is known as a two-tailed test.

21. Latin Square Balancing
better than standard Latin square designs are balanced Latin square
designs where each condition precedes and follows each other condition equally
often. This can help to minimize sequential effects

22. Quasi-Experimental Designs in HCI
Non-equivalent group designs
Pre-test/post-test design
Interrupted Time-Series Design
Infers the effects of an independent variable by comparing multiple measures obtained before and after an intervention takes place.
major threats to internal validity

23. Writing up Experimental Research
Answer “Why should anyone care?”
Use APA conventions—mostly accepted
Results should contain the study design and statistical tests
Discussion should contain how the results address the HCI research question operationalized using the metrics and tested.
Report effect size.
Pre-register large experiments with OSF
Open Science Framework

24. Assumptions of parametric data
Normally distributed data
Homogeneity of variance
Interval data
Independence
that the behaviour of one participant does not influence the behaviour of another.
the variances should be the
same throughout the data.

25. Statistical Power and Effect Sizes
Effect sizes are useful because they provide an objective measure of the importance of an effect.
Effect size depends on:
Sample size
Alpha
Statistical power of the test used
The power of a test is the probability that a given test will find an effect assuming that one exists in the population. If β, the probability of a Type II error, power = 1- β.
G*Power

26. Parametric tests
Correlation
Regression
T-test
ANOVA
GLM

27. Upcoming: Proposal due Sep 24, 11:59pm
Start working on your annotated bibliography