CS 594: Empirical Methods in HCC Experimental Research in HCI (Part 1) Dr. Debaleena Chattopadhyay Department of Computer Science debchatt@uic.edu debaleena.com hci.cs.uic.edu

Agenda Overview Sampling Significance Testing Study Design Parametric Statistics Correlation Regression T-test ANOVA Multilevel Linear Modeling

Overview Experimental research is used in HCI to answer questions of causality. To show how the manipulation of one variable of interest has a direct causal influence on another variable of interest Exp. Research in HCI builds upon the tradition of psychology, sociology, cognitive science, information science, and broadly social science. Exp. Research in HCI can be theoretically driven or engineering driven.

Advantages and Limitations of Exp. Research Internal validity Demonstrate a strong causal connection Provides a systematic process to test theoretical propositions and advance theory Disadvantages Requires well-defined, testable hypotheses, and a small set of well-controlled variables. Risk of low external and ecological validity A poorly executed experiment may have the veneer of “scientific validity” because of the methodological rigor, but ultimately provides little more than well-measured noise. External Validity. External validity refers to whether the study results can be generalized to the population that you are studying across setting and time. Ecological Validity. Ecological validity tells us whether or not the findings can be generalized to real-world settings.

Sampling

Sampling (cont….) Non-probability sampling Probability sampling Snowball sampling Quota sampling Probability sampling Random selection To err is human, to randomly err is statistically divine.

Hypothesis Formulation Precise Meaningful Testable Falsifiable A hypothesis both defi nes the variables involved and the relationship between them , and can take many forms: A causes B; A is larger, faster, or more enjoyable than B; etc.

Study Design Independent and Dependent variable Operational definition Manipulation check and check for operational confounds Reliability and Validity of the dependent variable Rules for quantifying Scope and boundaries of what is to be measured Face validity, concurrent validity, predictive validity Use standardized measures whenever you can Consider sensitivity and practicality

Significance Testing The first step in null hypothesis significance testing is to formulate the original research hypothesis as a null hypothesis and an alternative hypothesis. The null hypothesis (H0 ) is set up as a falsifiable statement that predicts no difference between experimental conditions. The alternative hypothesis (often written as HA or H1) captures departures from the null hypothesis.

Building a model out of data

Example model The standard deviation of sample means is known as the standard error of the mean (SE). sampling variation: that is, samples will vary because they contain different members of the population; A sampling distribution is simply the frequency distribution of sample means5 from the same population.

Parametric data

Skewness and Kurtosis Positively skewed Negatively skewed Platykurtic Leptokurtic Kurtosis cut-off 3

Significance Testing (cont…) Type 1 error = Pr(reject H 0 | H 0 true). (also significance level or alpha) p value describes the probability of obtaining the observed data, or more extreme data, if the null hypothesis were true. Pr(observed data| H 0 true) p < .05 means the chances of a Type I error occurring are less than 5 %.

Type 1 and Type 2 error

Type 1 and Type 2 error (cont…)

Type 1 and Type 2 error (cont…)

Study Design (cont…) Randomized Experiments Between Subject Experiments Within Subject Experiments Mixed or Factorial Designs How to balance to avoid order effect? Complete counterbalancing; >= n! participants needed Use Latin Square Designs or Balanced Latin Square Designs

Study Design (cont…)

One-tailed vs. Two-tailed Assumption: DV follows a normal distribution A statistical model that tests a directional hypothesis is called a one-tailed test, whereas one testing a non-directional hypothesis is known as a two-tailed test.

Latin Square Balancing better than standard Latin square designs are balanced Latin square designs where each condition precedes and follows each other condition equally often. This can help to minimize sequential effects

Quasi-Experimental Designs in HCI Non-equivalent group designs Pre-test/post-test design Interrupted Time-Series Design Infers the effects of an independent variable by comparing multiple measures obtained before and after an intervention takes place. major threats to internal validity

Writing up Experimental Research Answer “Why should anyone care?” Use APA conventions—mostly accepted Results should contain the study design and statistical tests Discussion should contain how the results address the HCI research question operationalized using the metrics and tested. Report effect size. Pre-register large experiments with OSF Open Science Framework https://osf.io/

Assumptions of parametric data Normally distributed data Homogeneity of variance Interval data Independence that the behaviour of one participant does not influence the behaviour of another. the variances should be the same throughout the data.

Statistical Power and Effect Sizes Effect sizes are useful because they provide an objective measure of the importance of an effect. Effect size depends on: Sample size Alpha Statistical power of the test used The power of a test is the probability that a given test will find an effect assuming that one exists in the population. If β, the probability of a Type II error, power = 1- β. G*Power

Parametric tests Correlation Regression T-test ANOVA GLM

Upcoming: Proposal due Sep 24, 11:59pm Start working on your annotated bibliography