GETTING COMFORTABLE WITH YOUR DATA II One way to turn your data into knowledge, and another way that’s probably better Winter Storm 2010 Stats workshop Dave Kleinschmidt

ANOVA What is it, anyway?

WHAT YOU WANT You’ve designed + run your experiment It sorts observations into groups Is there any difference between groups?

YOUR DATA IS NOISY This could be a big problem for you What if the noise is too big, and drowns out the effect of your groups? More importantly, how can you tell?

STATISTICS TO THE RESCUE Statistical models quantify noise ANOVA is one kind of model Mixed-effects models (MEMs) are another

ANOVA ANalysis Of VAriance Tells whether group means are identical (tests a null hypothesis) Compare variance between groups (good) with variance within groups (bad—noise)

ANOVA Figure from PDQ Statistics, Norman and Streiner

ANOVA If differences between groups outweigh noise within groups, then you can safely reject the null hypothesis (which is that your experiment did nothing)

ANOVA—ONE LAST NOTE ANOVAs come in different flavors: One-way ANOVA tests one grouping Factorial ANOVA tests multiple crossed groupings Repeated-measures ANOVA tests a design where each subject is exposed to each condition (a within-subjects design)

SO WHAT’S THE PROBLEM? ANOVA’s considered the gold-standard Especially for factorial designs However, ANOVA makes assumptions: Data is perfectly balanced Each group has identical variance No systematic variability between subjects or items

MIXED-EFFECTS MODELS TO THE RESCUE! MEMs can represent nearly any sort of variability between subjects/items. Balance these differences with the need to draw general conclusions about the average character of the whole population

MIXED-EFFECTS MODELS TO THE RESCUE! Do other nice things, too Far more robust to missing data Can model nearly any data distribution (not just normal, like ANOVA)

WHAT IS A MEM? Combines fixed and random effects: Fixed effects are deterministic and common to all subjects/itmes Random effects vary from subject-to- subject/item-to-item `

WHAT IS A MEM? Fixed effects describe how the experimental manipulations affect the observations Think of it as the slope of a line: data ij = fixed * x ij (x ij is the condition that data ij comes from) `

WHAT IS A MEM? Of course, we have to add noise. If the noise of each subject/item combination is independent, than we just get data ij = fixed * x ij + noise ij Where all of the noise ij s are independent and normally distributed (with mean zero) (this is the essence of an ANOVA) `

WHAT IS A MEM? What if some subjects are just faster/better than others? Then we just add another noise term by subjects: y ij = fixed * x ij + noise 0j + noise ij Note that this changes the intercept for the line for each subject, but leaves the slope the same for each `

WHAT IS A MEM? In the same way, we can let the slope of the line vary a little by subject, too. This is equivalent to saying that we believe the experimental manipulation affects some subjects more than others. `

SO WHY DOESN’T EVERYONE USE MEMs? Soon, everyone will (probably). No pencil-and-paper solution, unlike ANOVA (but software is widely available now) ANOVA is the established standard (but more and more are using MEMs)

LET’S TRY SOME