1. … or, as Uncle Ben would say

2. Mosteller & Tukey (1977). Data Analysis and Regression.

3. “Encouraging linguists to use linear mixed-effects models is like giving shotguns to toddlers.”
Gerry Altmann
(see Barr et al., 2013)

4. “A world of subjectivity”
Sarah Depaoli
“IF YOU BEAT THE DATA, AT SOME TIME IT WILL SPEAK”
data loss
- illustrate how proportions loose data by 2/4 … and 30/60 ….etc.

5. “A world of subjectivity”
Sarah Depaoli
“… and then you publish and get tenure.”
data loss
- illustrate how proportions loose data by 2/4 … and 30/60 ….etc.

6. distinguish between test and control variables
LMM
response ~ intercept slope * fixed effect error
distinguish between test and control variables
that’s the “model” part of it… we model the world as falling into two categories… things that we do understand, and things that we don’t understand

7. Test vs. Control Variable Example
pitch ~ politen * gender
test variable
control variable
pitch ~ gender
Both test and control variables are always fixed effects (because their influence is systematic)
Null Model

8. Test vs. Control Variable Example
vowdur ~ VowelType * Repetition
test variable
control variable
vowdur ~ Repetition
I’m testing whether polite speech has lower pitch (test variable) … but I’m not really interested in gender. I nevertheless have to control for gender because that co-determines pitch (control variable)
Null Model

9. Test vs. Control Variable Example
Response ~
BLACK
BOX
Critical Effect
Control 1
Control 2
Random Effects
I’m testing whether polite speech has lower pitch (test variable) … but I’m not really interested in gender. I nevertheless have to control for gender because that co-determines pitch (control variable)

10. Test vs. Control Variable Example
Response ~
BLACK
BOX
Critical Effect
Control 3
Control 2
Random Effects
I’m testing whether polite speech has lower pitch (test variable) … but I’m not really interested in gender. I nevertheless have to control for gender because that co-determines pitch (control variable)

11. Test vs. Control Variable Example
Response ~
Critical Effect
Control 3
Control 2
Random Effects
I’m testing whether polite speech has lower pitch (test variable) … but I’m not really interested in gender. I nevertheless have to control for gender because that co-determines pitch (control variable)

12. Trade-off #1 Model Simplicity Model Fit
How much do you want your models to be simple? How much do you want them to capture the real world?
GENERALIZABILITY; The number of data points should vastly exceed the number of fixed effects levels you estimate.

13. Trade-off #2 Data- driven Theory- driven “Exploratory End”
“Confirmatory End”
Data- driven
Theory- driven
Harald Baayen

14. Roger Mundry (and many others)
Trade-off #2
“Exploratory End”
“Confirmatory End”
Data- driven
Theory- driven
Roger Mundry (and many others)

15. Trade-off #2
Big Question:
How much do you allow the data to suggest new hypotheses? How much do you depend on a priori theory?

16. Approach 1:
more data-driven
Approach 2:
more theory-driven
e.g., test whether random slopes are needed (maybe not advisable)
e.g., test whether interaction for sth. is necessary or not (“o.k.” if the interaction is a control variable)
e.g., test whether sth. requires a non-linear or a linear effect (maybe o.k.)

17. Approach 1: more data-driven Approach 2: more theory-driven
THINGS TO WORRY ABOUT:
Taken to the extreme, this approach has a very high likelihood of finding any significant result
The model selection process is less transparent to outsiders (or, you have to write a LONG LONG stats section)
(in Baayen’s defense: he doesn’t go that far)

18. Approach 1:
more data-driven
Approach 2:
more theory-driven
ADVANTAGES:
You don’t miss important things in your data
Your model might thus be more accurate and “more true to the data”

19. Approach 1:
more data-driven
Approach 2:
more theory-driven
You formulate your model before you look at the data
The components of your model are guided by:
 Theory + Published Results
 General world-knowledge
 Research experience
Taken to the extreme, you can’t even make a plot before you formulate your model

20. Approach 1:
more data-driven
Approach 2:
more theory-driven
ADVANTAGES:
It forces you to think a lot
It’s fun!
It gives you a lot of responsibility, as a scientist
Your estimates are going to be more conservative

21. Think about model (before you conduct your experiment)
Approach 1:
more data-driven
Approach 2:
more theory-driven
Think about model (before you conduct your experiment)
Test whether control variables interact with test variable, or whether they are needed
Build model, evaluate the model’s assumptions
Build model that better fits the assumptions

22. Dialogue with your model
You need to know that there’s multiple responses per subject and item!
Token Researcher ;-)
People might speed up or slow down throughout an experiment.
You need to know that each item was repeated two times!

23. Keep in mind: You have to resolve non-independencies
Your random effects structure should be maximal with respect to your experimental design

24. Protecting your research from yourself:
Whatever you do,
your model decision should not be based on the significance of your effect

25. (JEPS Bulletin)

26. CONFIRM FIRST EXPLORE SECOND Important principle McArdle (2011: 335)
John McArdle
McArdle (2011: 335)
McArdle, J. J. (2011). Some ethical issues in factor analysis. In A.T. Panter & S. K. Sterba (Eds.), Handbook of Ethics in Quantitative Methodology (pp ). New York, NY: Routledge.

27. The write-up
The success of phonetics as a field depends on the transparency of reporting analyses

28. Important principle
BE HONEST
NOT PURE
John McArdle

29. Cool guidelines
United Nations Economic Commission for Europe (2009a). Making Data Meaningful Part 1: A guide to writing stories about numbers. New York and Geneva: United Nations. United Nations Economic Commission for Europe (2009b). Making Data Meaningful Part 2: A guide to presenting statistics. New York and Geneva: United Nations.
Write a story about the results, not a story about your statistical discovery process
However, do specify your model selection choices

30. “We tested a linear mixed effects model with subjects and items as random effects.”
without mentioning the random effects structure, you don’t know what’s going on

31. = Reproducible Research
The write-up should reflect (as adequately as possible) the details of your model… and your model selection procedure
precisely because there are so many researcher degrees of freedom… it is important for your audience to know what you did… and to potentially critique you
the purpose of reporting your analysis is replicability
= Reproducible Research

32. Rule of thumb:
“One needs to provide sufficient information for the reader to be able to recreate the analyses.”
Barr et al. (2013)
,,, therefore … it is advisable to keep your model lean
Ask yourself: With the information that I provided, could I, myself, replicate the analysis?

33. How to write up (1) "Phenomenon-oriented write-up"
(2) Appendix / Supplementary Materials
(even though I think they should be)

34. Example #1
“We used generalized linear mixed models to test the effect of Gender and Politeness on pitch. Subjects and items were random effects (random intercepts) (Baayen, Davidson & Bates, 2008), with random slopes for subjects and items for the effect Politeness (Barr, Levy, Scheepers & Tily, 2013). We also included a Gender * Politeness interaction into the model and if this interaction was not significant, only included the main effects. /// Q-Q plots and plots of residuals against fitted values revealed no obvious deviations from normality and homoskedasticity. We report p-values based on Likelihood Ratio Tests of the model with the main fixed effect in question (Politeness) against the model without the main fixed effect (null model, including Gender).”

35. Example #2: "Phenomenon-oriented"
“We used generalized linear mixed models to test the association between voice onset time and pitch. The fixed effects quantify the effect of VOT on politeness, as well as the effect of place of articulation, vowel type and gender on politeness. The random effects quantify the by-subject and by-item variability in pitch (random intercepts), as well as the variation of the effect of VOT on pitch for subjects and items (random slopes).”

36. Mentioning assumptions
“Visual inspection of residual plots revealed no obvious deviation from normality and homoskedasticity of errors.”
“We checked plots of residuals against fitted values and found no indication that the normality and homoskedasticity assumption were violated.”
“… indicated a problem with … We therefore log-transformed the data.”

37. Results
Provide results of likelihood ratio test (i.e., significance etc.)
Provide estimates and standard errors in the metric of the model
For poisson and logistic regression, additionally provide some exemplary back-transformed values (don’t back-transform the standard errors)

38. Likelihood Model Output
Data: mag Models: magmodel.maineffect: linelength ~ condition + city_status + german_side + gender + magmodel.maineffect: trial_order + (1 + condition * city_status | subjects) + magmodel.maineffect: (1 + condition * city_status | items) magmodel: linelength ~ condition * city_status + german_side + gender + magmodel: trial_order + (1 + condition * city_status | subjects) + magmodel: (1 + condition * city_status | items) Df AIC BIC logLik Chisq Chi Df Pr(>Chisq) magmodel.maineffect magmodel < 2.2e-16 *** --- Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

39. Important principle
BE HONEST
NOT PURE
John McArdle

40. Make your scripts orderly and reproducible

41. Reproducibility Make your script online available
Avoid modifying your data manually ... make a script that records your process
red one = in your own self-interest (for later use)

42. That’s it