1. PSY 626: Bayesian Statistics for Psychological Science
2/22/2019
Comparing analyses
Greg Francis
PSY 626: Bayesian Statistics for Psychological Science
Fall 2018
Purdue University
PSY200 Cognitive Psychology

2. Facial feedback
Is your emotional state influenced by your facial muscles?
If I ask you to smile, you may report feeling happier
But this could just be because you guess I want you to report feeling happier
Or because intentional smiling is associated with feeling happier
You can ask people to use smiling muscles without them realizing it

3. Facial feedback Within subjects design (n=21)
Judge “happiness” in a piece of abstract art while
Holding a pen in your teeth (smiling)
Holding a pen in your lips (frowning/pouting)
No pen
11 trials for each condition
Different art on each trial

4. Data
The HappinessRating is a number between 0 (no happiness) to 100 (lots of happiness)
The facial feedback hypothesis suggests that the mean HappinessRating values should be larger when the pen is held in the teeth and lower when the pen is held in the lips
File FacialFeedback.csv contains all the data

5. Models Consider three models # load data file
FFdata<-read.csv(file="FacialFeedback.csv",header=TRUE,stringsAsFactors=TRUE)
# By default, participant numbers are treated as _numbers_. Need to correct that.
FFdata$ParticipantFactor = factor(FFdata$Participant)
# load the brms library
library(brms)
# null model
model1 = brm(HappinessRating ~ 1, data = FFdata)
# without random effect on participant
model2 = brm(HappinessRating ~ 0 + Condition, data = FFdata)
# with random effect on participant (for shrinkage)
model3 = brm(HappinessRating ~ 0 + Condition + (Condition | Participant), data = FFdata)
# compare models
print(model_weights(model1, model2, model3, weight="loo") )
> print(model_weights(model1, model2, model3, weight="loo") )
model model model3
e e e-01

6. Model comparison What does this mean?
If you wanted to predict future data, the random effects model is your best choice among these models
That’s largely because there seem to be differences between participants, and only this model considers them
Maybe we should consider other models

7. Various null models We usually think of there being one null
But oftentimes there are many such models
# Different null model: no effect of condition, but different values across participants
model4 = brm(HappinessRating ~ ParticipantFactor, data = FFdata)
# Random effects null model: no effect of condition, but different values across participants that are related to each other
model5 = brm(HappinessRating ~ (1 | ParticipantFactor), data = FFdata)
# compare models
print(model_weights(model1, model2, model3, model4, model5, weight="loo"))
> model_weights(model1, model2, model3, model4, model5, weight="loo")
model model model model model5
e e e e e-05

8. Evidential support
What does the Facial feedback hypothesis actually predict?
Is your emotional state influenced by your facial muscles?
->Differences in happiness ratings across the conditions
> print(model3)
Family: gaussian
Links: mu = identity; sigma = identity
Formula: HappinessRating ~ 0 + Condition + (Condition | Participant)
Data: FFdata (Number of observations: 693)
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup samples = 4000
Group-Level Effects:
~Participant (Number of levels: 21)
Estimate Est.Error l-95% CI u-95% CI Eff.Sample Rhat
sd(Intercept)
sd(ConditionPenInLips)
sd(ConditionPenInTeeth)
cor(Intercept,ConditionPenInLips)
cor(Intercept,ConditionPenInTeeth)
cor(ConditionPenInLips,ConditionPenInTeeth)
Population-Level Effects:
ConditionNoPen
ConditionPenInLips
ConditionPenInTeeth
Family Specific Parameters:
sigma

9. Evidential support
What does the Facial feedback hypothesis actually predict?
Use of “smiling” facial muscles should lead to higher happiness ratings than “pouting” facial muscles
-> Higher ratings for teeth than for lips conditions
> print(model3)
Family: gaussian
Links: mu = identity; sigma = identity
Formula: HappinessRating ~ 0 + Condition + (Condition | Participant)
Data: FFdata (Number of observations: 693)
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup samples = 4000
Group-Level Effects:
~Participant (Number of levels: 21)
Estimate Est.Error l-95% CI u-95% CI Eff.Sample Rhat
sd(Intercept)
sd(ConditionPenInLips)
sd(ConditionPenInTeeth)
cor(Intercept,ConditionPenInLips)
cor(Intercept,ConditionPenInTeeth)
cor(ConditionPenInLips,ConditionPenInTeeth)
Population-Level Effects:
ConditionNoPen
ConditionPenInLips
ConditionPenInTeeth
Family Specific Parameters:
sigma

10. Evidential support
What does the Facial feedback hypothesis actually predict?
“Smiling” facial muscles leads to more happiness, “pouting” facial muscles leads to less happiness
-> Order of means: teeth > none > lips
> print(model3)
Family: gaussian
Links: mu = identity; sigma = identity
Formula: HappinessRating ~ 0 + Condition + (Condition | Participant)
Data: FFdata (Number of observations: 693)
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup samples = 4000
Group-Level Effects:
~Participant (Number of levels: 21)
Estimate Est.Error l-95% CI u-95% CI Eff.Sample Rhat
sd(Intercept)
sd(ConditionPenInLips)
sd(ConditionPenInTeeth)
cor(Intercept,ConditionPenInLips)
cor(Intercept,ConditionPenInTeeth)
cor(ConditionPenInLips,ConditionPenInTeeth)
Population-Level Effects:
ConditionNoPen
ConditionPenInLips
ConditionPenInTeeth
Family Specific Parameters:
sigma

11. Ordered means
You should build the model that best represents the theoretical claim
Consider a model for ordered effects: teeth > none > lips
First attempt (failure)
More generally, “hard” boundaries are dangerous.
Can lead to model convergence problems
Better to use “soft” boundaries via priors
priors = c(prior(normal(0, 10), class = "b", coef="ConditionPenInLips", ub=0), prior(normal(0, 10), class = "b", coef="ConditionPenInTeeth", lb=0))
> Error: Argument 'coef' may not be specified when using boundaries.

12. Ordered means
You should build the model that best represents the theoretical claim
Consider a model for ordered effects: teeth > none > lips
Intercept (None), bias negative for Lips, bias Positive for Teeth
Versus a model without ordered effects
priors = c(prior(normal(-5, 5), class = "b", coef="ConditionPenInLips"), prior(normal(5, 5), class = "b", coef="ConditionPenInTeeth"))
model6 = brm(HappinessRating ~ 1 + Condition + (1 + Condition | Participant), data = FFdata, prior=priors)
> model_weights(model3, model6, weights="loo")
model3 model6

13. Ordered means
You should build the model that best represents the theoretical claim
We set different priors, so this might not be a fair comparison
Not much difference
priors = c(prior(normal(0, 5), class = "b", coef="ConditionPenInLips"), prior(normal(0, 5), class = "b", coef="ConditionPenInTeeth"))
model7 = brm(HappinessRating ~ 1 + Condition + (1 + Condition | Participant), data = FFdata, prior=priors)
> model_weights(model6, model7, weights="loo")
model6 model7

14. Model comparison To do much more, we need better priors
Which means we need to better understand the precise predictions of the Facial Feedback hypothesis
Perhaps the theory does make more precise predictions
But oftentimes, psychological theories make only vague predictions

15. BayesFactor Package
It is less flexible, but it provides a nice interface for common comparisons
Favors the null
# load data file
FFdata<-read.csv(file="FacialFeedback.csv",header=TRUE,stringsAsFactors=TRUE)
# By default, participant numbers are treated as _numbers_. Need to correct that.
FFdata$ParticipantFactor = factor(FFdata$Participant)
# load the BayesFactor library
library(BayesFactor)
bf = anovaBF(HappinessRating ~ Condition + ParticipantFactor, data = FFdata, whichRandom="ParticipantFactor")
> bf
Bayes factor analysis
[1] Condition + ParticipantFactor : ±0.95%
Against denominator:
HappinessRating ~ ParticipantFactor
---
Bayes factor type: BFlinearModel, JZS

16. BRMS vs BayesFactor We can set up similar comparisons in BRMS
Favors an effect!
# Different null model: no effect of condition, but different values across participants
model4 = brm(HappinessRating ~ ParticipantFactor, data = FFdata, save_all_pars = TRUE)
model5b = brm(HappinessRating ~ Condition + (1 | ParticipantFactor), data = FFdata, save_all_pars = TRUE)
> model_weights(model4, model5b, weights="waic")
model4 model5b

17. Why the differences? Same model structure: Differences:
Condition + random effect of participant
Vs. participants only
Differences:
Priors
BF vs. WAIC/loo
Lets look at details

18. Bayes Factor NoPen=43.719 Lips=45.7805 Teeth=47.3425
Iterations = 1:1000
Thinning interval = 1
Number of chains = 1
Sample size per chain = 1000
1. Empirical mean and standard deviation for each variable,
plus standard error of the mean:
Mean SD Naive SE Time-series SE mu
Condition-NoPen
Condition-PenInLips
Condition-PenInTeeth
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
ParticipantFactor
sig
g_Condition
g_ParticipantFactor
Extract parameter estimates
NoPen=43.719
Lips=
Teeth=
> chains = posterior(bf, iterations = 1000) %
|----|----|----|----|----|----|----|----|----|----|
**************************************************|
> summary(chains)

19. BRMS NoPen=43.65 Lips=45.85 Teeth=47.42 Extract parameter estimates
Family: gaussian
Links: mu = identity; sigma = identity
Formula: HappinessRating ~ Condition + (1 | ParticipantFactor)
Data: FFdata (Number of observations: 693)
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup samples = 4000
Group-Level Effects:
~ParticipantFactor (Number of levels: 21)
Estimate Est.Error l-95% CI u-95% CI Eff.Sample Rhat
sd(Intercept)
Population-Level Effects:
Intercept
ConditionPenInLips
ConditionPenInTeeth
Family Specific Parameters:
sigma
Extract parameter estimates
NoPen=43.65
Lips=45.85
Teeth=47.42
model5b

20. Posteriors BayesFactor
Summarize
plot(chains)

21. Posteriors BRMS
Summarize
plot(model5b)

22. Posterior details Posterior distributions for BF
Have to look at chains format to extract what you want
# Create posterior for Teeth mean
TeethBFPosterior = chains[,1] + chains[,4]
dev.new()
plot(density(TeethBFPosterior))

23. Posterior details Posterior distributions for brms model5b
post<-posterior_samples(model5b)
TeethBRMSPosterior = post$b_Intercept + post$b_ConditionPenInTeeth

24. Together Almost identical! > plot(density(TeethBRMSPosterior))
> lines(density(TeethBFPosterior))

25. Lips

26. None condition

27. Bayes Factor from BRMS BRMS can compute a Bayes Factor from two models
Huge BF in favor of model4 (null model)
BFbrms = bayes_factor(model4, model5b)
> print(BFbrms)
The estimated Bayes factor in favor of x1 over x2 is equal to: e+31

28. BF vs. loo
A Bayes Factor makes sense when you have informative priors that are used to specify a pretty precise model
The default priors used by brms are the opposite of informative priors (by design)
This makes them a poor choice for computing Bayes Factors
Small differences can get blown up in a ratio
Which approach is right?
It depends on what you want to do with your model

29. What do you want/have?
Are you testing well specified models with informative priors? Do you believe the true model is among the ones you testing?
Maybe go for the Bayes Factor
Are you hoping to predict future data and avoid overfitting? Do you not believe that you can identify the true model?
Maybe go for WAIC or loo.
Personally, I think WAIC or loo is more appropriate for most situations in the social sciences

30. Posteriors
Both brms and the BayesFactor library provide fairly reasonable default priors
Both provide posterior distributions
Both produce pretty similar estimates of population parameters
They differ in details, and those details can matter some times
Ease of use
Interpretation

31. Conclusions Figure out what you want to do for your analysis
That includes knowing your audience
Some people will understand BayesFactors (or think they do), and that can help you communicate your findings