1. Sampling and Power

2. Cross-cultural research

3. General Sampling Issues
Thinking of the steps in sampling (from theoretical population to respondents)—what are some biases that can come in at each point?
What is the proximity similarity model? What are issues with that model?
How can you increase external validity?
When do you need a representative sample?

4. Probability Sampling Are random samples really random?
When would you use each?
Simple random sampling
Stratified random
Systematic random
Cluster
Multistage

5. Nonprobability Sampling
What are these?
When should they be used?
Convenience sample
Modal instance
Expert
Quota
Heterogeneity
Snowball

6. Henrich, Heine, & Norenzayan, 2010
Sears, 1986—college students in the laboratory
“According to this view people (a) are quite compliant and their behavior is easily socially influenced, (b) readily change their attitudes and (c) behave inconsistently with them, and (d) do not rest their self- perceptions on introspection. The narrow data base may also contribute to this portrait of human nature's (e) strong emphasis on cognitive processes and to its lack of emphasis on (f) personality dispositions, (g) material self-interest, (h) emotionally based irrationalities, (i) group norms, and (j) stage-specific phenomena.”
Henry, 2008 (still a problem, only covered prejudice)

7. What are WEIRD samples?
Why do we focus so much on WEIRD samples?
What should we report about demographics?

8. Industrialized societies vs. small-scale societies
What kinds of differences exist?
Similarities?
Why?

9. Western vs. non-Western
What are differences?
Similarities?
Why?

10. Americans vs. other Westerners
How are we weird?
Why are we weird?

11. American participants vs. others
What differences are there?
How do college students differ from others?
For what topics are less likely or not to have an effect?
How do we differ from Americans in the past?

12. Overall What problems do WEIRD samples cause?
How much can we generalize our results? When does generalization make sense?
When is it okay to use WEIRD samples?
How does it affect what we study?
How does it affect what we “know”?
How can we deal with these issues?

13. COG statements (Simons, Shoda, Lindsay, 2017)
What is a COG statement?
What problems does it solve?
Should they be required?
What do you think of their sample ones?

14. Sampling distributions
How are sampling distributions relevant to research?
What is the difference between the variance, standard deviation, and standard error?
How does the standard error relate to n? SD?
What do 68, 95, and 99 refer to?
What are confidence intervals? What do they tell us? When should they be used?

15. Significance, power, and effect sizes
The problem of p
The relationship between p, effect size, power, and
sample size
Practical vs. statistical significance
Small, medium, and large effect sizes (Cohen, 1992)

16. Power Why have people ignored power for so long?
Why does power matter?
What are typical levels of power in psychology?
Why do studies get published even if they are underpowered?
How does low power affect a) researchers; and b) science?

17. How do people usually determine power? Why is this a problem?
Publication bias and small n studies
Multiple tests
Heterogeneity (“method factors”; McShane & Bockenholt, 2014)
ES can be defined in different ways

18. What are other possible approaches to power?
Assess Type M and Type S errors (Gelman & Carlin, 2014)
Include heterogeneity estimate (McShane & Bockenholt, 2014)
Incorporate estimates for publication bias and uncertainty (Anderson, Kelley, & Maxwell, 2017; Taylor & Muller, 1996)
Sample size planning for accuracy (AIPE: Maxwell, Kelley, & Rausch, 2008)

19. Fig. 1. Power at standard sample size
Fig. 1. Power at standard sample size. 80% power is achieved at the standard sample size when heterogeneity τ2 is zero, but power diminishes as it increases.
Published in: Blakeley B. McShane; Ulf Böckenholt; Perspect Psychol Sci  9,
DOI: /
Copyright © 2014 Association for Psychological Science

20. Table 1. Results of the Monte Carlo Simulations for a 3 × 4 Split-Plot Analysis of Variance
Published in: Samantha F. Anderson; Ken Kelley; Scott E. Maxwell; Psychol Sci  28,
DOI: /
Copyright © 2017 Association for Psychological Science

21. Sample size for power vs. accuracy

22. Post-hoc power What’s post hoc power? What’s the problem with it?
When can it be useful?
Gelman & Carlin, 2014
Giner-Sorolla, 2017 (Report sensitivity analyses)

23. How can you increase power?
Increase sample size or alpha
Decrease mean square error by using better measures, increasing control, and getting high quality data
Use within-participant designs or use covariates
Increase the variance of the IV (use a more powerful treatment)
Use orthogonal contrasts or get predictors that aren’t correlated to each other
Ensure that you’re not violating assumptions of your stats
Look at theory and previous research to find the best, most powerful predictors
Use a more homogeneous sample
Do field studies
Increase sample size
Treat missing data in a more appropriate way
Funder et al., 2014; McClelland, 2000

24. Next week Methods assignment 1 due (can have til Friday)
Proposal references due—IN APA STYLE. Give tentative title of proposal at the top.
2 chapters
Articles on method bias, confounds, and alpha

25. To lab…
To do power analyses