SAMPLING

Next week  2 book chapters  Outline of thesis proposal/paper intro  Find a scale and answer questions  Thought paper

General Sampling Issues  What is the sampling model?  What types of biases can come in at each point?  What is the proximity similarity model? What are issues with that model?  How can you increase external validity?  When do you need a representative sample?

Sampling Distributions  How are sampling distributions relevant to research?  What is the difference between the variance, standard deviation, and standard error?  How does the standard error relate to n? SD?  What do 68, 95, and 99 refer to?  What are confidence intervals? What do they mean?

Probability Sampling  Are random samples really random?  When would you use each?  Simple random sampling  Stratified random  Systematic random  Cluster  Multistage

Nonprobability Sampling  What are these?  When should they be used?  Convenience sample  Modal instance  Expert  Quota  Heterogeneity  Snowball

Census  How do censuses do sampling?  What are problems with the national census?  Ways to deal with them?

Power  What is power, and why does it matter?  Why do studies get published even if they are underpowered? What group-level consequence (for science) does this have?  How do people determine power?  How should they?

Power vs. accuracy  How does sample size planning for power vs. accuracy differ?  When would you want to do one vs. the other?  Figure 1. For AIPE, effect size doesn’t matter

Power  What stats should we report in a study? What does APA manual say?  How can simulations be used to estimate power?  How does power relate to meta-analyses?  How does power differ for omnibus vs. specific tests?  What’s post hoc power? What’s the problem with it?  For your papers in here, estimate power and justify your sample size (don’t just plug into g-power)

Other stat issues  The problem of p   The relationship between p, effect size, and n  Practical vs. statistical significance  How much power should we have?  Registries, multi-site trials  Standardized vs. unstandardized effect sizes

How can you increase power?  Increase sample size or alpha  Decrease mean square error by using better measures, increasing control, and getting high quality data  Use within-participant designs or use covariates  Increase the variance of the IV (use a more powerful treatment)  Use orthogonal contrasts or get predictors that aren’t correlated to each other  Ensure that you’re not violating assumptions of your stats  Look at theory and previous research to find the best, most powerful predictors  Use a more homogeneous sample  Do field studies  Increase sample size  Treat missing data in a more appropriate way  McClelland, 2000; Funder et al., 2014

Funder et al., 2014  SPSP Task Force on Publication and Research Practices  Recommendations:  Describe choice of N and issues of power  Report effect sizes and 95% CI  Avoid “questionable research practices”  Give all IV and DV instructions and measures in an Appendix or online  Provide data and coding to those who ask

 Get better outlets for replication studies  Be open to differences in methods, groups, etc.  Recommendations for education:  Encourage “getting it right” over “finding significant results” (p. 9)  Tell the “whole story” rather than a “good story” (p. 9)  Teach things like power, effect sizes, CI, questionable practices, etc.  Be good models

Cross-cultural research  What is a culture?  Why study things across cultures?  Best practices:  Have at least one insider on the research team  Match samples on typicality and as many things as you can besides culture  Translation and back translation  Are they cultural differences or miscommunications or differences of response styles?

Henrich, Heine, & Norenzayan, 2010  Sears, 1988  What are WEIRD samples?  How much can we generalize our results? When does generalization make sense?  Why do we focus so much on WEIRD samples?  What should we report about demographics?

Industrialized societies vs. small-scale societies  What kinds of differences exist?  Similarities?  Why?

Western vs. non-Western  What are differences?  Similarities?  Why?

Americans vs. other Westerners  How are we weird?  Why are we weird?  What does this suggest about what we “know” in psychology?

American participants vs. others  Differences?  How do college students differ from others?  For what topics are less likely or not to have an effect?  How do we differ from Americans in the past?

Overall  What problems do WEIRD samples cause?  When is it okay to use WEIRD samples?  How can we deal with these issues?  So is your research just a worthless pile of ….?