1. Study Pre-Registration
Why We Need It and How to Do It

2. Replication Crisis in Psychology
Open Science Collaboration (2015) attempted to replicate 100 studies published in 3 top psychology journals in 2008
39% of the original studies were successfully replicated
25% of social psychology studies replicated
50% of cognitive psychology studies replicated
Results may not replicate for many reasons (e.g., sampling variability, hidden moderators), but attention has focused on the prevalence of false positives in the original studies

3. False Positives False positives are Type I errors
Claim an effect exists when it actually doesn’t
p < .05 means that you are willing to accept a 5% chance of false positives
False positive rate is inflated by various practices:
Publication bias
Journals biased towards significant, novel results
Researcher degrees of freedom and questionable research practices (QRPs)

4. Researcher Degrees of Freedom
Scientists are incentivized to publish
Earns jobs, promotions, grants, etc.
Motivated to make decisions that maximize chances of producing significant, novel, tidy findings
May not intend to be dishonest
QRPs are pervasive
Open science movement aims to make the research process more transparent and eliminate QRPs

5. Researcher Degrees of Freedom
Researchers have many decisions to make when conducting a study:
Choosing a sample size – when to stop data collection
How to deal with outliers/illegitimate responses
Which conditions/groups should be compared
Creating variables
Which items? Transformations?
Which variables should be included in analyses
IVs, DVs, controls, mediators, moderators
Making each decision increases the false positive rate

6. Questionable Research Practices
P-hacking
“Data fishing expedition”
Try different types of analyses until p-value is driven below .05
HARKing
Hypothesizing After Results are Known
Look at the data first and then create a post-hoc hypothesis; present it as if it were developed a priori

7. Solutions: Transparent Research Process
Researchers need to disclose their degrees of freedom so reviewers/other researchers can fully evaluate their work
This includes:
Open access to data
Public repository of findings to reduce publication bias/file drawer problem
E.g., PsyArXiv (OSF preprints)
Preregistration

8. Pre-Registration
Pre-registration documents all hypotheses, variables, materials, conditions, procedures, and analyses before data is collected or analysed
Thus, pre-registration reduces selective reporting of results, p-hacking, HARKing
Pre-registration clarifies what part of the study is confirmatory, and what part is exploratory
Confirmatory: analyses specified in pre-registration
Exploratory: analyses not specified in advance

9. Pre-Registration Doesn’t Mean That You Can’t Conduct Any Exploratory Analyses
Exploratory work necessary and important
Preregistration just means that you cannot pass off exploratory work as confirmatory
Be up front and indicate which analyses are exploratory
E.g., “In addition to the pre-registered analyses, we explored the association of…”
Confirm the results of exploratory analyses in an independent, pre-registered study
Resource and time-intensive

10. Advantages of Pre-Registration (from OSF)
Confirmatory analyses help to reduce risk of false positives
Pre-registration increases credibility of your findings and, by extension, the credibility of psychological science
Staking claim to your ideas reduces “scooping” by other researchers

11. When Should You Pre-Register? (from OSF)
Before you start collecting data for a study
If you are asked by reviewers to collect more data for a study under review
Before analysing data that has already been collected
E.g., secondary data set

12. How Do I Pre-Register a Study?
AsPredicted:
Open Science Framework (OSF):

13. AsPredicted: 9 Pre-Registration Questions
Have any data been collected for this study already? (must answer “no” to pre-register)
What’s the main hypothesis being tested in this study?
Describe the key dependent variable(s) specifying how they will be measured.
Describe the conditions participants will be assigned to
Specify exactly which analyses you will conduct to examine the main hypothesis
Describe any secondary analyses
Describe intended sample size
Anything else you would like to pre-register (e.g., data exclusions, variables included for exploratory purposes)

14. Open Science Framework
Complete archive of your research project
Pre-registration
Hypotheses, all variables and conditions, materials, procedures, analytic plan
Amendments to pre-registration as you go along
Data archive
Analysis scripts
Preprints of articles

15. https://cos.io/our-services/prereg-more-information/