Study Pre-Registration Why We Need It and How to Do It

Replication Crisis in Psychology Open Science Collaboration (2015) attempted to replicate 100 studies published in 3 top psychology journals in 2008 39% of the original studies were successfully replicated 25% of social psychology studies replicated 50% of cognitive psychology studies replicated Results may not replicate for many reasons (e.g., sampling variability, hidden moderators), but attention has focused on the prevalence of false positives in the original studies

False Positives False positives are Type I errors Claim an effect exists when it actually doesn’t p < .05 means that you are willing to accept a 5% chance of false positives False positive rate is inflated by various practices: Publication bias Journals biased towards significant, novel results Researcher degrees of freedom and questionable research practices (QRPs)

Researcher Degrees of Freedom Scientists are incentivized to publish Earns jobs, promotions, grants, etc. Motivated to make decisions that maximize chances of producing significant, novel, tidy findings May not intend to be dishonest QRPs are pervasive Open science movement aims to make the research process more transparent and eliminate QRPs

Researcher Degrees of Freedom Researchers have many decisions to make when conducting a study: Choosing a sample size – when to stop data collection How to deal with outliers/illegitimate responses Which conditions/groups should be compared Creating variables Which items? Transformations? Which variables should be included in analyses IVs, DVs, controls, mediators, moderators Making each decision increases the false positive rate

Questionable Research Practices P-hacking “Data fishing expedition” Try different types of analyses until p-value is driven below .05 HARKing Hypothesizing After Results are Known Look at the data first and then create a post-hoc hypothesis; present it as if it were developed a priori

Solutions: Transparent Research Process Researchers need to disclose their degrees of freedom so reviewers/other researchers can fully evaluate their work This includes: Open access to data Public repository of findings to reduce publication bias/file drawer problem E.g., PsyArXiv (OSF preprints) Preregistration

Pre-Registration Pre-registration documents all hypotheses, variables, materials, conditions, procedures, and analyses before data is collected or analysed Thus, pre-registration reduces selective reporting of results, p-hacking, HARKing Pre-registration clarifies what part of the study is confirmatory, and what part is exploratory Confirmatory: analyses specified in pre-registration Exploratory: analyses not specified in advance

Pre-Registration Doesn’t Mean That You Can’t Conduct Any Exploratory Analyses Exploratory work necessary and important Preregistration just means that you cannot pass off exploratory work as confirmatory Be up front and indicate which analyses are exploratory E.g., “In addition to the pre-registered analyses, we explored the association of…” Confirm the results of exploratory analyses in an independent, pre-registered study Resource and time-intensive

Advantages of Pre-Registration (from OSF) Confirmatory analyses help to reduce risk of false positives Pre-registration increases credibility of your findings and, by extension, the credibility of psychological science Staking claim to your ideas reduces “scooping” by other researchers

When Should You Pre-Register? (from OSF) Before you start collecting data for a study If you are asked by reviewers to collect more data for a study under review Before analysing data that has already been collected E.g., secondary data set

How Do I Pre-Register a Study? AsPredicted: https://aspredicted.org/ Open Science Framework (OSF): https://osf.io/

AsPredicted: 9 Pre-Registration Questions Have any data been collected for this study already? (must answer “no” to pre-register) What’s the main hypothesis being tested in this study? Describe the key dependent variable(s) specifying how they will be measured. Describe the conditions participants will be assigned to Specify exactly which analyses you will conduct to examine the main hypothesis Describe any secondary analyses Describe intended sample size Anything else you would like to pre-register (e.g., data exclusions, variables included for exploratory purposes)

Open Science Framework Complete archive of your research project Pre-registration Hypotheses, all variables and conditions, materials, procedures, analytic plan Amendments to pre-registration as you go along Data archive Analysis scripts Preprints of articles

https://cos.io/our-services/prereg-more-information/