Practical Steps for Increasing Openness and Reproducibility Courtney Soderberg Statistical and Methodological Consultant Center for Open Science

INFRASTRUCTURE COMMUNITY METASCIENCE

Scientific Ideals - Innovative ideas - Reproducible results - Accumulation of knowledge

What is reproducibility? Computation Reproducibility: – If we took your data and code/analysis scripts and reran it, we can reproduce the numbers/graphs in your paper Empirical Reproducibility: – We have enough information to rerun the experiment or survey the way it was originally conducted Replicability: – We use your exact methods and analyses, but collect new data, and we get the same statistical results

Search and discover Develop idea Design study Acquire materials Collect data Store data Analyze data Interpret findings Write report Publish report

Why should you care? Your own work less efficient – Hard to build off our own work, or work of others in our lab We may not have the knowledge we think we have – Hard to even check this if reproducibility low

Current Barriers ● Statistical o Low Power o Researcher degrees of freedom ● Transparency o Poor documentation o Lack of openness

Steps 1.Create a structured workspace

Open Science Framework

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration

Pre-registration Before conducting a study registering: – The what of the study: General information about what you are investigating and how Research question Population and sample size General design Variables you’ll be collecting, or dataset you’ll be using

Pre-registration Study pre-registration decreases file-drawer effects – Helps with discovery of unpublished, usually null findings

Figure 1. Positive Results by Discipline. Fanelli D (2010) “ Positive ” Results Increase Down the Hierarchy of the Sciences. PLoS ONE 5(4): e doi: /journal.pone

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power

Button et al. (2013) Power in Neuroscience

Low Power ● Low replicability due to power: o 16% chance of finding the effect twice ● Inflated effect size estimates ● Decreased likelihood of true positives

Low Power ● Low replicability due to power: o 16% chance of finding the effect twice ● Inflated effect size estimates

Distribution Shape 30% Power 90% Power

Significant effect sizes 30% Power 90% Power

True Cohen’s d =.50

Low Power ● Low replicability due to power: o 16% chance of finding the effect twice ● Inflated effect size estimates ● Decreased likelihood of true positives

Positive Predictive Value Curves Button, Ioannidis, Mokrysz, Nosek, Flint, Robinson, & Munafo (2011)

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research

Pre-analysis plan Like a pre-registration – Detail the analyses planned for confirmatory hypothesis testing Decrease researcher degrees of freedom

Researcher Degrees of Freedom ● All data processing and analytical choices made after seeing and interacting with your data Should I collect more data? Which observations should I exclude? Which conditions should I compare? What should be my main DV? Should I look for an interaction effect?

False positive inflation Simmons, Nelson, & Simonsohn (2012)

Solution: Pre-registered analyses ● Before data is collected, specify o Sample size o Data processing and cleaning procedures o Exclusion criterion o Statistical Analyses ● Registered in a read-only format so it can’t be changed

Exploratory vs. Confirmatory Analyses Exploratory – Interested in exploring possible patterns/relationships in data to develop hypotheses Confirmatory – Have a specific hypothesis you want to test Pre-registration of analyses clarifies which are exploratory and which are confirmatory

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study 6.Analyze and document analyses

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study 6.Analyze and document analyses 7.Share study data, code, materials

Why you might want to share Journal/Funder mandates Increase impact of work Recognition of good research practices

Incentives for Researchers Badges for open practices

Signals: Making Behaviors Visible Promotes Adoption

Data Availability in Psychological Science

Incentives for Researchers Badges for open practices Registered Reports

Design Collect & Analyze ReportPublish PEER REVIE W

Registered Reports Design Collect & Analyze ReportPublish PEER REVIE W

Registered Reports AIMS Neuroscience Attention, Perception, & Psychophysics Cognition and Emotion Comprehensive Results in Social Psychology Cortex Drug and Alcohol Dependence eLife Euro Journal of Neuroscience Experimental Psychology Journal of Accounting Research Journal of Business and Psychology Journal of Personnel Psychology Journal of Media Psychology Nutrition and Food Science Journal Perspectives on Psych. Science Royal Society Open Science Social Psychology Stress and Health Work, Aging, and Retirement

Incentives for Researchers Badges for open practices Registered Reports Pre-Reg Challenge

The $1,000,000 Preregistration Challenge Endorse TOP Guidelines Badges for Open Practices Registered Reports

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study 6.Analyze and document analyses 7.Share study data, code, materials

How to make this more efficient? Have conversations with collaborators early – What is our data management plan? – What/when will we share? Be consistent across studies – If an entire lab has the same structure, then it’s easier to find things Document from the beginning

Where to get help: ● Reproducible Research Practices? o ● The OSF? o ● Have feedback for how we could support you more? o o

API Docs

Now OSF

OpenSesame Soon OSF 29 grants to develop open tools and services:

Awaiting Instruction…