Practical Steps for Increasing Openness and Reproducibility Courtney Soderberg Statistical and Methodological Consultant Center for Open Science

INFRASTRUCTURE COMMUNITY METASCIENCE

Scientific Ideals - Innovative ideas - Reproducible results - Accumulation of knowledge

What is reproducibility? Computation Reproducibility: – If we took your data and code/analysis scripts and reran it, we can reproduce the numbers/graphs in your paper Empirical Reproducibility: – We have enough information to rerun the experiment or survey the way it was originally conducted Replicability: – We use your exact methods and analyses, but collect new data, and we get the same statistical results

Search and discover Develop idea Design study Acquire materials Collect data Store data Analyze data Interpret findings Write report Publish report

Why should you care? Your own work less efficient – Hard to build off our own work, or work of others in our lab We may not have the knowledge we think we have – Hard to even check this if reproducibility low

Current Barriers ● Statistical o Low Power o Researcher degrees of freedom ● Transparency o Poor documentation o Lack of openness

Steps 1.Create a structured workspace

Open Science Framework

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration

Pre-registration Before conducting a study registering: – The what of the study: General information about what you are investigating and how Research question Population and sample size General design Variables you’ll be collecting, or dataset you’ll be using

Pre-registration Study pre-registration decreases file-drawer effects – Helps with discovery of unpublished, usually null findings

Figure 1. Positive Results by Discipline. Fanelli D (2010) “ Positive ” Results Increase Down the Hierarchy of the Sciences. PLoS ONE 5(4): e doi: /journal.pone

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power

Button et al. (2013) Power in Neuroscience

Low Power ● Low replicability due to power: o 16% chance of finding the effect twice ● Inflated effect size estimates ● Decreased likelihood of true positives

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research

Pre-analysis plan Like a pre-registration – Detail the analyses planned for confirmatory hypothesis testing Decrease researcher degrees of freedom

Researcher Degrees of Freedom ● All data processing and analytical choices made after seeing and interacting with your data Should I collect more data? Which observations should I exclude? Which conditions should I compare? What should be my main DV? Should I look for an interaction effect?

False positive inflation Simmons, Nelson, & Simonsohn (2012)

Solution: Pre-registered analyses ● Before data is collected, specify o Sample size o Data processing and cleaning procedures o Exclusion criterion o Statistical Analyses ● Registered in a read-only format so it can’t be changed

Exploratory vs. Confirmatory Analyses Exploratory – Interested in exploring possible patterns/relationships in data to develop hypotheses Confirmatory – Have a specific hypothesis you want to test Pre-registration of analyses clarifies which are exploratory and which are confirmatory

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study 6.Analyze and document analyses

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study 6.Analyze and document analyses 7.Share study data, code, materials

Why you might want to share Journal/Funder mandates Increase impact of work Recognition of good research practices

What to share Sharing is a continuum – Data underlying just results reported in a paper – Data underlying publication + information about other variables collected – Data underlying publication + embargo on full dataset – All data collected for that study

Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study 6.Analyze and document analyses 7.Share study data, code, materials

How to make this more efficient? Have conversations with collaborators early – What is our data management plan? – What/when will we share? Be consistent across studies – If an entire lab has the same structure, then it’s easier to find things Document from the beginning

Where to get help: ● Reproducible Research Practices? o ● The OSF? o ● Have feedback for how we could support you more? o o

Registered Reports Committee

Early Adopters 1.AIMS Neuroscience 2.Attention, Perception & Psychophysics 3.Comparative Political Studies 4.Comprehensive Results in Social Psychology 5.Cortex 6.Drug and Alcohol Dependence 7.eLife 8.Experimental Psychology 9.Frontiers in Cognition 10.Journal of Business and Psychology 11.Nutrition and Food Science Journal 12.Perspectives on Psychological Science 13.Social Psychology 14.Working, Aging, and Retirement

Signals: Making Behaviors Visible Promotes Adoption

Data Availability in Psychological Science

Connecting silos to enable increased discovery

API Docs

free, open source

Put data, materials, and code on the OSF

Manage access and permissions

Automate versioning With hashes

Share your work

Now OSF

OpenSesame Soon OSF 29 grants to develop open tools and services:

Get a persistent identifier

See the impact File downloads

API Docs

Awaiting Instruction…