The Reproducible Research Advantage Why + how to make your research more reproducible Presentation for Research Week February 22, 2016 April Clyburne-Sherin

Objectives What is reproducibility? Why practice reproducibility? What materials are necessary for reproducibility? How can you make your research reproducible?

What is reproducibility? Scientific method Replication of findings is highest standard of evaluating evidence Focuses on validating the scientific claim Requires transparency of methods, data, and code Minimum standard for any scientific study Observation Question Hypothesis Prediction Testing Analysis Replication Replicability: How well scientific findings can be replicated. A study is considered replicable when the original findings were produced again when the study was completely repeated.

Why practice reproducibility? Research that produces novel results, statistically significant results, clean results, more likely to be published (Fanelli, D. Negative results are disappearing from most disciplines and countries. Scientometrics 90, 891–904 (2012). Greenwald, A. G. Consequences of prejudice against the null hypothesis. Psychol. Bull. 82, 1–20 (1975)) As a consequence, researchers have strong incentives to engage in research practices that make their findings publishable quickly, even if those practices reduce the likelihood that the findings reflect a true effect . Such practices include using flexible study designs and flexible statistical analyses and running small studies with low statistical power1,5 .

Why practice reproducibility? There is evidence that our published literature is too good to be true. Daniele Fanelli did an analysis of what gets published across scientific disciplines and found that all disciplines had positive result rates of 70% or higher. From physics through psychology, the rates were 85-92%. Consider our field’s 92% positive result rate in comparison to the average power of published studies. Estimates suggest that the average psychology study has a power of somewhere around .5 to .6 to detect its effects. So, if all published results were true, we’d expect somewhere between 50-60% of the critical tests to reject the null hypothesis. But we get 92%. That does not compute. Something is askew in the accumulating evidence. [It is not in my interest to write up negative results, even if they are true, because they are less likely to be published (negative) – file-drawer effect] The accumulating evidence suggests an alarming degree of mis-estimation. Across disciplines, most published studies demonstrate positive results – results that indicate an expected association between variables or a difference between experimental conditions (Fanelli, 2010, 2012; Sterling, 1959). Fanelli observed a positive result rate of 85% in neuroscience (Fanelli, 2010). Fanelli D (2010) “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE 5(4): e10068.

Why practice reproducibility? Study report Study report is not enough to: Assess the sensitivity of findings to assumptions Replicate the findings Distinguish confirmatory from exploratory analyses Identify protocol deviations Cannot evaluate the study analyses and findings using a study report alone. Reported results Figures Even the best-reported study report does not contain enough information to completely evaluate the study analyses and findings. Some describe scientific publications as simply “advertising” the actual research. Tables Numerical summaries

Why practice reproducibility? Study report Processing Analysing Reporting Reported results Figures Raw data Analytic data Raw results Following data collection, there are: Many materials created that are not routinely published with the study report Many steps are taken in processing and analysing that are not routinely published in the study report Tables Collecting Numerical summaries

Why practice reproducibility? Study report Processing Analysing Reporting Reported results Figures Raw data Analytic data Raw results To fully assess the analyses and findings of a study, we need more information from scientists than the study report alone. Why should scientists care? Why should scientists put in the time to prepare and share additional materials and information? Tables Collecting Numerical summaries To fully assess the analyses and findings of a study, we need more information.

What materials are necessary for reproducibility? Study report Processing Analysing Reporting Reported results Figures Raw data Analytic data Raw results In order for research to be reproducible, the methods, the data, and the code must be shared. Although the raw data can be shared if desired or appropriate, the minimum requirement for reproducibility is: Documentation of your methods Processing, analytic, and presentation code Analytic data Tables Collecting Data + metadata Code Documentation of methods Numerical summaries

Why practice reproducibility? The idealist The pragmatist Shoulders of giants! Minimum scientific standard Allows others to build on your findings Improved transparency Increased transfer of knowledge Increased utility of your data + methods Data sharing citation advantage (Piwowar 2013) “It takes some effort to organize your research to be reproducible… the principal beneficiary is generally the author herself.”- Schwab & Claerbout Improves capacity for complex and large datasets or analyses Increased productivity Reproducible research practices are good for science! Reproducible research practices are good for YOU. By sharing your research materials, you may bump up your citations! You will NOT REMEMBER what you did. You will NOT REMEMBER why you did it. By documenting what you did clearly, you are communicating what you did with your future self: Defend what you did Means you can build on it with your next project Pass your work on to the next lab personnel Reproducible research practices involve research planning, organization, and automation. This can improve your research capacity and productivity.

How can you make your research reproducible? Power Data management plan Informative naming + location Study plan + pre-analysis plan 1. Plan for reproducibility before you start Version control Documentation 2. Keep track of things Reporting Confirmatory vs. exploratory analyses 3. Contain bias 4. Archive + share your materials

1. Plan for reproducibility before you start Power How? Calculate your power Low power means: Low probability of finding true effects Low probability that a positive is a true positive (positive predictive value) Exaggerated estimate of the magnitude of effect when true effect discovered Greater vibration of effects Estimate the size of effect you are studying Design your study with sufficient power to detect that effect If you need more power, consider collaborating If your study is underpowered, report this and acknowledge this limitation in the interpretation of your results

1. Plan for reproducibility before you start Data management plan How? Prepare to share Data that is well-managed from the start is easier to prepare for sharing Smooths transitions between researchers Protects you if questions are raised about data validity Metadata provides context Document metadata while collecting to save time Use open data formats rather than proprietary: .csv, .txt , .png Data: Collected Stored Documented Managed Metadata: Documented / Version control Metadata provides context How When What were the experimental conditions

1. Plan for reproducibility before you start Informative name + location Plan your file naming + location system a priori Names and locations should be distinctive, consistent, and informative: What it is Why it exists How it relates to other files

1. Plan for reproducibility before you start Informative name + location The rules don’t matter. That you have rules matters. Make it machine readable: Default ordering Use of meaningful deliminators and tags Example: use “_” and “-” to store metadata in name (eg, YYYY-MM-DD_assay_sample-set_well) Make it human readable: Choose self-explanatory names and locations

1. Plan for reproducibility before you start Study plan How? Pre-register your study plan before you look at your data Public registration of all studies counters publication bias Counters selective reporting and outcome reporting bias Distinguishes a priori design decisions from post hoc Corroborates the rigor of your findings Hypothesis Study design Type of design Sampling Power and sample size Randomization? Variables measured Meaningful effect size Variables constructed Data processing Open Science Framework ClinicalTrials.gov

1. Plan for reproducibility before you start Pre-analysis plan How? Pre-register your analysis plan before you look at your data Defines your confirmatory analyses Corroborates the rigor of your findings Define data analysis set Statistical analyses Primary Secondary Exploratory Missing data Outliers Multiplicity Subgroups + covariates (Adams-Huet and Ahn, 2009) Your analysis plan should include the planning all the way from raw to processed data, from analysis to presentation. Processing Analysing Raw data Analytic data Raw results

2. Keep track of things Version control Version control for data Track your changes Everything created manually should use version control Tracks changes to files, code, metadata Allows you to revert to old versions Make incremental changes: commit early, commit often Git / GitHub / BitBucket Metadata should be version controlled

2. Keep track of things Documentation Document everything done by hand Document your software environment (eg, dependencies, libraries, sessionInfo () in R) Everything done by hand or not automated from data and code should be precisely documented: README files Make raw data read only You won’t edit it by accident Forces you to document or code data processing Document in code comments Track software environment: Computer architecture operating system software toolchain supporting software + infrastructure (libraries, r packages, dependencies) external dependencies (data repositories, web sites) version numbers for everything

3. Contain bias CONSORT SAMPL Reporting How? Report transparently + completely Transparently means: Readers can use the findings Replication is possible Users are not misled Findings can be pooled in meta-analyses Completely means: All results are reported, no matter their direction or statistical significance Use reporting guidelines CONSORT Consolidated Standards of Reporting Trials SAMPL Statistical analyses and methods in the published literature

3. Contain bias Confirmatory vs. exploratory How? Distinguish confirmatory from exploratory analyses Provide access to your pre-registered analysis plan Avoid HARKing: Hypothesizing After the Results are Known Report all deviations from your study plan Report which decisions were made after looking at the data Share with RPubs

4. Archive + share your materials Open Science Framework Where doesn’t matter. That you share matters. Get credit for your code, your data, your methods Increase the impact of your research

How can you make your research reproducible? Power – Calculate your power Data management plan – Prepare to share Informative naming + location – The rules don’t matter. That you have rules matters. Study plan + pre-analysis plan – Pre-register your plans 1. Plan for reproducibility before you start Version control – Track your changes Documentation – Document everything done by hand 2. Keep track of things Reporting – Report transparently + completely Confirmatory vs. exploratory analyses – Distinguish confirmatory from exploratory 3. Contain bias Where doesn’t matter. That you share matters. 4. Archive + share your materials

How to learn more Organizing a project for reproducibility Reproducible Science Curriculum by Jenny Bryan https://github.com/reproducible-science-curriculum/ Data management Data Management from Software Carpentry by Orion Buske http://software-carpentry.org/v4/data/mgmt.html Literate programming Literate Statistical Programming by Roger Peng https://www.youtube.com/watch?v=YcJb1HBc-1Q Version control Version Control by Software Carpentry http://software-carpentry.org/v4/vc/ Sharing materials Open Science Framework by Center for Open Science https://osf.io/