David Mellor osf.io/qthsf @EvoMellor Building infrastructure to connect, preserve, speed up, and improve scholarship David Mellor osf.io/qthsf @EvoMellor Center for Open Science cos.io @OSFramework

Evidence to encourage change Incentives to embrace change Mission: To increase the openness, integrity, and reproducibility of scholarship. Strategy: Evidence to encourage change Incentives to embrace change Infrastructure to enable change Improving scientific ecosystem

Infrastructure Metascience Community

The combination of a strong bias toward statistically significant findings and flexibility in data analysis results in irreproducible research

The combination of a strong bias toward statistically significant findings and flexibility in data analysis results in irreproducible research Fanelli D (2010) “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE 5(4): e10068. doi:10.1371/journal.pone.0010068 http://127.0.0.1:8081/plosone/article?id=info:doi/10.1371/journal.pone.0010068

The Garden of Forking Paths The combination of a strong bias toward statistically significant findings and flexibility in data analysis results in irreproducible research The Garden of Forking Paths Control for time? Exclude outliers? Median or mean? “Does X affect Y?” Gelman and Loken, 2013

The combination of a strong bias toward statistically significant findings and flexibility in data analysis results in irreproducible research p-values Original Studies Replications 97% “significant” 37% “significant”

Incentives for individual success are focused on getting it published, not getting it right Nosek, Spies, & Motyl, 2012

Incentives to embrace change Improving scientific ecosystem

cos.io/prereg

Preregistration increases credibility by specifying in advance how data will be analyzed, thus preventing biased reasoning from affecting data analysis. cos.io/prereg

What is preregistration? A time-stamped, read-only version of your research plan created before the study. Preregistration Study plan: Hypothesis Data collection procedures Manipulated and measured variables Preregistration Study plan: Hypothesis Data collection procedures Manipulated and measured variables Analysis plan: Statistical model Inference criteria

What is preregistration? A time-stamped, read-only version of your research plan created before the study. Preregistration Study plan: Hypothesis Data collection procedures Manipulated and measured variables Preregistration Study plan: Hypothesis Data collection procedures Manipulated and measured variables Analysis plan: Statistical model Inference criteria

What is preregistration? A time-stamped, read-only version of your research plan created before the study. Preregistration Study plan: Hypothesis Data collection procedures Manipulated and measured variables Preregistration Study plan: Hypothesis Data collection procedures Manipulated and measured variables Analysis plan: Statistical model Inference criteria When the research plan undergoes peer review before results are known, the preregistration becomes part of a Registered Report

What problems does preregistration fix? The file drawer

What problems does preregistration fix? The file drawer P-Hacking: Unreported flexibility in data analysis

What problems does preregistration fix? The file drawer P-Hacking: Unreported flexibility in data analysis HARKing: Hypothesizing After Results are Known Dataset Hypothesis Kerr, 1998

What problems does preregistration fix? The file drawer P-Hacking: Unreported flexibility in data analysis HARKing: Hypothesizing After Results are Known Dataset Hypothesis Kerr, 1998

What problems does preregistration fix? Preregistration makes the distinction between confirmatory (hypothesis testing) and exploratory (hypothesis generating) research more clear. The file drawer P-Hacking: Unreported flexibility in data analysis HARKing: Hypothesizing After Results are Known Dataset Hypothesis Kerr, 1998

Confirmatory versus exploratory analysis Context of confirmation Traditional hypothesis testing Results held to the highest standards of rigor Goal is to minimize false positives P-values interpretable Context of discovery Pushes knowledge into new areas Finds unexpected relationships Goal is to minimize false negatives P-values meaningless! “In statistics, hypotheses suggested by a given dataset, when tested with the same dataset that suggested them, are likely to be accepted even when they are not true. This is because circular reasoning (double dipping) would be involved: something seems true in the limited data set, therefore we hypothesize that it is true in general, therefore we (wrongly) test it on the same limited data set, which seems to confirm that it is true. Generating hypotheses based on data already observed, in the absence of testing them on new data, is referred to as post hoc theorizing (from Latin post hoc, "after this"). The correct procedure is to test any hypothesis on a data set that was not used to generate the hypothesis.” Presenting exploratory results as confirmatory increases the publishability of results at the expense of credibility of results.

Confirmatory versus exploratory analysis Context of confirmation Traditional hypothesis testing Results held to the highest standards of rigor Goal is to minimize false positives P-values interpretable Context of discovery Pushes knowledge into new areas Finds unexpected relationships Goal is to minimize false negatives P-values meaningless! “In statistics, hypotheses suggested by a given dataset, when tested with the same dataset that suggested them, are likely to be accepted even when they are not true. This is because circular reasoning (double dipping) would be involved: something seems true in the limited data set, therefore we hypothesize that it is true in general, therefore we (wrongly) test it on the same limited data set, which seems to confirm that it is true. Generating hypotheses based on data already observed, in the absence of testing them on new data, is referred to as post hoc theorizing (from Latin post hoc, "after this"). The correct procedure is to test any hypothesis on a data set that was not used to generate the hypothesis.” Presenting exploratory results as confirmatory increases the publishability of results at the expense of credibility of results.

Confirmatory versus exploratory analysis Context of confirmation Traditional hypothesis testing Results held to the highest standards of rigor Goal is to minimize false positives P-values interpretable Context of discovery Pushes knowledge into new areas/ data-led discovery Finds unexpected relationships Goal is to minimize false negatives P-values meaningless “In statistics, hypotheses suggested by a given dataset, when tested with the same dataset that suggested them, are likely to be accepted even when they are not true. This is because circular reasoning (double dipping) would be involved: something seems true in the limited data set, therefore we hypothesize that it is true in general, therefore we (wrongly) test it on the same limited data set, which seems to confirm that it is true. Generating hypotheses based on data already observed, in the absence of testing them on new data, is referred to as post hoc theorizing (from Latin post hoc, "after this"). The correct procedure is to test any hypothesis on a data set that was not used to generate the hypothesis.” Presenting exploratory results as confirmatory increases the publishability of results at the expense of credibility of results.

Confirmatory versus exploratory analysis Context of confirmation Traditional hypothesis testing Results held to the highest standards of rigor Goal is to minimize false positives P-values interpretable Context of discovery Pushes knowledge into new areas/ data-led discovery Finds unexpected relationships Goal is to minimize false negatives P-values meaningless “In statistics, hypotheses suggested by a given dataset, when tested with the same dataset that suggested them, are likely to be accepted even when they are not true. This is because circular reasoning (double dipping) would be involved: something seems true in the limited data set, therefore we hypothesize that it is true in general, therefore we (wrongly) test it on the same limited data set, which seems to confirm that it is true. Generating hypotheses based on data already observed, in the absence of testing them on new data, is referred to as post hoc theorizing (from Latin post hoc, "after this"). The correct procedure is to test any hypothesis on a data set that was not used to generate the hypothesis.” Presenting exploratory results as confirmatory increases the publishability of results at the expense of credibility of results.

Collect New Data Example Workflow 1 Theory driven, a-priori expectations Collect New Data

Collect New Data Confirmation Phase Hypothesis testing Example Workflow 1 Theory driven, a-priori expectations Collect New Data Confirmation Phase Hypothesis testing

Hypothesis generating Example Workflow 1 Theory driven, a-priori expectations Collect New Data Confirmation Phase Hypothesis testing Discovery Phase Exploratory research Hypothesis generating

Example Workflow 2 Few a-priori expectations Collect Data

Hypothesis generating Example Workflow 2 Few a-priori expectations Collect Data Keep these data secret! Split Data Discovery Phase Exploratory research Hypothesis generating

Hypothesis generating Example Workflow 2 Few a-priori expectations Collect Data Keep these data secret! Split Data Discovery Phase Exploratory research Hypothesis generating Confirmation Phase Hypothesis testing

How do you preregister?

https://osf.io/prereg

Tips for writing up preregistered work Include a link to your preregistration (e.g. https://osf.io/f45xp) Report the results of ALL preregistered analyses ANY unregistered analyses must be transparent

Registered Reports

Registered Reports Are the hypotheses well founded? Are the methods and proposed analyses feasible and sufficiently detailed? Is the study well powered? (≥90%) Have the authors included sufficient positive controls to confirm that the study will provide a fair test?

Registered Reports Did the authors follow the approved protocol? Did positive controls succeed? Are the conclusions justified by the data?

None of these things matter Chambers, 2017

49 Journals currently accept RRs. Learn more at: cos.io/rr

Technology to enable change Improving scientific ecosystem

http://osf.io/ free, open source Share data, share materials, show the research process – confirmatory result make it clear, exploratory discovery make it clear; demonstrate the ingenuity, perspiration, and learning across false starts, errant procedures, and early hints – doesn’t have to be written in painstaking detail in the final report, just make it available. http://osf.io/ free, open source

Collaboration Documentation Archiving Content management and collaboration system Free service Connect, curate, search all aspects of the research project We don’t want to repeat ourselves Dataverse S3 Figshare Dropbox Service vs Application interface the service build new applicatoins this is right in line with SHARE-NS - openning, unlcoking this data allows for innovation From day one, we’ve been very excited about the SHARE partnerhship: community, expertise bringing to the table as well as shared interests mission Technical perspective, this project fit very much in line with what are building and more importantly how we are building it

Content management and collaboration system Free service Connect, curate, search all aspects of the research project We don’t want to repeat ourselves Dataverse S3 Figshare Dropbox Service vs Application interface the service build new applicatoins this is right in line with SHARE-NS - openning, unlcoking this data allows for innovation From day one, we’ve been very excited about the SHARE partnerhship: community, expertise bringing to the table as well as shared interests mission Technical perspective, this project fit very much in line with what are building and more importantly how we are building it

Put data, materials, and code on the OSF quite simply, it is a file repository that allows any sort of file to be stored and many types to be rendered in the browser without any special software. This is very important for increasing accessibility to research.

Manage access and permissions

Automate versioning With hashes

Connects Services Researchers Use

OpenSesame

OpenSesame

Thank you! Find this presentation at: https://osf.io/ Contact me at david@cos.io Our mission is to provide expertise, tools, and training to help researchers create and promote open science within their teams and institutions. Promoting these practices within the research funding and publishing communities accelerates scientific progress.

Literature cited Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., & Etchells, P. (2014). Instead of “playing the game” it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neuroscience, 1(1), 4–17. https://doi.org/10.3934/Neuroscience2014.1.4 Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484 Franco, A., Malhotra, N., & Simonovits, G. (2015). Underreporting in Political Science Survey Experiments: Comparing Questionnaires to Published Results. Political Analysis, 23(2), 306–312. https://doi.org/10.1093/pan/mpv006 Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University. Retrieved from www.​stat.​columbia.​edu/​~gelman/​research/​unpublished/​p_​hacking.​pdf Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. PLoS ONE, 10(8), e0132382. https://doi.org/10.1371/journal.pone.0132382 Kerr, N. L. (1998). HARKing: Hypothesizing After the Results are Known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4 Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science, 7(6), 615–631. https://doi.org/10.1177/1745691612459058 Ratner, K., Burrow, A. L., & Thoemmes, F. (2016). The effects of exposure to objective coherence on perceived meaning in life: a preregistered direct replication of Heintzelman, Trent & King (2013). Royal Society Open Science, 3(11), 160431. https://doi.org/10.1098/rsos.160431