Open science: State of play and perspectives Arnaud Vaganay Lausanne, 25 Sept 2017

Disclosure “As a social scientist and meta-researcher, I am interested in researchers’ practices… good and bad”.

Disclosure I consider that reproducible research is good scientific practice.

Disclosure A study is reproducible if its results can be corroborated by independent investigators using the same data and codes. Different from replicability (not addressed today). Peng, R. (2009). Reproducible research and Biostatistics. Biostatistics, 10 (3): 405-408 . https://doi.org/10.1093/biostatistics/kxp014

Disclosure This leads me to formulate 4 questions.

To what extent do researchers comply with this norm? Why do they (fail to) comply? Where do we go from here? Does it matter?

To what extent do researchers comply with this norm? Why do they (fail to) comply? Where do we go from here? Does it matter?

Compliance Two key problems of researchers trying to reproduce results: Not enough information about what was done in the first place; The statistical significance of results rarely holds between studies.

To what extent do researchers comply with this norm? Why do they (fail to) comply? Where do we go from here? Does it matter?

Explanations 1. Inadequate infrastructure: Limited word count No data repository Little interoperability IP (closed access journals)

Explanations 2. Wrong incentives: Low productivity is sanctioned Inconvenient results are sanctioned Irreproducibility is not

Explanations 3. Legal/ethical obstacles: Data privacy

Explanations 4. Insufficient skills: Misunderstanding about the meaning and properties of the p-value P-values are not the reliable benchmark many people think

Explanations “When a measure becomes a target, it ceases to be a good measure”(*) (*) Rephrased by Marilyn Strathern in: Improving Ratings. Audit in the British University System European Review 5: 305–321.

Explanations What a difference a P-value makes! Introducing a new drug on the market vs. not Rolling out a new policy in the country vs. not

To what extent do researchers comply with this norm? Why do they (fail to) comply? Where do we go from here? Does it matter?

Solutions 1. Research teams need to skill up in: Statistics Data science

Solutions 2. Research teams need to pre-register their studies.

Solutions 3. Research teams need to better curate their workflows, for example, by using tools like: The Open Science Framework Github And many others!

Solutions 4. Research teams need to reflect on what could affect their professional judgment. For example, the effect of funding sources on results.

Solutions 5. Research teams would be well-advised to attempt a replication Excellent learning technique

To what extent do researchers comply with this norm? Why do they (fail to) comply? Where do we go from here? Does it matter?

Does it matter?

Does it matter?

Does it matter?

Stay in touch! Arnaud Vaganay arnaud@meta-lab.co @arnaudvag