1. Open science: State of play and perspectives
Arnaud Vaganay
Lausanne, 25 Sept 2017

2. Disclosure
“As a social scientist and meta-researcher, I am interested in researchers’ practices… good and bad”.

3. Disclosure
I consider that reproducible research is good scientific practice.

4. Disclosure
A study is reproducible if its results can be corroborated by independent investigators using the same data and codes.
Different from replicability (not addressed today).
Peng, R. (2009). Reproducible research and Biostatistics. Biostatistics, 10 (3):

5. Disclosure
This leads me to formulate 4 questions.

6. To what extent do researchers comply with this norm?
Why do they (fail to) comply?
Where do we go from here?
Does it matter?

7. To what extent do researchers comply with this norm?
Why do they (fail to) comply?
Where do we go from here?
Does it matter?

12. Compliance
Two key problems of researchers trying to reproduce results:
Not enough information about what was done in the first place;
The statistical significance of results rarely holds between studies.

13. To what extent do researchers comply with this norm?
Why do they (fail to) comply?
Where do we go from here?
Does it matter?

14. Explanations 1. Inadequate infrastructure: Limited word count
No data repository
Little interoperability
IP (closed access journals)

15. Explanations 2. Wrong incentives: Low productivity is sanctioned
Inconvenient results are sanctioned
Irreproducibility is not

16. Explanations
3. Legal/ethical obstacles:
Data privacy

17. Explanations 4. Insufficient skills:
Misunderstanding about the meaning and properties of the p-value
P-values are not the reliable benchmark many people think

18. Explanations
“When a measure becomes a target, it ceases to be a good measure”(*)
(*) Rephrased by Marilyn Strathern in:
Improving Ratings.
Audit in the British University System
European Review 5: 305–321.

19. Explanations What a difference a P-value makes!
Introducing a new drug on the market vs. not
Rolling out a new policy in the country vs. not

20. To what extent do researchers comply with this norm?
Why do they (fail to) comply?
Where do we go from here?
Does it matter?

21. Solutions 1. Research teams need to skill up in: Statistics
Data science

22. Solutions
2. Research teams need to pre-register their studies.

23. Solutions
3. Research teams need to better curate their workflows, for example, by using tools like:
The Open Science Framework
Github
And many others!

24. Solutions
4. Research teams need to reflect on what could affect their professional judgment.
For example, the effect of funding sources on results.

25. Solutions
5. Research teams would be well-advised to attempt a replication
Excellent learning technique

26. To what extent do researchers comply with this norm?
Why do they (fail to) comply?
Where do we go from here?
Does it matter?

27. Does it matter?

28. Does it matter?

29. Does it matter?

30. Stay in touch!
Arnaud