1. for research reproducibility
The role of peer review
for research reproducibility
Flaminio Squazzoni GECS-Research Group on Experimental and Computational Sociology
University of Brescia, Italy
Saint Malo 2017

2. A few bio notes
Saint Malo 2017

3. A few bio notes
Saint Malo 2017

4. Sign of the times!
Saint Malo 2017

5. Peer review & reproducibility
Peer review is the process through which (1) a manuscript is scrutinized by peers to ensure that scientific claims satisfy certain quality standards and (2) peers by confrontation add knowledge content of a manuscript
Reproducibility is a set of conditions that make scientific knowledge and procedures in principle fully re-executable by others in order to replicate, re-use and extend procedures
These processes improve collective learning and reduce research waste
This improve science credibility and legitimacy
at stakeholders
Saint Malo 2017

6. What do peer review and reproducibility actually have in common?
Saint Malo 2017

7. What do peer review and reproducibility actually have in common?
Saint Malo 2017

8. What do reproducibility and data sharing (and peer review) have in common?
Saint Malo 2017

9. A complex collective action problem?
Reproducibility requires sophisticated technical and methodological skills that are rarely part of the ‘regular’ training of a scientist
It requires reducing competitive pressures (high data collection investment, risk of plagiarism)
It requires resources, which are drained by innovation
Reproducibility and replication are not top priorities: journal editors, reviewers, authors, everyone looking for the next new thing
Reproducibility requires a cultural change: reconsidering bias towards positive results and ‘experimental’ innovations?
Saint Malo 2017

10. Top journals are not leading!
Saint Malo 2017

11. However, do reproducibility and replication come without a cost?
Saint Malo 2017

12. So what?
There is a misalignment between technology, data-sharing/transparency/accountability outside-in academic culture, organizational practices, reward systems and scientist behavior
Science is a collective enterprise that emerges from a complex ecosystem full of ambiguity and uncertainty
The “crisis” guy is falling into a mix of romantic and futuristic syndromes: idealizing the past and playing with the future?
Saint Malo 2017

13. An example: open peer review
Saint Malo 2017

14. Does open peer review work?
Saint Malo 2017

15. The model A population of N agents (authors & referees)
Resources and quality
Evaluation process:
intrinsic vs. perceived quality
Publish or perish
Saint Malo 2017

16. Simulation scenarios Confidential peer review All referees are fair
All referees are unreliable
Scientists strategically reciprocate
their previous publication/rejection
when casted as referees (i.e., indirect
reciprocity)
Open peer review
Authors strategically reciprocate with
previous referees when casted as referees (i.e., TIT for TAT direct reciprocity)
Referees are influenced by the author status and are more positive with authors of higher status
1, 2, 3 referees
Saint Malo 2017

17. Publication bias with confidential peer review
Saint Malo 2017

18. Publication bias with open peer review
Saint Malo 2017

19. Status bias in open peer review
Saint Malo 2017

20. The impact of multiple reviewers on publication bias
Saint Malo 2017

21. Resource drain
Saint Malo 2017

22. Impact of reviewer behaviour on quality of publications
Saint Malo 2017

23. Here is the sociologist speaking
Saint Malo 2017

24. Back to the point
Research reproducibility is challenging funding schemes, peer review and scholarly journals
It calls for data sharing guidelines, practices and instruments
It requires infrastructure integration: dataset journals, “general” or specific data repositories, scholarly journals (note: funding agencies must support infrastructures, first!)
Regulatory bodies: guidelines and standards that help managing the whole process, not single pieces
New training initiatives: not only technical but especially ethical
Reconsidering rewards at funding agencies and research institutes: valuing reviewing effort (Publons) and data providing
Saint Malo 2017

25. Semi-conclusive and serious conclusions
While the “crisis mood” is instrumental to call for action, it can be detrimental if it nurtures dis-embedded views of science as a complex ecosystem
Science must point to reproducibility but is redundant and explorative by definition
Science is full of trade-offs and conflicting priorities, which have to be publicly discussed and considered cross-sectorially
The misalignment between changes, cultures and practices will be eventually solved in the near future, but not simply “technologically” or “normatively”
Rewards and ethics will be key
Saint Malo 2017

26. Saint Malo 2017