for research reproducibility The role of peer review for research reproducibility Flaminio Squazzoni GECS-Research Group on Experimental and Computational Sociology University of Brescia, Italy flaminio.squazzoni@unibs.it www.gecs.unibs.it Saint Malo 2017

A few bio notes Saint Malo 2017

A few bio notes Saint Malo 2017

Sign of the times! Saint Malo 2017

Peer review & reproducibility Peer review is the process through which (1) a manuscript is scrutinized by peers to ensure that scientific claims satisfy certain quality standards and (2) peers by confrontation add knowledge content of a manuscript Reproducibility is a set of conditions that make scientific knowledge and procedures in principle fully re-executable by others in order to replicate, re-use and extend procedures These processes improve collective learning and reduce research waste This improve science credibility and legitimacy at stakeholders Saint Malo 2017

What do peer review and reproducibility actually have in common? Saint Malo 2017

What do peer review and reproducibility actually have in common? Saint Malo 2017

What do reproducibility and data sharing (and peer review) have in common? Saint Malo 2017

A complex collective action problem? Reproducibility requires sophisticated technical and methodological skills that are rarely part of the ‘regular’ training of a scientist It requires reducing competitive pressures (high data collection investment, risk of plagiarism) It requires resources, which are drained by innovation Reproducibility and replication are not top priorities: journal editors, reviewers, authors, everyone looking for the next new thing Reproducibility requires a cultural change: reconsidering bias towards positive results and ‘experimental’ innovations? Saint Malo 2017

Top journals are not leading! Saint Malo 2017

However, do reproducibility and replication come without a cost? Saint Malo 2017

So what? There is a misalignment between technology, data-sharing/transparency/accountability outside-in academic culture, organizational practices, reward systems and scientist behavior Science is a collective enterprise that emerges from a complex ecosystem full of ambiguity and uncertainty The “crisis” guy is falling into a mix of romantic and futuristic syndromes: idealizing the past and playing with the future? Saint Malo 2017

An example: open peer review Saint Malo 2017

Does open peer review work? Saint Malo 2017

The model A population of N agents (authors & referees) Resources and quality Evaluation process: intrinsic vs. perceived quality Publish or perish Saint Malo 2017

Simulation scenarios Confidential peer review All referees are fair All referees are unreliable Scientists strategically reciprocate their previous publication/rejection when casted as referees (i.e., indirect reciprocity) Open peer review Authors strategically reciprocate with previous referees when casted as referees (i.e., TIT for TAT direct reciprocity) Referees are influenced by the author status and are more positive with authors of higher status 1, 2, 3 referees Saint Malo 2017

Publication bias with confidential peer review Saint Malo 2017

Publication bias with open peer review Saint Malo 2017

Status bias in open peer review Saint Malo 2017

The impact of multiple reviewers on publication bias Saint Malo 2017

Resource drain Saint Malo 2017

Impact of reviewer behaviour on quality of publications Saint Malo 2017

Here is the sociologist speaking Saint Malo 2017

Back to the point Research reproducibility is challenging funding schemes, peer review and scholarly journals It calls for data sharing guidelines, practices and instruments It requires infrastructure integration: dataset journals, “general” or specific data repositories, scholarly journals (note: funding agencies must support infrastructures, first!) Regulatory bodies: guidelines and standards that help managing the whole process, not single pieces New training initiatives: not only technical but especially ethical Reconsidering rewards at funding agencies and research institutes: valuing reviewing effort (Publons) and data providing Saint Malo 2017

Semi-conclusive and serious conclusions While the “crisis mood” is instrumental to call for action, it can be detrimental if it nurtures dis-embedded views of science as a complex ecosystem Science must point to reproducibility but is redundant and explorative by definition Science is full of trade-offs and conflicting priorities, which have to be publicly discussed and considered cross-sectorially The misalignment between changes, cultures and practices will be eventually solved in the near future, but not simply “technologically” or “normatively” Rewards and ethics will be key Saint Malo 2017

Saint Malo 2017