Disrupting Scholarly Communication Brian Nosek University of Virginia -- Center for Open Science http://briannosek.com/ -- http://cos.io/

Metascience to encourage change Technology to enable change Training to enact change Incentives to embrace change Improving scientific ecosystem

The State of Scholarly Communication Origins and Operation The research lifecycle – doing research, submitting to journal, ad hoc review, serial search through journals until arriving at one

Idea Design Collection Analysis Write Submit Publish Review Idea Design Collection Analysis Write Submit Publish Review Idea Design Collection Analysis Write Submit Publish Review Idea Design Collection Analysis Write Submit Publish Review

What’s wrong? Conflates actual aim (evaluation) with concrete indicator (publication) Slow Ineffective (reliability and validity) Inaccessible (pay walls or file-drawer) Inflexible (e.g., errors hard to correct) Reduces reproducibility

Solution Decouple publication and evaluation Foster open marketplace for scholarly communication innovation

Fix 1: Preprints

721 days Benefits Faster More accessible More impact Risks Idea Design Collection Analysis Write Submit Publish Review 721 days Review Idea Design Collection Analysis Write Publish Benefits Faster More accessible More impact Risks No evaluation Information overload

Fix 2: Versioning Corrections are fast and discoverable Version history captures impact of evaluation Idea Design Collection Analysis Write Publish Publish Review

Fix 3: Diversify Evaluation Pre- vs. Post-publication Open vs. Closed Editor-selected vs. Open Commenting Review Idea Design Collection Analysis Write Publish Publish

“journals” are not gatekeepers, they are evaluators and amplifiers Fix 4: Article:Journal Change 1:1 to 1:Many “journals” are not gatekeepers, they are evaluators and amplifiers

Future CV Axt, J., Nguyen, H., & Nosek, B. A. (2017, January 18). The Judgment Bias Task: A reliable, flexible method for assessing individual differences in social judgment biases. Retrieved from osf.io/bhzz6. Evaluations: Accepted at Science; JPSP 92/100; Rated by Cog Measures, Rand Lab, EJ’s Faves Review service aggregate: 89/100 (7 evaluations) Community aggregate: 85/100 (108 evaluations)

Fix 5: Registered Reports++ http://cos.io/rr Independent expertise throughout process Clarity of confirmatory vs. exploratory Solves file-drawer Review Review of intro and methods prior to data collection; published regardless of outcome Beauty vs. accuracy of reporting Publishing negative results Conducting replications Peer review focuses on quality of methods Idea Design Collection Analysis Write Publish Publish

Marketplace of paper services

Services Ecosystem Data

OpenSesame

OpenSesame

Community Interfaces Interfaces Services Ecosystem Data OSF

http://osf.io/preprints/

Getting it done Open marketplace of evaluation services Publishing, Versioning, Commenting, Review Seed with demonstration cases Foster diversification in evaluation practices Continuous evaluation 33% Complete

Outcome An open marketplace to accelerate innovation in scholarly communication, evaluation, and research

COS Strategic Plan https://osf. io/x2w9h These slides https://osf COS Strategic Plan https://osf.io/x2w9h These slides https://osf.io/5dakh/ Nosek & Bar-Anan (2012) https://arxiv.org/abs/1205.1055 My general substantive interest in the gap between values and practices. The work that I am discussing today is a practical application of this interest to the gap between scientific values and practices. In particular, how can I best advance knowledge and my career at the same time? Challenges I face when working to advance scientific knowledge and my career at the same time. And, how my scientific practices can be adapted to meet my scientific values. Take a picture