Reinventing Scholarly Communication by Separating Publication From Evaluation Brian Nosek University of Virginia -- Center for Open Science http://briannosek.com/ -- http://cos.io/

The Current State of Scholarly Communication The research lifecycle – doing research, submitting to journal, ad hoc review, serial search through journals until arriving at one

Idea Design Collection Analysis Write Submit Publish Review Idea Design Collection Analysis Write Submit Publish Review Idea Design Collection Analysis Write Submit Publish Review Idea Design Collection Analysis Write Submit Publish Review

What’s wrong? Conflates actual aim (evaluation) with concrete indicator (publication)

Incentives for individual success are focused on getting it published, not getting it right Nosek, Spies, & Motyl, 2012

Also present of schol comm is… Slow Ineffective (reliability and validity) Inaccessible (pay walls or file-drawer) Inflexible (e.g., errors hard to correct)

Solution Decouple publication and evaluation Foster open marketplace of scholarly communication innovation

Fix 1: Preprints

721 days Benefits Faster More accessible More impact Risks Idea Design Collection Analysis Write Submit Publish Review 721 days Review Idea Design Collection Analysis Write Publish Benefits Faster More accessible More impact Risks No evaluation Information overload

Fix 2: Versioning Corrections are fast and discoverable Version history captures impact of evaluation Idea Design Collection Analysis Write Publish Publish Review

Fix 3: Diversify Evaluation ”Standard” vs. Post-publication Open vs. Closed Commenting Review Idea Design Collection Analysis Write Publish Publish

Diversify Evaluation Some solicit articles, some self-select Some open review, some closed Some accept/reject, some grade Some focus on methods, some substance area Some society, some commercial, some lab, some individuals

Diversity of Evaluation Science Submit, Closed, Editor, Accept/Reject JPSP Submit, Open, Editor+Comment, A/R Psych Sci Submit, Open, Editor, Score Lab Journal Select, Open, Editor Individuals Select, Open, Comment, Score

Fix 4: Article:Journal = 1:Many “journals” are not gatekeepers, they are evaluators and amplifiers

Future CV Accepted Science JPSP = 92/100 Axt, J., Nguyen, H., & Nosek, B. A. (2017, January 18). The Judgment Bias Task: A reliable, flexible method for assessing individual differences in social judgment biases. Retrieved from osf.io/bhzz6 Accepted Science JPSP = 92/100 Rated by Cog Measures, Rand Lab, Adam’s Faves

Fix 5: Registered Reports++ http://cos.io/rr Independent expertise throughout process Clarity of confirmatory vs. exploratory Solves file-drawer Review Review of intro and methods prior to data collection; published regardless of outcome Beauty vs. accuracy of reporting Publishing negative results Conducting replications Peer review focuses on quality of methods Idea Design Collection Analysis Write Publish Publish

Getting it done Open marketplace of evaluation services Publishing, Versioning, Commenting, Review Seed with demonstration cases Foster diversification in evaluation practices Continuous evaluation 33% Complete Center for Open Science Progress Bar

http://osf.io/preprints/

COS: http://cos. io/ OSF: http://osf. io/ OSF Preprints: http://osf COS: http://cos.io/ OSF: http://osf.io/ OSF Preprints: http://osf.io/preprints/ COS Strategic Plan: https://osf.io/x2w9h These slides: https://osf.io/rmvf2/ My general substantive interest in the gap between values and practices. The work that I am discussing today is a practical application of this interest to the gap between scientific values and practices. In particular, how can I best advance knowledge and my career at the same time? Challenges I face when working to advance scientific knowledge and my career at the same time. And, how my scientific practices can be adapted to meet my scientific values.

Open Access Open Data Open Workflows Outcomes Content Process Improving scientific ecosystem Make outcomes more accessible Make research content more accessible Make research process more accessible

http://osf.io

OpenSesame

OpenSesame

Content Experts Schol Comm Experts Technical Experts Three layers (let experts be experts, reduce redundancy and cost, accelerate innovation) Top = content and interfaces -> researchers care Middle = services -> schol comm innovators care Bottom = tool-kit -> developers care Services Ecosystem Data Technical Experts