1. Reinventing Scholarly Communication by Separating Publication From Evaluation
Brian Nosek
University of Virginia -- Center for Open Science

2. The Current State of Scholarly Communication
The research lifecycle – doing research, submitting to journal, ad hoc review, serial search through journals until arriving at one

3. Idea Design Collection Analysis Write Submit Publish
Review
Idea Design Collection Analysis Write Submit Publish
Review
Idea Design Collection Analysis Write Submit Publish
Review
Idea Design Collection Analysis Write Submit Publish
Review

4. What’s wrong?
Conflates actual aim (evaluation) with concrete indicator (publication)

5. Incentives for individual success are focused on getting it published, not getting it right
Nosek, Spies, & Motyl, 2012

6. Also present of schol comm is…
Slow
Ineffective (reliability and validity)
Inaccessible (pay walls or file-drawer)
Inflexible (e.g., errors hard to correct)

7. Solution Decouple publication and evaluation
Foster open marketplace of scholarly communication innovation

8. Fix 1: Preprints

9. 721 days Benefits Faster More accessible More impact Risks
Idea Design Collection Analysis Write Submit Publish
Review
721 days
Review
Idea Design Collection Analysis Write Publish
Benefits
Faster
More accessible
More impact
Risks
No evaluation
Information overload

10. Fix 2: Versioning Corrections are fast and discoverable
Version history captures impact of evaluation
Idea Design Collection Analysis Write Publish Publish
Review

11. Fix 3: Diversify Evaluation
”Standard” vs. Post-publication
Open vs. Closed
Commenting
Review
Idea Design Collection Analysis Write Publish Publish

12. Diversify Evaluation Some solicit articles, some self-select
Some open review, some closed
Some accept/reject, some grade
Some focus on methods, some substance area
Some society, some commercial, some lab, some individuals

13. Diversity of Evaluation
Science Submit, Closed, Editor, Accept/Reject
JPSP Submit, Open, Editor+Comment, A/R
Psych Sci Submit, Open, Editor, Score
Lab Journal Select, Open, Editor
Individuals Select, Open, Comment, Score

14. Fix 4: Article:Journal = 1:Many
“journals” are not gatekeepers, they are evaluators and amplifiers

15. Future CV Accepted Science JPSP = 92/100
Axt, J., Nguyen, H., & Nosek, B. A. (2017, January 18). The Judgment Bias Task: A reliable, flexible method for assessing individual differences in social judgment biases. Retrieved from osf.io/bhzz6
Accepted Science
JPSP = 92/100
Rated by Cog Measures, Rand Lab, Adam’s Faves

16. Fix 5: Registered Reports++ http://cos.io/rr
Independent expertise throughout process
Clarity of confirmatory vs. exploratory
Solves file-drawer
Review
Review of intro and methods prior to data collection; published regardless of outcome
Beauty vs. accuracy of reporting
Publishing negative results
Conducting replications
Peer review focuses on quality of methods
Idea Design Collection Analysis Write Publish Publish

17. Getting it done Open marketplace of evaluation services
Publishing, Versioning, Commenting, Review
Seed with demonstration cases
Foster diversification in evaluation practices
Continuous evaluation
33% Complete
Center for Open Science Progress Bar

24. COS: http://cos. io/ OSF: http://osf. io/ OSF Preprints: http://osf
COS: OSF: OSF Preprints: COS Strategic Plan: These slides:
My general substantive interest in the gap between values and practices. The work that I am discussing today is a practical application of this interest to the gap between scientific values and practices. In particular, how can I best advance knowledge and my career at the same time?
Challenges I face when working to advance scientific knowledge and my career at the same time. And, how my scientific practices can be adapted to meet my scientific values.

25. Open Access Open Data Open Workflows
Outcomes
Content
Process
Improving scientific ecosystem
Make outcomes more accessible
Make research content more accessible
Make research process more accessible

27. OpenSesame

28. OpenSesame

29. Content Experts Schol Comm Experts Technical Experts
Three layers (let experts be experts, reduce redundancy and cost, accelerate innovation)
Top = content and interfaces -> researchers care
Middle = services -> schol comm innovators care
Bottom = tool-kit -> developers care
Services
Ecosystem
Data
Technical Experts