Achieving Open Science Brian Nosek University of Virginia -- Center for Open Science http://briannosek.com/ -- http://cos.io/ Shifting the scholarly culture toward open access, open data, and open workflow is partly an incentives problem, partly an infrastructure problem, and partly a coordination problem.  The Center for Open Science (COS; http://cos.io/) is a non-profit technology and culture change organization working on all three.  The central components of COS’s strategy are [1] fostering a commercial environment that monetizes on service delivery, not controlling access to content, [2] providing free, open, public goods infrastructure that scholarly communities brand and operate based on their local norms, and [3] coordinating across disciplinary and stakeholder silos to align scholarly practices with scholarly values.  

Changing scientific culture is a coordination problem ecosystem UNIVERSITIES PUBLISHING FUNDERS SOCIETIES

Technology to enable open Training to enact open Incentives to embrace open Improving scientific ecosystem

Open Science Framework http://osf.io/ OpenSesame

Munafo et al., 2017, Nature Human Behavior

Example: Preregistration Challenge http://cos.io/prereg Training Example: Preregistration Challenge http://cos.io/prereg Scholarly Paper Guided Workflow Available Support https://osf.io/2dxu5 http://osf.io/prereg Help guides, consulting

Incentives ecosystem UNIVERSITIES PUBLISHING FUNDERS SOCIETIES

Incentives for individual success are focused on getting it published, not getting it right Nosek, Spies, & Motyl, 2012

Norms Counternorms Communality Universalism Disinterestedness Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Secrecy Closed Particularism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume

University Incentives Hiring, promotion, tenure

Journals and Funders Incentives Badges Standards Innovation in Publishing

Signals: Making Behaviors Visible Promotes Adoption Badges Open Data Open Materials Preregistration Psychological Science (Jan 2014) https://cos.io/our-services/open-science-badges/; 18 adopting journals Kidwell et al., 2016

40% 30% % Articles reporting data available in repository 20% 10% 0%

http://cos.io/top

TOP Guidelines Data citation Design transparency Research materials transparency Data transparency Analytic methods (code) transparency Preregistration of studies Preregistration of analysis plans Replication

Some TOP Signatory Organizations AAAS/Science American Academy of Neurology American Geophysical Union American Heart Association American Meterological Society American Society for Cell Biology Association for Psychological Science Association for Research in Personality Association of Research Libraries Behavioral Science and Policy Association BioMed Central Committee on Publication Ethics Electrochemical Society Frontiers MDPI PeerJ Pensoft Publishers Public Library of Science The Royal Society Springer Nature Society for Personality and Social Psychology Society for a Science of Clinical Psychology Ubiquity Press Wiley

Registered Reports https://cos.io/rr/ Design Collect & Analyze Report Publish PEER REVIEW Review of intro and methods prior to data collection; published regardless of outcome Beauty vs. accuracy of reporting Publishing negative results Conducting replications Peer review focuses on quality of methods Committee Chair: Chris Chambers; 49 adopting journals as of Feb 2017

Society Incentives Norms Counternorms Communality Universalism Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Secrecy Closed Particularism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume

Anderson, Martinson, & DeVries, 2007

The Kindergartener’s Guide to Improving Research

1. Show your work 2. Share

Technology to enable open Training to enact open Incentives to embrace open Metascience to evaluate open Improving scientific ecosystem

COS Strategic Plan https://osf.io/x2w9h These slides My general substantive interest in the gap between values and practices. The work that I am discussing today is a practical application of this interest to the gap between scientific values and practices. In particular, how can I best advance knowledge and my career at the same time? Challenges I face when working to advance scientific knowledge and my career at the same time. And, how my scientific practices can be adapted to meet my scientific values. Take a picture