Scientific Utopia: I. Improving Scientific Communication Brian Nosek University of Virginia Center for Open Science

PURPOSE OF SCIENCE To acquire knowledge about nature

COMMUNICATION PROBLEMS None Slow Incomplete Inaccurate Unmodifiable

Objections Quality control Incentive structure Information overload Ego protection Status quo bias Too many with a stake in current system Commercial interests for secrecy Financial benefits for societies

NO MORE PAPER Solution: DEMO Problem: Scientific communication is inefficient

OPEN ACCESS Solution: DEMO Problem: Scientific communication is inefficient CoN

Public University Granting Agencies Scientists Publishers pays for doing research Gives ownership of research report, free Pays to access the research

SEPARATE PUBLICATION FROM EVALUATION Solution: Problem: Scientific communication is inefficient DEMODEMO 1 DEMODEMO 2

GRADING SYSTEM + DIVERSIFIED DISSEMINATION (END THE 1 JOURNAL MODEL) Solution: Problem: Scientific communication is inefficient DEMODEMO 1 DEMODEMO 2

OPEN PEER REVIEW Solution: DEMO Problem: Scientific communication is inefficient

CONTINUOUS PEER REVIEW Solution: DEMO Problem: Scientific communication is inefficient

Closing – the new reality

Acknowledgments Yoav Bar-Anan Members of the implicit cognition lab

Part 2 preview 1.No more am(bem)iguity between confirmatory and exploratory research 2.No more unpublished research 3.No more materials or data left out 4.No more articles

UniversitiesSocieties Individuals Commercial Publishers

Source of Inefficiency Mixing exploratory and confirmatory research + lack of emphasis on replication + emphasis on rejecting the null + lack of reporting null effects = literature with an overrepresentation of false positives + little chance of correcting false positives once published

Addressing the inefficiency with a Study Registry 1.Clearly identify the degree to which research is exploratory versus confirmatory – reduce false-positive, increase value of replication 2.Clearly identify what research has been done, whether or not it results in a published article – eliminate file-drawer problem

NO MORE ARTICLES Solution: DEMO Problem: Scientific communication is inefficient

The Empirical Research Process * with the “study” as the unit of analysis * CONCEPTUALIZATIONEXECUTIONEVALUATION 1.Theory/Background Research 2.Question/Hypothesis 3.Methods/Design 4.Analysis Plan a.Variable and response coding plan b.Cleaning/exclusion/ transformation plan c.Tests and evaluation plan 1.Sample Identification 2.Sample Recruiting 3.Data Collection 1.Analysis 2.Interpretation 3.Report Time

NO MORE MATERIALS OR DATA LEFT OUT Solution: DEMO Problem: Scientific communication is inefficient

Assumptions 1.Academic science is more efficient as an open, shared process 2.The current system is much more closed than it needs to be 3.The current incentive structure for individual contributors does not encourage openness 4.Current infrastructure does not make it easy to be open or share 5.Infrastructure can assist openness and increase sharing, leading to more efficient (and rapid) knowledge accumulation 6.Infrastructure can also make data accumulation, aggregation, and analysis more efficient

Macro Goals 1.Develop infrastructure to make openness and sharing effortless 2.Develop infrastructure, recognition systems, and process models to create incentives for individual contributors for openness and sharing 3.Develop infrastructure to make data accumulation, aggregation, and analysis more efficient

VALUES OF SCIENCE Transparency Accuracy Sharing Reproducibility Cumulative Progress Economy / Efficiency

Peer Review Submit for publication Draft article Analyze data Conduct a study Article published New research inspired Design a study