Shifting the research culture toward openness and reproducibility Brian Nosek University of Virginia -- Center for Open Science http://briannosek.com/ -- http://cos.io/

Incentives for individual success are focused on getting it published, not getting it right Nosek, Spies, & Motyl, 2012

Norms Counternorms Communality Universalism Disinterestedness Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume Anderson, Martinson, & DeVries, 2007

Anderson, Martinson, & DeVries, 2007

What is good for me, the scientist versus What is good for science Psychological COI What is good for me, the scientist versus What is good for science

Signals: Making Behaviors Visible Promotes Adoption Badges Open Data Open Materials Preregistration Psychological Science (Jan 2014) Kidwell et al., 2016, PLOS Biology

40% 30% % Articles reporting that data was available 20% 10% 0%

Options with Badges Adopt! Collaborate on Randomized Trial 100 journals Randomly assign to offer badges or not Assess after a year

http://cos.io/top

TOP Guidelines Data citation Design transparency Research materials transparency Data transparency Analytic methods (code) transparency Preregistration of studies Preregistration of analysis plans Replication

Data sharing 1 2 3 Article states whether data are available, and, if so, where to access them Data must be posted to a trusted repository. Exceptions must be identified at article submission. Data must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication.

Some TOP Signatory Organizations AAAS/Science American Academy of Neurology American Geophysical Union American Heart Association American Meterological Society American Society for Cell Biology Association for Psychological Science Association for Research in Personality Association of Research Libraries Behavioral Science and Policy Association BioMed Central Committee on Publication Ethics Electrochemical Society Frontiers MDPI PeerJ Pensoft Publishers Public Library of Science The Royal Society Springer Nature Society for Personality and Social Psychology Society for a Science of Clinical Psychology Ubiquity Press Wiley

Options with TOP Become a signatory! Collaborate on education/adoption campaign with Editors Randomized trial

Context of Justification Confirmation Data independent Hypothesis testing Context of Discovery Exploration Data contingent Hypothesis generating p-values interpretable PREREGISTRATION p-values NOT interpretable Presenting exploratory as confirmatory increases publishability of results at the cost of credibility of results

Preregistration Challenge http://cos.io/prereg/

Options with Prereg Challenge Make more Elsevier journals eligible Co-marketing to promote engagement with Challenge at eligible journals

Registered Reports http://cos.io/rr Design Collect & Analyze Report Publish PEER REVIEW Review of intro and methods prior to data collection; published regardless of outcome Beauty vs. accuracy of reporting Publishing negative results Conducting replications Peer review focuses on quality of methods http://cos.io/rr, Committee Chair: Chris Chambers

Options with Registered Reports Adopt Special issues as pilots Randomized trial (grant application pending)

The Kindergartener’s Guide to Improving Research

1. Show your work 2. Share

http://osf.io

OpenSesame

OpenSesame

Will be added soon: And interest from 50+ research universities and organizations

What can you do? OSF: http://osf.io/ Prereg Challenge: http://cos.io/prereg Training: https://cos.io/our-services/training-services/ Adopt TOP Guidelines: http://cos.io/top/ Hackathon July 11-12. Community meeting: July 12-13.  Themes:  1) Pairing automatic enhancement with expert curation and the creation of tools to support these efforts 2) Pedagogy to develop expert curation of local data and technical skills to get SHARE data into local services (and then give back to SHARE) 3) Accessing (meta)data across the research workflow See below: I hope you'll consider attending SHARE's 2016 Community Meeting during the week of July 11, 2016 in Charlottesville, VA at the Center for Open Science (COS). If you're unfamiliar with SHARE or the free, open dataset we are creating, you can read more at share-research.org.    The meeting will include a hackathon and a working meeting and you are welcome to register for one or both of these components. We welcome a diversity of skills, skill levels, backgrounds, and interests at the hackathon. This diversity is not only welcome, but will result in a better, more impactful event. Please do consider attending both events, whether or not you are an (experienced) programmer. There is no registration fee for this meeting, but we are asking participants to cover their own travel and hotel costs in Charlottesville. We have a room block at the Omni Charlottesville with a rate of $149/night if booked by May 27. You can make your reservation by calling the Omni directly and mentioning the Center for Open Science. We do have a limited budget for travel support if needed. http://www.omnihotels.com/hotels/charlottesville Please fill out the registration form by Friday, April 15: https://arl.formstack.com/forms/share_community_meeting Thank you and please let me know if you have questions or concerns. These slides are shared at: https://osf.io/es6xz/ [take a picture] Email: Support@cos.io or nosek@virginia.edu