David Mellor osf.io/qthsf @EvoMellor Building infrastructure to connect, preserve, speed up, and improve scholarship David Mellor osf.io/qthsf @EvoMellor Center for Open Science cos.io @OSFramework

Evidence to encourage change Incentives to embrace change Mission: To increase the openness, integrity, and reproducibility of scholarship. Strategy: Evidence to encourage change Incentives to embrace change Infrastructure to enable change Improving scientific ecosystem

Infrastructure Metascience Community

The combination of a strong bias toward statistically significant findings and flexibility in data analysis results in irreproducible research

The combination of a strong bias toward statistically significant findings and flexibility in data analysis results in irreproducible research Fanelli D (2010) “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE 5(4): e10068. doi:10.1371/journal.pone.0010068 http://127.0.0.1:8081/plosone/article?id=info:doi/10.1371/journal.pone.0010068

The Garden of Forking Paths The combination of a strong bias toward statistically significant findings and flexibility in data analysis results in irreproducible research The Garden of Forking Paths Control for time? Exclude outliers? Median or mean? “Does X affect Y?” Gelman and Loken, 2013

The combination of a strong bias toward statistically significant findings and flexibility in data analysis results in irreproducible research p-values Original Studies Replications 97% “significant” 37% “significant”

Incentives for individual success are focused on getting it published, not getting it right Nosek, Spies, & Motyl, 2012

Barriers Perceived norms (Anderson, Martinson, & DeVries, 2007) Motivated reasoning (Kunda, 1990) Minimal accountability (Lerner & Tetlock, 1999) I am busy (Me & You, 2016) We can understand the nature of the challenge with existing psychological theory. For example: 1. The goals and rewards of publishing are immediate and concrete; the rewards of getting it right are distal and abstract (Trope & Liberman) 2. I have beliefs, ideologies, and achievement motivations that influence how I interpret and report my research (motivated reasoning; Kunda, 1990). And, even if I am trying to resist this motivated reasoning. I may simply be unable to detect it in myself, even when I can see those biases in others. 3. And, what biases might influence me. Well, pick your favorite. My favorite in this context is the hindsight bias. 4. What’s more is we face these potential biases in a context of minimal accountability. What you know of my laboratory work is only what you get in the published report. … 5. Finally, even if I am prepared to accept that I have these biases and am motivated to address them so that I can get it right. I am busy. So are you. If I introduce a whole bunch of new things that I must now do to check and correct for my biases, I will kill my productivity and that of my collaborators. So, the incentives lead me to think that my best course of action is to just to the best I can and hope that I’m doing it okay.

Incentives to embrace change Improving scientific ecosystem

cos.io/prereg

Preregistration increases credibility by specifying in advance how data will be analyzed, thus preventing biased reasoning from affecting data analysis. cos.io/prereg

Peer review before results are known to reduce bias and increase rigor Registered Reports Peer review before results are known to reduce bias and increase rigor cos.io/rr

Technology to enable change Improving scientific ecosystem

http://osf.io/ free, open source Share data, share materials, show the research process – confirmatory result make it clear, exploratory discovery make it clear; demonstrate the ingenuity, perspiration, and learning across false starts, errant procedures, and early hints – doesn’t have to be written in painstaking detail in the final report, just make it available. http://osf.io/ free, open source

Collaboration Documentation Archiving Content management and collaboration system Free service Connect, curate, search all aspects of the research project We don’t want to repeat ourselves Dataverse S3 Figshare Dropbox Service vs Application interface the service build new applicatoins this is right in line with SHARE-NS - openning, unlcoking this data allows for innovation From day one, we’ve been very excited about the SHARE partnerhship: community, expertise bringing to the table as well as shared interests mission Technical perspective, this project fit very much in line with what are building and more importantly how we are building it

Content management and collaboration system Free service Connect, curate, search all aspects of the research project We don’t want to repeat ourselves Dataverse S3 Figshare Dropbox Service vs Application interface the service build new applicatoins this is right in line with SHARE-NS - openning, unlcoking this data allows for innovation From day one, we’ve been very excited about the SHARE partnerhship: community, expertise bringing to the table as well as shared interests mission Technical perspective, this project fit very much in line with what are building and more importantly how we are building it

Put data, materials, and code on the OSF quite simply, it is a file repository that allows any sort of file to be stored and many types to be rendered in the browser without any special software. This is very important for increasing accessibility to research.

Manage access and permissions

Automate versioning With hashes

Connects Services Researchers Use

OpenSesame

OpenSesame

Registration

Template Forms

Share your work

Persistent Citable Identifiers

SHARE (http://share.osf.io/) Gather Notify Providers Consumers So this is what SHARE is doing - taking metadata about research from different kinds of digital repositories, normalizing it, and providing a feed, an API that can be queried, and a database that can be searched. This was Phase I - the data processing pipeline, workflow for building harvesters, development of the schema. Phase II is cleaning, enhancing, linking.

The OSF is one vision of an application framework

Toolkit Ecosystem Data OSF

Community Services Interfaces Toolkit Ecosystem Data OSF

Content Experts Schol Comm Experts Technical Experts Three layers (let experts be experts, reduce redundancy and cost, accelerate innovation) Top = content and interfaces -> researchers care Middle = services -> schol comm innovators care Bottom = tool-kit -> developers care Toolkit Ecosystem Data Technical Experts

http://osf.io/preprints/

osf.io/registries

Content Experts Schol Comm Experts Technical Experts Three layers (let experts be experts, reduce redundancy and cost, accelerate innovation) Top = content and interfaces -> researchers care Middle = services -> schol comm innovators care Bottom = tool-kit -> developers care Toolkit Ecosystem Data Technical Experts

Connecting, preserving, speeding up, and improving scholarship These slides are shared at: https://osf.io/ [Take a picture!] Thank you! David@cos.io @EvoMellor