1. David Mellor osf.io/qthsf @EvoMellor
Building infrastructure to connect, preserve, speed up, and improve scholarship
David Mellor
osf.io/qthsf
@EvoMellor
Center for Open Science
cos.io
@OSFramework

2. Evidence to encourage change Incentives to embrace change
Mission: To increase the openness, integrity, and reproducibility of scholarship.
Strategy:
Evidence to encourage change
Incentives to embrace change
Infrastructure to enable change
Improving scientific ecosystem

3. Infrastructure
Metascience
Community

4. The combination of a strong bias toward statistically significant findings and flexibility in data analysis results in irreproducible research

5. The combination of a strong bias toward statistically significant findings and flexibility in data analysis results in irreproducible research
Fanelli D (2010) “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE 5(4): e doi: /journal.pone

6. The Garden of Forking Paths
The combination of a strong bias toward statistically significant findings and flexibility in data analysis results in irreproducible research
The Garden of Forking Paths
Control for time?
Exclude outliers?
Median or mean?
“Does X affect Y?”
Gelman and Loken, 2013

7. The combination of a strong bias toward statistically significant findings and flexibility in data analysis results in irreproducible research
p-values
Original Studies Replications
97% “significant”
37% “significant”

8. Incentives for individual success are focused on getting it published, not getting it right
Nosek, Spies, & Motyl, 2012

9. Barriers Perceived norms (Anderson, Martinson, & DeVries, 2007)
Motivated reasoning (Kunda, 1990)
Minimal accountability (Lerner & Tetlock, 1999)
I am busy (Me & You, 2016)
We can understand the nature of the challenge with existing psychological theory. For example:
1. The goals and rewards of publishing are immediate and concrete; the rewards of getting it right are distal and abstract (Trope & Liberman)
2. I have beliefs, ideologies, and achievement motivations that influence how I interpret and report my research (motivated reasoning; Kunda, 1990). And, even if I am trying to resist this motivated reasoning. I may simply be unable to detect it in myself, even when I can see those biases in others.
3. And, what biases might influence me. Well, pick your favorite. My favorite in this context is the hindsight bias.
4. What’s more is we face these potential biases in a context of minimal accountability. What you know of my laboratory work is only what you get in the published report. …
5. Finally, even if I am prepared to accept that I have these biases and am motivated to address them so that I can get it right. I am busy. So are you. If I introduce a whole bunch of new things that I must now do to check and correct for my biases, I will kill my productivity and that of my collaborators. So, the incentives lead me to think that my best course of action is to just to the best I can and hope that I’m doing it okay.

10. Incentives to embrace change
Improving scientific ecosystem

12. cos.io/prereg

13. Preregistration increases credibility by specifying in advance how data will be analyzed, thus preventing biased reasoning from affecting data analysis.
cos.io/prereg

14. Peer review before results are known to reduce bias and increase rigor
Registered Reports
Peer review before results are known to reduce bias and increase rigor
cos.io/rr

15. Technology to enable change
Improving scientific ecosystem

16. http://osf.io/ free, open source
Share data, share materials, show the research process – confirmatory result make it clear, exploratory discovery make it clear; demonstrate the ingenuity, perspiration, and learning across false starts, errant procedures, and early hints – doesn’t have to be written in painstaking detail in the final report, just make it available.
free, open source

17. Collaboration Documentation Archiving
Content management and collaboration system
Free service
Connect, curate, search all aspects of the research project
We don’t want to repeat ourselves
Dataverse S3
Figshare
Dropbox
Service vs Application interface
the service build new applicatoins
this is right in line with SHARE-NS - openning, unlcoking this data
allows for innovation
From day one, we’ve been very excited about the SHARE partnerhship:
community, expertise bringing to the table
as well as
shared interests
mission
Technical perspective, this project fit very much in line with what are building and more importantly how we are building it

18. Content management and collaboration system
Free service
Connect, curate, search all aspects of the research project
We don’t want to repeat ourselves
Dataverse S3
Figshare
Dropbox
Service vs Application interface
the service build new applicatoins
this is right in line with SHARE-NS - openning, unlcoking this data
allows for innovation
From day one, we’ve been very excited about the SHARE partnerhship:
community, expertise bringing to the table
as well as
shared interests
mission
Technical perspective, this project fit very much in line with what are building and more importantly how we are building it

19. Put data, materials, and code on the OSF
quite simply, it is a file repository that allows any sort of file to be stored and many types to be rendered in the browser without any special software. This is very important for increasing accessibility to research.

20. Manage access and permissions

21. Automate versioning
With hashes

22. Connects Services
Researchers Use

24. OpenSesame

25. OpenSesame

26. Registration

27. Template Forms

28. Share your work

29. Persistent
Citable
Identifiers

31. SHARE (http://share.osf.io/)
Gather
Notify
Providers
Consumers
So this is what SHARE is doing - taking metadata about research from different kinds of digital repositories, normalizing it, and providing a feed, an API that can be queried, and a database that can be searched.
This was Phase I - the data processing pipeline, workflow for building harvesters, development of the schema. Phase II is cleaning, enhancing, linking.

33. The OSF is one vision of an application framework

34. Toolkit
Ecosystem
Data
OSF

35. Community
Services
Interfaces
Toolkit
Ecosystem
Data
OSF

38. Content Experts Schol Comm Experts Technical Experts
Three layers (let experts be experts, reduce redundancy and cost, accelerate innovation)
Top = content and interfaces -> researchers care
Middle = services -> schol comm innovators care
Bottom = tool-kit -> developers care
Toolkit
Ecosystem
Data
Technical Experts

46. osf.io/registries

47. Content Experts Schol Comm Experts Technical Experts
Three layers (let experts be experts, reduce redundancy and cost, accelerate innovation)
Top = content and interfaces -> researchers care
Middle = services -> schol comm innovators care
Bottom = tool-kit -> developers care
Toolkit
Ecosystem
Data
Technical Experts

48. Connecting, preserving, speeding up, and improving scholarship
These slides are shared at: [Take a picture!]
Thank you!
@EvoMellor