1. Modularity and Interoperability
Jeffrey Spies
COS | SHARE | UVA
@jeffspies
Presented at PASIG, 2016/10/25

2. Mission Increase openness, integrity, and reproducibility of scholarly research.
Non-profit, 50 employees, 26M

3. How we do it Among other things, we create free, open source software.
Non-profit, 50 employees, 26M

5. Reproducibility Project: Psychology http://osf.io/ezcuj
- the 2008 issues of 3 prominent journals (PSCI, JEPLMC, & JPSP)
-The 2008 issue was chosen because it’s recent enough that original authors should still have materials, but old enough to have impact metrics to use as predictors of reproducibility
-Replicators chose a study to replicate from this list.
-In the case of multi-study articles, the last study was the key target of replication unless it was not feasible, in which case the before last study was selected
JEP:LMC
PSCI
JPSP
Open Science Collaboration, 2015

6. 97% 37% xx Open Science Collaboration, 2015, Science Biomedical:
Amgen: 11%, Bayer 21%
Lots of commentary; one of the virtues of open
Open Science Collaboration, 2015, Science

7. Figure by fivethirtyeight.com Silberzahn et al., 2015
29 statisticians, were refs more l ikely to give Red cards
Many decisions some influenced by our
Perspective, Perceptions, motivations, and biases
Knowledge and experience we bring
Garden of forking paths: andrew gelman
Figure by fivethirtyeight.com
Silberzahn et al., 2015

8. The $1,000,000 Preregistration Challenge
Another incentive for researchers to try out preregistration.

9. Registered Reports Design Collect & Analyze Report Publish PEER REVIEW
Review of intro and methods prior to data collection; published regardless of outcome
Beauty vs. accuracy of reporting
Publishing negative results
Conducting replications
Peer review focuses on quality of methods

10. Registered Reports Design Collect & Analyze Report Publish PEER REVIEW
Review of intro and methods prior to data collection; published regardless of outcome
Beauty vs. accuracy of reporting
Publishing negative results
Conducting replications
Peer review focuses on quality of methods

11. Incentives for individual success are focused on getting it published, not getting it right.
Nosek, Spies, & Motyl, 2012

12. mod·u·lar of, relating to, or based on a module
A standard or unit of measure
having parts that can be connected or combined in different ways
Merriam-Webster

13. modularity independent, closed, domain- specific processing modules
compartmentalization and inter-relation of the parts 
logical partitioning…that allows complex software to be manageable for the purpose of implementation and maintenance

14. Modularity fosters specialization. Let experts be experts.

15. Reproducibility Requires Context
The problems we deal with are challenging
That context is disguised by simple, clean narratives
We are human
Biased
Make decisions and interpret results based upon unique experiences
Forgetful
Busy

16. Publish Report Search / Discovery Develop Idea Design Study
Collect Data
Store Data
Analyze Data
Write Report
Incentives drive me get around this cycle as quickly and as often as possible

17. Publish Report Search / Discovery Develop Idea Design Study
Reproducibility
Publish Report
Search / Discovery
Develop Idea
Design Study
Collect Data
Store Data
Analyze Data
Write Report
If you try to append something to that workflow, it’s either going to done with low quality or ignored
There’s integration costs,
Learning costs,
Transition costs
Ask me to open data after publicaiton….

18. We need to preserve the context

19. Publish Report Search / Discovery Develop Idea Design Study
Preservation
Publish Report
Search / Discovery
Develop Idea
Design Study
Collect Data
Store Data
Analyze Data
Write Report
If you try to append something to that workflow, it’s either going to done with low quality or ignored
There’s integration costs,
Learning costs,
Transition costs
Ask me to open data after publicaiton….

20. Reproducibility and preservation must be integrated rather than appended to research workflow

21. OSF Publish Report Search/ Write Discovery Report Analyze Develop Data
Idea
In this case…they should be trying to get around this cycle efficiently! That’s where COS’s flagship product comes in.
Don’t’ focus on gaps
The open science framework is meant to close or fill in these literal gaps between verticals.
If openness is important it has to in this cycle—it has to be a step that either makes the cycle faster or at least doesn’t hurt it.
Meet users where they are
In many ways, the OSF is a workflow management tool—or even workflow repository.
Design
Study
Store Data
Collect
Data

22. Connects Services Researchers Use Getting close to users
Don’t want to reinvent – great stuff
Flexibility – workflow is personal
Abstracted apis
Github: drag and drop
Collaborative editor

23. Getting close to users
Don’t want to reinvent – great stuff
Flexibility – workflow is personal
Abstracted apis
Github: drag and drop
Collaborative editor

24. Adam Summers aka fish guy is ct scanning all the fish.
This is a 3d renderer—anyone can create renderers, by the way, very easy—modular system.
Just write a bit of python or javascript code to do conversion and submit the module on github.

25. Now Nine extensions right now Publish Report Search / Discovery
Develop Idea
Design Study
Collect Data
Store Data
Analyze Data
Write Report
Nine extensions right now
Now

26. Publish Report
Search / Discovery
Develop Idea
Design Study
Collect Data
Store Data
Analyze Data
Write Report
Adding 20 more developed by the community in order to start filling these gaps.
OpenSesame
Soon
29 grants to develop open tools and services:

27. In partnership with Fedora, JHU, and Notre Dame
Publish Report
Search / Discovery
Develop Idea
Design Study
Collect Data
Store Data
Analyze Data
Write Report
In partnership with Fedora, JHU, and Notre Dame

28. Publish Report Write Report Develop Idea Analyze Data Design Study
Preservation
Search / Discovery
Develop Idea
Design Study
Collect Data
Store Data
Analyze Data
Write Report

29. OSF Application Framework
Workflow
Authentication
Permissions
File Storage
File Rendering
Meta-database
Persistence
Integrations
Search
SHARE
osf.io
journals
registries
preprints
grants management
university systems

32. OSF Application Framework
Workflow
Authentication
Permissions
File Storage
File Rendering
Meta-database
Persistence
Integrations
Search
SHARE

34. OSF Application Framework
Workflow
Authentication
Permissions
File Storage
File Rendering
Meta-database
Persistence
Integrations
Search
SHARE
osf.io
journals
registries
preprints
grants management
Preprint server like socarxiv backed by existing preservation systems
Or a preservation group saying, we’ll act as a preservation partner for all biomedical preprints
university systems

35. http://osf.io/b7bdu jeff@cos.io @jeffspies http://cos.io
Find this presentation at
_pasig_spies.pptx
Scholarly workflow integration: The key to increasing reproducibility  and preservation efficacy
@jeffspies