1. Open Science Framework
Jeffrey Spies
Center for Open Science | University of Virginia
@jeffspies

2. Wait

3. CENTRE
Non-profit Charlottesville VA

4. Mission Increase openness, integrity, and reproducibility of scientific research.
$25 since 2013
41 full time staff, 36 interns
Most of the staff are on the technical team
We do community and metascience work—all supported by the tech
Everything we do is free, open source

5. Problem The gap between scholarly values and practices.
Could talk a lot about this

6. Incentives for individual success are focused on getting it published, not getting it right.
Nosek, Spies, & Motyl, 2012

7. Key incentive = Publication
What is published?
What is not?
Novel results Positive results Clean results
Replications Negative/Nulls Mixed evidence
But this isn’t how science works!
And this causes a lot of problem
I already have biases, now I have these incentvies

8. Norms Counternorms Communality Secrecy Open sharing Closed
Anderson et al.

9. Norms Counternorms Communality Universalism Secrecy Particularlism
Open sharing
Universalism
Evaluate research on own merit
Secrecy
Closed
Particularlism
Evaluate research by reputation

10. Norms Counternorms Communality Universalism Disinterestedness Secrecy
Open sharing
Universalism
Evaluate research on own merit
Disinterestedness
Motivated by knowledge and discovery
Secrecy
Closed
Particularlism
Evaluate research by reputation
Self-interestedness
Treat science as a competition

11. Norms Counternorms Communality Universalism Disinterestedness
Open sharing
Universalism
Evaluate research on own merit
Disinterestedness
Motivated by knowledge and discovery
Organized skepticism
Consider all new evidence, even against one’s prior work
Secrecy
Closed
Particularism
Evaluate research by reputation
Self-interestedness
Treat science as a competition
Organized dogmatism
Invest career promoting one’s own theories, findings

12. Norms Counternorms Communality Universalism Disinterestedness
Open sharing
Universalism
Evaluate research on own merit
Disinterestedness
Motivated by knowledge and discovery
Organized skepticism
Consider all new evidence, even against one’s prior work
Quality
Secrecy
Closed
Particularism
Evaluate research by reputation
Self-interestedness
Treat science as a competition
Organized dogmatism
Invest career promoting one’s own theories, findings
Quantity
Then they asked researchers to endorse either the norm or the counter norm. Here’s what they found….

13. Anderson, Martinson, & DeVries, 2007
When asking them what they subscribed to—what they believed—the majority endorsed more norms than counter norms
Anderson, Martinson, & DeVries, 2007

14. Anderson, Martinson, & DeVries, 2007
However, when asked how they behaved, the story changed a bit—more people endorsed the same number of counter norms as norms or even more counter norms. And then the kicker…
Anderson, Martinson, & DeVries, 2007

15. Anderson, Martinson, & DeVries, 2007
So what are researchers supposed to do?
Psychology to back this up: motivated reasoning, minimal accountability, people are busy.
Anderson, Martinson, & DeVries, 2007

16. Problem The gap between scholarly values and practices.
Again, back to this core problem. What’s the solution?

17. Solution Openness.
This solves a lot of problems.

18. Openness Increases process transparency Increases accountability
Facilitates reproducibility
Facilitates metascience
Fosters collaboration
Fosters inclusivity
Fosters innovation
Protects against lock-in
Open Content + APIs + Open Source Services

19. Openness is a means to increase research quality and efficiency.
All this is great, but it’s in service of a broader mission.

20. Solution Technology can operationalize and incentivize scholarly values.
And why COS is a technology company because we believe we can use technology to operationalize and incentivize scholarly values and that will increase research quality and efficiency.

21. Openness is the solution; Workflow makes it practical.
Technology facilitates
Workflow that makes it practical.

22. Publish Report
Search/Discovery
Develop Idea
Design Study
Collect Data
Store Data
Analyze Data
Write Report
Researchers find their day to day efficiency—mostly in getting published as quickly as possible—via the success of their workflow.

23. Publish Report
Search/Discovery
Develop Idea
Design Study
Collect Data
Store Data
Analyze Data
Write Report
If you try to append something to that workflow, it’s either going to be ignored or done with low quality.
If you ask me to make my data open, I’ll say sure, but after my next paper is published.

24. Let experts be experts.
Efficiency and quality gain comes from try to allow experts to be experts. You’ll see this repeated.

25. Scholars should focus on scholarship.
In this case…they should be trying to get around this cycle efficiently! That’s where COS’s flagship product comes in.

26. OSF Literal gaps Close or fill these
Publish Report
Search/Discovery
Develop Idea
Design Study
Collect Data
Store Data
Analyze Data
Write Report
OSF
Literal gaps
Close or fill these
Openness must make the cycle faster or at least doesn’t hurt it.
OSF is a workflow management tool—or even workflow repository.

27. OSF Application Framework
Workflow
Authentication
Permissions
File Storage
File Rendering
Meta-database
Persistence
Integrations
Search
SHARE
osf.io
journals
registries
preprint servers
grants management
university systems
But it’s a bit more.
Framework that osf.io is just one view of—and it builds on this engine that does things like authentication and file storage….
Let’s talk about that view.

28.
The OSF is a free, open source service—you can register right now

29. Collaboration Documentation Archiving CMS
Balances things researchers care about while either automatically or making it very easy to align with values.
Scholars should focus on scholarship--and we respect that they have a workflow—this has to be frictionless.

30. So here, it’s very easy to curate a project, add contributors, share, but we also keep an activity log.
To the users that’s either meaningless or an easy way to keep up-to-date, but to us, this is provenance
this is important information about how science happens—science is a process, this is great time series metadata for metascience

31. Version Control
The same goes for version control

32. To the user, it’s easy backup—drag and drop version control.
For the ideals, it’s important provenance.

33. Other Features Granular permissions Analytics dashboards Forking
Persistent, citable identifiers
Persistent content
Project snapshotting (i.e., registration)

34. Connects Services Researchers Use
Integrations: we want to meet users where they are.
Workflow is very custom—we can’t please everyone—so we try to connect to many services.
We also don’t want to reinvent.
Flexible to many workflows.

35. API Abstractions
Now you can drag and drop to any of 8 storage services (including OSF)
Collaborative editor

36. This is a 3d renderer—anyone can create renderers, by the way, very easy—modular system.
Just write a bit of python or javascript code to do conversion and submit the module on github.

37. Now Nine extensions right now Publish Report Search/Discovery
Develop Idea
Design Study
Collect Data
Store Data
Analyze Data
Write Report
Nine extensions right now
Now

38. Publish Report
Search/Discovery
Develop Idea
Design Study
Collect Data
Store Data
Analyze Data
Write Report
OpenSesame
Adding 19 more developed by the community in order to start filling these gaps.
Getting close to users…
Soon
29 grants to develop open tools and services:

39. http://osf.nd.edu …Getting close to users
Institutional offering: ND, UCR, USC
Single sign on
Dashboards
Affiliated Projects

40. Let experts be experts.
We apply this motto to scientists, but also when it comes to building infrastructure.

41. OSF Application Framework
Workflow
Authentication
Permissions
File Storage
File Rendering
Meta-database
Persistence
Integrations
Search
SHARE
osf.io
journals
registries
preprint servers
grants management
university systems
Showed you osf.io, let’s talk about other views

42.
Preregistration: biases affect decision process, so state them in advance before the data in front of you

43. Don’t have to care about projects
You create a preregistration
Title, name is grabbed from registration, research questions…

44. Go further in the process, asks for scripts

45. http://osf.io/prereg Education conference 4 questions…
So start a new preregistration

46. Project is created in the background

47. Login, projects, users, metadata, files
What that society wanted is all here: Remove branding

48. Society for the Study of Autism Registry
Because we have no monetization strategy, this is great!
We don’t need eyes on page, we have expertise in the tech, they have expertise in their community

49. Society for the Study of... </title>
<menu>
<title>
Society for the Study of...
</title>
<osf-search /> <osf-login />
</menu>
<society-questionnaire>
Author: <osf-user name />
Question 1: <osf-metadata />
Analysis code:
<osf-file upload />
</society-questionnaire>
And here’s the code
Anyone can create these views
Render that file? <osf-file render>

50. OSF Application Framework
Workflow
Authentication
Permissions
File Storage
File Rendering
Meta-database
Persistence
Integrations
Search
SHARE
osf.io
journals
registries
preprint servers
grants management
university systems
Back to mantra…

51. Let experts be experts.
This should extend to scholarly tool development

52. OSF Application Framework
Workflow
Authentication
Permissions
File Storage
File Rendering
Meta-database
Persistence
Integrations
Search
SHARE
registries
Fund both?

53. registries
Or just the frontend—working with the community, doing outreach, usability testing, etc.

54. OSF Application Framework
Workflow
Authentication
Permissions
File Storage
File Rendering
Meta-database
Persistence
Integrations
Search
SHARE
preprint servers
Same: Fund both?

55. preprint servers
Or working with the community, doing outreach, usability testing, etc.

56. preprint servers
Coming soon:

57. OSF Application Framework
Workflow
Authentication
Permissions
File Storage
File Rendering
Meta-database
Persistence
Integrations
Search
SHARE
osf.io
journals
registries
preprint servers
grants management
university systems
This is the power of a framework approach on an open platform

58. Substantive experts shouldn’t deal with…
Workflow integration
Authentication
Permissions
File storage
File rendering
Database
Metadata/Annotations/Commenting
Persistence
External service integrations
Search
SHARE Data

59. …or scaling these: Workflow integration Authentication Permissions
File storage
File rendering
Database
Metadata/Annotations/Commenting
Persistence
External service integrations
Search
SHARE Data

60. @jeffspies | jeff@cos.io
Find this presentation at
@jeffspies |