A Framework for Managing and Sharing Research Workflow Jeffrey Spies Center for Open Science University of Virginia

Non-profit tech company in Charlottesville, VA Openness, integrity, and reproducibility $25 since 2013 I’ll introduce you to the team as I tell you more

If I disconnect email jeff@cos If I disconnect email jeff@cos.io Please do not let me talk to myself for 40 minutes.

Mission Increase openness, integrity, and reproducibility of scientific research.

Problem The gap between scholarly values and practices.

Adelson, 1995

Local contract Shadows have soft edges x junctions Adelson, 1995

Incentives for individual success are focused on getting it published, not getting it right. Novel, positive, clean stories with confirmatory successful narratives Now how science works--arrogant Nosek, Spies, & Motyl, 2012

Key incentive = Publication What is published? What is not? Novel results Positive results Clean results Replications Negative/Nulls Mixed evidence Arrogant

Norms Counternorms Communality Open sharing Secrecy Closed

Norms Counternorms Communality Secrecy Particularlism Universalism Open sharing Universalism Evaluate research on own merit Secrecy Closed Particularlism Evaluate research by reputation

Norms Counternorms Communality Secrecy Particularlism Universalism Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition

Norms Counternorms Communality Secrecy Particularlism Universalism Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings

Norms Counternorms Communality Secrecy Particularlism Universalism Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity

Anderson, Martinson, & DeVries, 2007

Anderson, Martinson, & DeVries, 2007

Anderson, Martinson, & DeVries, 2007

Problem The gap between scholarly values and practices.

Solution Openness.

Openness is a means to increase research quality and efficiency.

Solution Technology can operationalize and incentivize scholarly values.

Let experts be experts.

Openness is the solution; Workflow makes it practical.

Publish Report Search/Discovery Develop Idea Design Study Collect Data Store Data Analyze Data Write Report

Publish Report Search/Discovery Develop Idea Design Study Collect Data Store Data Analyze Data Write Report

Scholars should focus on scholarship.

OSF Publish Report Search/Discovery Develop Idea Design Study Collect Data Store Data Analyze Data Write Report OSF

OSF Application Framework Workflow Authentication Permissions File Storage File Rendering Meta-database Persistence Integrations Search SHARE osf.io journals registries preprint servers grants management university systems

Free, login -> http://osf.io

Collaboration Documentation Archiving

Projects, Collaborators, sharing, Wikis Provenance with logs Curating content Makes collaboration easier—email based management is messy

Version Control

Provenance is important Most people have some form Cumbersome error prone Automatic – easy backup

Connects Services Researchers Use Getting close to users Meet them where they are Don’t want to reinvent when we don’t have to Flexible to many workflows

Drag and drop Collaborative editor

OSF Now 9 Publish Report Search/Discovery Develop Idea Design Study Collect Data Store Data Analyze Data Write Report OSF 9 Now

Publish Report Search/Discovery Develop Idea Design Study Collect Data Store Data Analyze Data Write Report OSF 9 OpenSesame Soon 29 grants to develop open tools and services: https://cos.io/pr/2015-09-24/

Coming soon http://osf.nd.edu/

Let experts be experts.

OSF Application Framework Workflow Authentication Permissions File Storage File Rendering Meta-database Persistence Integrations Search SHARE osf.io journals registries preprint servers grants management university systems

Adam Summers aka fish guy

<menu> <osf-search /> <osf-login /> </menu> <osf-file> <osf-file title /> <osf-file authors /> <osf-file date /> <osf-file renderer /> <osf-file analytics /> <osf-file comments /> <osf-file download /> <osf-file summary /> <osf-file tags /> </osf-file>

<menu> <osf-search /> <osf-login /> </menu> <osf-file> <osf-file title /> <osf-file authors /> <osf-file date /> <osf-file renderer /> <osf-file analytics /> <osf-file comments /> <osf-file download /> <osf-file summary /> <osf-file tags /> </osf-file>

<menu> <osf-search /> <osf-login /> </menu> <osf-file> <osf-file title /> <osf-file authors /> <osf-file date /> <osf-file renderer /> <osf-file analytics /> <osf-file comments /> <osf-file download /> <osf-file summary /> <osf-file tags /> </osf-file>

<menu> <osf-search /> <osf-login /> </menu> <osf-file> <osf-file title /> <osf-file authors /> <osf-file date /> <osf-file renderer /> <osf-file analytics /> <osf-file comments /> <osf-file download /> <osf-file summary /> <osf-file tags /> </osf-file>

<menu> <osf-search /> <osf-login /> </menu> <osf-file> <osf-file title /> <osf-file authors /> <osf-file date /> <osf-file renderer /> <osf-file analytics /> <osf-file comments /> <osf-file download /> <osf-file summary /> <osf-file tags /> </osf-file>

<menu> <osf-search /> <osf-login /> </menu> <osf-file> <osf-file title /> <osf-file authors /> <osf-file date /> <osf-file renderer /> <osf-file analytics /> <osf-file comments /> <osf-file download /> <osf-file summary /> <osf-file tags /> </osf-file>

OSF Application Framework Workflow Authentication Permissions File Storage File Rendering Meta-database Persistence Integrations Search SHARE osf.io journals registries preprint servers grants management university systems

http://osf.io/prereg Authenticate Started a new prereg Motor control and autism

Go further in the process, asks for scripts

Login, projects, users, metadata, files Remove branding

Society for the Study of Autism Registry

<menu> <title> Society for the Study of... </title> <osf-search /> <osf-login /> </menu> <society-questionnaire> Author: <osf-user name /> Question 1: <osf-metadata /> Analysis code: <osf-file upload /> </society-questionnaire>

OSF Application Framework Workflow Authentication Permissions File Storage File Rendering Meta-database Persistence Integrations Search SHARE osf.io journals registries preprint servers grants management university systems

Awaiting Instruction…

Awaiting Instruction…

Let experts be experts.

OSF Application Framework Workflow Authentication Permissions File Storage File Rendering Meta-database Persistence Integrations Search SHARE registries

registries

OSF Application Framework Workflow Authentication Permissions File Storage File Rendering Meta-database Persistence Integrations Search SHARE osf.io journals registries preprint servers grants management university systems

Substantive experts shouldn’t deal with… Workflow integration Authentication Permissions File storage File rendering Database Metadata/Annotations/Commenting Persistence External service integrations Search SHARE Data

…or scaling these: Workflow integration Authentication Permissions File storage File rendering Database Metadata/Annotations/Commenting Persistence External service integrations Search SHARE Data

@osframework, @jeffspies jeff@cos.io Find this presentation at http://osf.io/rvtfk 20160607_iwsg_spies.pptx A Framework for Managing and Sharing Research Workflow @osframework, @jeffspies jeff@cos.io