Translating Open Science into Daily Practice The Open Science Framework Katherine S. Corker Kenyon College Ambassador, Center for Open Science http://cos.io | http://osf.io Many scientific behaviors are habitual – throughout our training we develop certain habits, and these habits tend to stick with us throughout our careers. We’re realizing now that some of these habits have gotten us in to trouble. Some are a product of their time – like not worrying too much about data archival in a pre-digital age – but others are a product of a broken incentive structure that prioritizes getting science published over getting it right. The goal of my presentation is to give you a few tools that can help you develop new habits that will ultimately save you time and effort, while simultaneously incorporating scientific best practices that will help you to do higher quality science too.

SCIENCE! Publish report Search and discover Develop idea Design study Acquire materials Collect data Store data Analyze Data Interpret findings Write report SCIENCE! The scientific method involves many steps – and as technology advances, each of these steps has become more complicated.

A one-stop resource for scientists OSF Publish report Search and discover Develop idea Design study Acquire materials Collect data Store data Analyze Data Interpret findings Write report The OSF is a one-stop resource for scientists, and it helps simplify each part of the scientific workflow.

The Open Science Framework Free, open-source software that manages projects at all stages of the research lifecycle and connects the tools researchers use The OSF is a free and open-source collaboration and project management tool that is also a registry and repository. The OSF is designed by researchers, for researchers - to manage research projects at all stages of the research lifecycle - from planning, to execution, to reporting, to archiving, and to discovery. Let me give you just a taste of the functionality at each of these five major stages of the research process. Planning Execution Reporting Archiving Discovery

The Open Science Framework Collaboration Collaboration: The OSF is organized around “projects.” A researcher can fully customize a project to use the OSF as a living lab notebook. Planning Execution Reporting Archiving Discovery

The Open Science Framework Collaboration Documentation Documentation: Each team member’s contribution is credited and documented. There is also a time-stamped record of everything related to your project. Planning Execution Reporting Archiving Discovery

The Open Science Framework Collaboration Documentation Pre-registration Pre-registration: The OSF also has a robust pre-registration feature that I’d like to dive into later on in the talk. Planning Execution Reporting Archiving Discovery

Planning Execution Reporting Archiving Discovery Here’s what a project page might look like. The project’s overall contributors are acknowledged here (beneath title), and the timestamped activity record is here (lower right). The wiki (middle left) allows collaborative editing, so that team members can document the status of a project as it occurs. Each component (middle right) can have its own set of authors separate from the main project. Let’s take a look at a few of the other key features for planning and executing your projects. Planning Execution Reporting Archiving Discovery

Merges Public-Private Workflows

You can even generate anonymous links to send out for peer review, so that reviewers can examine your supplemental materials without compromising the anonymity of the process.

Incentives for Openness File downloads We can do small things like offer rich analytics to incentivize more open practices. Public projects gain immediate access to analytics showing visits over time, sources of traffic, geographic location of traffic, time of traffic, and download counts for files. This is a much faster reward for one’s effort than waiting months or longer until something is published and then even longer until it is cited.

File downloads We can do small things like offer rich analytics to incentivize more open practices. Public projects gain immediate access to analytics showing visits over time, sources of traffic, geographic location of traffic, time of traffic, and download counts for files. This is a much faster reward for one’s effort than waiting months or longer until something is published and then even longer until it is cited. Forks

Persistent Citable Identifiers DOIs available for public registrations

DOIs available for public registrations

The Open Science Framework Collaborative wiki can be used to jointly work on projects OSF connects to services Free and Open-source -- the OSF is a collaboration and project management tool that is also a registry and repository. The OSF is designed by researchers, for researchers - to manage research projects at all stages of the research lifecycle - from planning, to execution, to reporting, to archiving, and to discovery. The rest of this talk will take you through these stages of the research lifecycle, demonstrating the OSF’s present functionality and giving a taste of what’s to come. Planning Execution Reporting Archiving Discovery

The OSF Connects Services You Already Use.

Version Control The OSF helps you keep track of your manuscript as you write and revise it with built in version control.

You can recover previous versions of a document easily You can recover previous versions of a document easily. This also increases transparency for the community, provided the documents are public. Everyone can see exactly what reviewer 2 made you do from draft 1 to draft 2.

The Open Science Framework The OSF is an archive and a repository (each file up to 5 gb) The OSF connects other repositories (e.g. Dataverse) When you’ve finished your research project, you’ll likely want to archive the data and materials. There are a few ways the OSF supports this. One is by being an archive itself. If there isn’t a domain-specific repository for your field, the OSF itself serves as an archive. Because of a preservation endowment, you can be sure your data won’t go anywhere. New users can cite your data, reuse it, ask you to collaborate, and easily incorporate your data into meta-analyses. No more file drawer! Planning Execution Reporting Archiving Discovery

The Open Science Framework Search Projects Researchers Datasets OSF for meetings The OSF has a robust search feature that makes discovering content on the site easy. Your public pages are also indexed in search engines so other researchers can find and access your work. OSF for meetings is another feature that societies can use to archive posters, talks, and more in a coordinated way. Planning Execution Reporting Archiving Discovery

Examples 1 2 Lab management, communication, and archiving Pre-registration: Varieties and options

Simple Lab Management System 1 Create a table in Google doc listing projects Important to note – first – that inspiration for this solution came from Lorne Campbell. It’s a really simple, elegant solution, and I’ve taken Lorne’s basic idea and modified it to suit my own lab’s needs. Here’s the steps: 1. Create a table in Google doc listing your projects: (A link to my table is in the handout and OSF page for this seminar. You can copy that file and use it yourself if you want. The link to Lorne’s table is in there too. He uses Sheets instead of Docs.) I include both published works for archival purposes and in progress works as well. I have fields for title, team members, year of pub, and the stage of the project. Most importantly, I have spots for OSF links to the data, code, materials, and pre-registration (if applicable). It’s a subtle prompt to me and my team to both stay organized and to work completely

Simple Lab Management System 1. Create a table in Google doc listing your projects: (A link to my table is in the handout and OSF page for this seminar. You can copy that file and use it yourself if you want. The link to Lorne’s table is in there too. He uses Sheets instead of Docs.) I include both published works for archival purposes and in progress works as well. I have fields for title, team members, year of pub, and the stage of the project. Most importantly, I have spots for OSF links to the data, code, materials, and pre-registration (if applicable). It’s a subtle prompt to me and my team to both stay organized. The table can serve as a checklist of sorts.

Simple Lab Management System 1 Create a table in Google doc listing projects 2 Create OSF project with Google file storage 2. Create OSF project with Google file storage

Simple Lab Management System 2. Create OSF project with Google file storage

Simple Lab Management System 2. Create OSF project with Google file storage

Simple Lab Management System 1 Create a table in Google doc listing projects 2 Create OSF project with Google file storage 3. Put your doc in the linked folder. Whenever you make changes to the sheet, they show up in OSF too. You can give your RAs and grad students editing privileges, which means everyone on the team can update the status of projects, if you want. It’s a central organizing document for the lab. It also implicitly nudges the team to think about availability of different components of the research process. If you wanted to, you could use this same procedure to create an electronic lab notebook. Perhaps you’d have one document per day that you add to a linked folder. It’s a simple way to make your whole process open and transparent. Or you could use the system for internal documentation only. 3 Put your doc in the linked folder

Simple Lab Management System 3. Put your doc in the linked folder. Whenever you make changes to the sheet, they show up in OSF too. You can give your RAs and grad students editing privileges, which means everyone on the team can update the status of projects, if you want. It’s a central organizing document for the lab. It also implicitly nudges the team to think about availability of different components of the research process. If you wanted to, you could use this same procedure to create an electronic lab notebook. Perhaps you’d have one document per day that you add to a linked folder. It’s a simple way to make your whole process open and transparent. Or you could use the system for internal documentation only.

Simple Lab Management System 3. Put your doc in the linked folder. Whenever you make changes to the sheet, they show up in OSF too. You can give your RAs and grad students editing privileges, which means everyone on the team can update the status of projects, if you want. It’s a central organizing document for the lab. It also implicitly nudges the team to think about availability of different components of the research process. If you wanted to, you could use this same procedure to create an electronic lab notebook. Perhaps you’d have one document per day that you add to a linked folder. It’s a simple way to make your whole process open and transparent. Or you could use the system for internal documentation only.

Commitment to Specific Hypotheses & Analyses Prior to Data Collection Pre-Registration Commitment to Specific Hypotheses & Analyses Prior to Data Collection Reduce HARKing Increase p value meaning Improve Design Lower researcher df Prevent post-hoc rationalization In terms of the scientific workflow, to me, pre-registration deserves to be discussed right alongside things like lab organization and documentation. You’ll notice that it’s one of the key columns in the table I just showed you. The definition of pre-registration appears on the slide, together with a variety of benefits of practicing it.

Flexible, data-dependent Pre-Registration Exploratory Flexible, data-dependent Confirmatory Constrained, a priori Does all work need to be pre-registered? No. Does pre-registration of work prevent you from fully exploring the data later? No. But it does keep you honest Research can be considered on a dimension from exploratory to confirmatory. Pre-registration enables confirmatory hypothesis testing, which allows researchers to have a higher level of confidence in their findings, but there is lots of value in exploratory research. We can think of exploratory and confirmatory as different parts of the research cycle, in which researchers develop hypotheses in the exploratory phase and become more confident in the veracity of those hypotheses in the confirmatory phase. You could try out pre-registration internally before going fully public, if you wanted. My experience was that doing a pre-registration made me much more thoughtful about my study design and ultimately improved the study overall (independent of the benefits of going public with a commitment to my research plan). See De Groot (1956): http://www.ejwagenmakers.com/inpress/DeGroot1956_TA.pdf Chris Chambers: https://osf.io/2gesj/

Options for Pre-Registration 1 Fully internal (non-public) 2 Aspredicted.org There are lots of choices that you have when pre-registering your work. You could opt to pre-register just for your own internal benefit. As I said before, going through the exercise of pre-registering can be valuable even if you don’t intend to share the pre-registration with others. A really simple option is aspredicted.org, which I’ll show you in a moment. The major benefit of aspredicted is it’s simplicity and ease of use. A researcher can get a timestamped confirmation of their intention to test a particular hypothesis using a particular analysis strategy. However, registrations on this site have the option to stay private indefinitely, meaning this isn’t a great solution for science on the whole. The reason that registries, such as the registries that biomedical researchers use to log clinical trials, need to be public is so that (1) publication bias can be assessed, and (2) to prevent fraudulent duplicate registrations. In my mind, the first issue is a lot more pressing than the second. In order to have a chance at all to measure publication bias, we need to have a better handle on how many studies are actually being done. A third option is to use OSF, which as Erica showed you is not so complex once you start using it. I wanted to note that you could use OSF for a fully internal, non-public option if you wanted. as I understand it, currently registrations can be embargoed for up to 4 years at which point they become public. However, if someone wants to use OSF only for internal, private documentation, they should use other functions of the site (e.g., posting documents which will be timestamped), but not use the "registration" function.   3 Open Science Framework

Here’s what the aspredicted.org pre-registration looks like. We can also see what it looks like on the web: https://aspredicted.org/create.php It’s a super simple interface – the responder just answers a few questions (nine at the moment), and the .pdf of the pre-registration is automatically generated. Responses can stay private forever if desired, but as a field-wide solution this is probably not such a good thing. For biomedical registries, it’s a standard requirement that eventually pre-registrations become public to help combat file drawer problems.

The Open Science Framework And here’s how it looks when you are filling out the registration recipe on OSF. This is Michael Kraus’ pre-registered replication of his own work. The direct link is: https://osf.io/tbw3n/register/564d31db8c5e4a7c9694b2c2

COS Pre-Reg Challenge cos.io/prereg Don’t have time to go into great detail, but just wanted to mention that the Center for Open Science currently has a pre-reg incentive going on at the moment, in which 1,000 researchers will each win $1,000 for publishing their pre-registered research. Details are at cos.io/prereg (CHECK ON LINK).

Thank You! Slides: https://osf.io/7hr8z/ Email: k.corker@gmail.com Center for Open Science http://cos.io | http://osf.io