Session 4 Open Workflow: OSF and Pre-registration exercise Good afternoon everyone! My name is Kaitlyn Werner and I am an ambassador for the Center for Open Science. Today I am excited to be here to talk to you about some of the services that the Center for Open Science has to offer, more specifically with respect to the Open Science Framework. As I am sure you all know, this idea of transitioning to a more open psychological science has been quite a hot topic within many fields over the past few years, and so my goal here today is to show you how the Open Science Framework can help you become more productive and efficient researchers.

RT2 Roadmap Design Conduct Dissemination Archiving Motivating Issues Researchers degrees of freedom Scientific misconduct Publication bias Failure to replicate To achieve Open materials, data, code, & access Transparent reporting & disclosure Reproducible & replicable results Cumulative meta-analyses Organized Workflow and File Management (OSF, Github) Design Pre-Registration Pre-Analysis Plans Power Planning Conduct Data Management Version Control Open Notebooks (Jupyter/Docker) Dissemination Transparent Reporting & Disclosure Preprints Open Access Archiving Data Repositories Dynamic Documents

Kaitlyn Werner, Carleton University: I work primarily with multi-level longitudinal data in social, personality, and health psychology.

Outline What is the Center for Open Science? OSF Tutorial Open Science Practices (registration) FAQs

Openness. Integrity. Reproducibility.

What does COS offer? Infrastructure Community Metascience COS has developed a variety of software tools and workflows based on our free Open Science Framework to help researchers manage and archive their research, privately or publicly. Community COS helps connect and build open science communities of researchers, their affiliated institutions, their funders, and the publishers of research outcomes. Metascience COS supports and conducts research on scientific practices. Our team members are thought leaders on open science and our tools serve as critical research platforms.

The Open Science Framework osf.io Project management Easily share files Document project changes Project analytics Archive data Facilitate collaboration Research efficiency Registration

Tutorial A brief, but simple tutorial on how to create a project, and manage files and components

The Open Science Framework osf.io

The Open Science Framework osf.io

OSF: Creating a Project osf.io

OSF: Organizing a Project osf.io/r6yhz

OSF: Uploading Documents osf.io/r6yhz

OSF: Uploading Documents osf.io/r6yhz

The Open Science Framework – Privacy Controls osf.io/r6yhz Can create a link for people to access your project without editing (i.e., it’s read only) Can create an anonymized version for peer review – just make sure all documents are also anonymous!

OSF: General Navigation osf.io

Connect with services you already use! …and more!

Open Science Practices What exactly can you do to increase transparency? A guide to various registration formats on OSF

What are open science practices? Practices that promote transparent research from start to finish 100% p-hacking free Here’s how you can replicate our work Here, check our numbers What does this really mean?

Registration with the Open Science Framework Registration creates a frozen version of the project that can never be edited or deleted but can be withdrawn. Your original project remains editable but will now have the registration linked to it. Open-Ended Registration AsPredicted Pre-registration OSF Standard Registration Special Registration Types Provide a narrative summary of what is contained in the registration, or how it differs from prior registrations. Eight (8) questions based on the criteria of aspredicted.org Data collected? (Y/N) Question or hypothesis Describe key variables Number of participants Primary analyses plan Secondary analyses Sample size determination Additional (e.g., exclusions) Basic registration; Similar to open-ended registration with two extra questions (3 total) Data collected? (Y/N) Looked at data? (Y/N) Additional comments Pre-registration Challenge Replication recipe (pre and post) (Brandt et al., 2013) Pre-registration in social psychology (van’t Veer & Giner-Sorolla, 2013) Dynamic Document: Tracking the Research Process and its Development

Registration Options osf.io/r6yhz

Register: Open-Ended Registration osf.io/r6yhz

Register: As Predicted osf.io/r6yhz

Register: Open-Ended Registration osf.io/r6yhz

Pre-Registration Challenge Pull in example

EXAMPLE: As Predicted Registration osf.io/r6yhz

Dynamic Documents: Documenting Your Workflow As it Happens Create a word document with any updates as they happen – “lab notebook” Use your registration as a template and document changes using bullet points, another colour, etc. The more detailed you are in the registration, the more time you save later Report any deviations from your registration in your manuscript – refer to dynamic document Pull in example Background List a few bullet points explaining your idea; Include key references Research Questions What are the main questions that your research is trying to answer? Hypotheses Operationalize your research questions and list your specific predictions Method Describe your method in as much detail as possible; If using secondary data, describe the sample and variables of interest Exploratory Questions List any questions that novel, you are unsure of, and/or don’t have any specific predictions for Analytic Strategy Describe exclusion criteria, cutoffs, etc. Then, describe exactly how you plan to analyze your hypotheses. Exploratory analyses optional

FAQ Demystifying common misconceptions about open science and pre-registration

Common Concerns Do I have to make my data, materials, etc. public? No. Your registration can be embargoed for up to 4 years, but components can be kept private and shared among (assigned) project contributors What if I have sensitive data? (e.g., special populations, linked data) Consider sharing the non-sensitive portions of the data (e.g., demographics) with a list of missing variables made upon request What if I have a large dataset that I’m not finished using? Consider sharing the variables used in the analyses along with a detailed, complete list of all study variables

Common Concerns What if I can’t share my data, materials, etc.? (e.g., not included in original consent form) Then DON’T. Provide a rationale for why this is the case However, when possible, share data when requested individually (not sharing anything EVER is never a good idea) e.g., Protocol at Psychological Science: There are circumstances in which it is not possible or advisable to share any or all data, materials, or a research plan publicly. For example, there are cases in which sharing participants’ data could violate confidentiality. If you would like your article to include an explanation of such circumstances and/or provide links to any data or materials you have made available—even if not under conditions eligible to earn a badge—you may write an alternative note that will be published in the Open Practices note in the article. Please check this box if you would like your article to include an alternative note and provide the text of the note below:

Take Home Messages Flexibility Security Be honest Do what works best for you! Don’t feel as though you are tied to a specific protocol, analysis, etc. – just keep track of and report any changes that you make. Journals are pretty cool about it, too. Security Open science does not mean no security! There are ways to adapt your open science practices to protect sensitive data and/or cater to your preferences in data sharing. Let it all out or hold a tighter ship – it’s up to you! Just don’t keep anything locked away in your file drawer Be honest No matter what you do or what your preferences are, just be honest and transparent about it! Online supplements (either through a specific journal or OSF) are your friend.

kaitlynwerner.wix.com/research THANK YOU! kaitlyn.werner@carleton.ca @kmwerner osf.io/ftk25 kaitlynwerner.wix.com/research carleton.ca/goallab