Lorne Campbell University of Western Ontario Twitter: @LorneJCampbell Open and Reproducible Sexual Science Link to Slides: https://osf.io/45nfj/ Lorne Campbell University of Western Ontario Twitter: @LorneJCampbell

Open Science Resources A lot has been developed in a short period of time Improving Science in Individual Labs

Why Should Science Be Open? Evaluation requires understanding the methods used, and in what context Reproducibility Replicability See also this talk by Victoria Stodden (2011) on Open Computational Science: https://www.youtube.com/watch?v=sZIxzTsvWhw

Incentives Incentive structure within science has not encouraged open science practices Researchers may therefore feel adopting such practices will put them at a competitive disadvantage But, incentive structure slowly changing And, there are more external pressures to adopt open science practices

Canadian Government Commitment to Open Science Commitment 14: Increase Openness of Federal Science Activities (Open Science) …the Government of Canada wants to build on past work by taking bold steps to make government-funded* science open and transparent to Canadians

The Transparency and Openness Promotion Guidelines (TOP) Published in Science (2015) Eight modular standards, each with three levels of increasing stringency for openness Signatories Over 5000 journals and organizations Many psychology journals are signatories E.g., JESP, JSPR, PR, Perspectives, Social Psychology, PLoS One, Frontiers, Collabra Also: Archives of Sexual Behavior, Current Sexual Health Reports, Sexuality Research and Social Policy, among others

New Submission Policies at Journal of Sex Research

Transformation of the Research Process is Required

The Research Process Ask a Question Review Literature Formulate Hypothesis Design/Run Study Make Inferences Report Results

Report (positive) Results The Research Process: Status Quo Public Ask a Question Review Literature Formulate Hypothesis Design/Run Study Make Inferences Report (positive) Results Private

The Research Process: Open Science Public Ask a Question Review Literature Formulate Hypothesis Design/Run Study Make Inferences Report (all) Results Documenting the Research Workflow

Doing Open Science We receive a lot of training on research methods and statistical procedures (but likely not enough—another talk!) But, not much (if any) on how to do open science (Feynman, 1974, and Lykken, 1991 made similar arguments) Technology today allows for open science practices

Where to Start?

Challenges, not Roadblocks “Well I tell them there’s no problem, only solutions”

The Open Research Process Ask a Question Review Literature Formulate Hypothesis Design/Run Study Make Inferences Report Results Open Notebook

The Open Research Process Pre-registration OSF Aspredicted.org Ask a Question Review Literature Formulate Hypothesis Design/Run Study Make Inferences Report Results Open Notebook

What’s the main question being asked or hypothesis being tested in this study? Describe the key dependent variable(s) specifying how they will be measured. How many conditions will participants be assigned to? Specify exactly which analyses you will conduct to examine the main question/hypothesis. Any secondary analyses? How many observations will be collected or what will determine sample size? Anything else you would like to pre-register? Have any data been collected already from this study?

The Open Research Process Pre-registration OSF Aspredicted.org Registered Reports Ask a Question Review Literature Formulate Hypothesis Design/Run Study Make Inferences Report Results Open Notebook

Over 80 Journals now Offer RRs

The Open Research Process Pre-registration OSF Aspredicted.org Registered Reports Ask a Question Review Literature Formulate Hypothesis Design/Run Study Make Inferences Report Results Register study design, procedure, materials, data analytic plan Open Notebook

The Open Research Process Pre-registration OSF Aspredicted.org Registered Reports Ask a Question Review Literature Formulate Hypothesis Design/Run Study Make Inferences Report Results Register study design, procedure, materials, data analytic plan Open Notebook Data Management Plan

The Open Research Process Confirmatory and Exploratory analyses Pre-registration OSF Aspredicted.org Registered Reports Ask a Question Review Literature Formulate Hypothesis Design/Run Study Make Inferences Report Results Register study design, procedure, materials, data analytic plan Open Notebook Data Management Plan

Dr. A.D. de Groot: The Meaning of “Significance” for Different Types of Research https://pdfs.semanticscholar.org/77e4/c0aea8c39b13d4a7b8532f92964a75eb31cc.pdf Confirmatory Analyses Exploratory Analyses

The Open Research Process Confirmatory and Exploratory analyses Pre-registration OSF Aspredicted.org Registered Reports Ask a Question Review Literature Formulate Hypothesis Design/Run Study Make Inferences Report Results Preprint Server Psyarxiv Socarxiv Register study design, procedure, materials, data analytic plan Open Notebook Data Management Plan

The Open Research Process Confirmatory and Exploratory analyses Pre-registration OSF Aspredicted.org Registered Reports Ask a Question Review Literature Formulate Hypothesis Design/Run Study Make Inferences Report Results Preprint Server Psyarxiv Socarxiv Register study design, procedure, materials, data analytic plan Open Notebook Data Management Plan Journal Traditional Open Access

The Open Research Process Confirmatory and Exploratory analyses Pre-registration OSF Aspredicted.org Share data, Code, and Meta-data Registered Reports Ask a Question Review Literature Formulate Hypothesis Design/Run Study Make Inferences Report Results Preprint Server Psyarxiv Socarxiv Register study design, procedure, materials, data analytic plan Open Notebook Data Management Plan Journal Traditional Open Access

Available Upon Request. Vanpaemel, Vermorgen, & Deriemaecker (2015) Available Upon Request? Vanpaemel, Vermorgen, & Deriemaecker (2015). Are we wasting a good crisis? The availability of Psychological Research Data after the Storm. Collabra (link on image below)

Available Upon Request. Vanpaemel, Vermorgen, & Deriemaecker (2015) Available Upon Request? Vanpaemel, Vermorgen, & Deriemaecker (2015). Are we wasting a good crisis? The availability of Psychological Research Data after the Storm. Collabra (link on image below)

“…we found that half of all published psychology papers that use NHST contained at least one p-value that was inconsistent with its test statistic and degrees of freedom. One in eight papers contained a grossly inconsistent p-value that may have affected the statistical conclusion.” 

25% of authors shared usable code.

But, Sharing Data and Code Does not Guarantee it is Usable Data files can be massive Variable names are not always intuitive How variables are aggregated etc. not always clear Popular statistical software programs cost a lot of money and thus not used by all (e.g., SPSS, SAS) Free software may be perceived as too difficult to learn (e.g., R, Python)

At Minimum… Share annotated analytic code to reproduce results presented Share de-identified data (if possible) used in analyses presented Share relevant meta-data for study variables, data structure, etc.

Complete Reproducibility with Code Ocean

There are Signs of Open Science Progress https://elifesciences.org/articles/16800

Growth of Pre-Registrations at OSF

Open Science Badges

Badges Work

Data Sharing Citation Advantage

Preprint Citation Advantage

Open Access Citation Advantage “…In a logistic regression model, controlling for potential confounders, OA articles compared to non-OA articles remained twice as likely to be cited (odds ratio = 2.1 [1.5–2.9]) in the first 4–10 mo after publication (April 2005), with the odds ratio increasing to 2.9 (1.5–5.5) 10–16 mo after publication (October 2005).” http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.0040157&imageURI=info:doi/10.1371/journal.pbio.0040157.t003

Take Home Message Open Science Practices are Necessary for properly evaluating scientific claims Nullius in verba (Take nobody’s word for it) Technology today allows for engaging in open science practices Funding agencies, journals and societies are beginning to encourage open science practices Incentive structures are changing to reward open science practices So, embrace the future

Let’s not Fool Ourselves Thank you for your time

Supplementary Slides

Key Terms for Using the Open Science Framework (OSF) Types of Pages Project dynamic Registration fixed Can be: Private or Public And can contain: Components Nested within higher order components and projects

Key Terms for Using the Open Science Framework (OSF) Version Control Wiki, commenting Licensing and DOIs Compatibility with other websites and/or apps E.g., Dropbox, Google Drive, Github, Figshare, Dataverse, Box

Example: How My Lab Uses the OSF to Organize our Open Science

Disclosure Statements Study Rationale & Hypotheses Methods, Procedures and Study Scales Data Analytic Plan Participant Recruitment Plan (if applicable) Post-Analytic Discussion

Sharing Analytic Code

Sharing Analytic Code

How to use the OSF 1) create an account: https://osf.io

How to use the OSF 2) Create a new project

New Project: Are we Scared More in the Dark?

Example: Using the Wiki

Example: Using the “Storage”

Example of Version Control

Different Types of “Contributors” Add colleagues for different projects Use “view only” links, including sharing anonymous links for reviewers