1. Lorne Campbell University of Western Ontario Twitter: @LorneJCampbell
Open and Reproducible Sexual Science Link to Slides:
Lorne Campbell
University of Western Ontario

4. Open Science Resources
A lot has been developed in a short period of time
Improving Science in Individual Labs

5. Why Should Science Be Open?
Evaluation requires understanding the methods used, and in what context
Reproducibility
Replicability
See also this talk by Victoria Stodden (2011) on Open Computational Science:

6. Incentives
Incentive structure within science has not encouraged open science practices
Researchers may therefore feel adopting such practices will put them at a competitive disadvantage
But, incentive structure slowly changing
And, there are more external pressures to adopt open science practices

7. Canadian Government Commitment to Open Science
Commitment 14: Increase Openness of Federal Science Activities (Open Science)
…the Government of Canada wants to build on past work by taking bold steps to make government-funded* science open and transparent to Canadians

8. The Transparency and Openness Promotion Guidelines (TOP)
Published in Science (2015)
Eight modular standards, each with three levels of increasing stringency for openness
Signatories
Over 5000 journals and organizations
Many psychology journals are signatories
E.g., JESP, JSPR, PR, Perspectives, Social Psychology, PLoS One, Frontiers, Collabra
Also: Archives of Sexual Behavior, Current Sexual Health Reports, Sexuality Research and Social Policy, among others

9. New Submission Policies at Journal of Sex Research

10. Transformation of the Research Process is Required

11. The Research Process Ask a Question Review Literature
Formulate Hypothesis
Design/Run Study
Make Inferences
Report Results

12. Report (positive) Results
The Research Process:
Status Quo
Public
Ask a Question
Review Literature
Formulate Hypothesis
Design/Run Study
Make Inferences
Report (positive) Results
Private

13. The Research Process: Open Science Public
Ask a Question
Review Literature
Formulate Hypothesis
Design/Run Study
Make Inferences
Report (all)
Results
Documenting the Research Workflow

14. Doing Open Science
We receive a lot of training on research methods and statistical procedures (but likely not enough—another talk!)
But, not much (if any) on how to do open science (Feynman, 1974, and Lykken, 1991 made similar arguments)
Technology today allows for open science practices

15. Where to Start?

16. Challenges, not Roadblocks
“Well I tell them there’s no problem, only solutions”

17. The Open Research Process
Ask a Question
Review Literature
Formulate Hypothesis
Design/Run Study
Make Inferences
Report Results
Open Notebook

18. The Open Research Process
Pre-registration
OSF
Aspredicted.org
Ask a Question
Review Literature
Formulate Hypothesis
Design/Run Study
Make Inferences
Report Results
Open Notebook

20. What’s the main question being asked or hypothesis being tested in this study?
Describe the key dependent variable(s) specifying how they will be measured.
How many conditions will participants be assigned to?
Specify exactly which analyses you will conduct to examine the main question/hypothesis.
Any secondary analyses?
How many observations will be collected or what will determine sample size?
Anything else you would like to pre-register?
Have any data been collected already from this study?

21. The Open Research Process
Pre-registration
OSF
Aspredicted.org
Registered
Reports
Ask a Question
Review Literature
Formulate Hypothesis
Design/Run Study
Make Inferences
Report Results
Open Notebook

22. Over 80 Journals now Offer RRs

23. The Open Research Process
Pre-registration
OSF
Aspredicted.org
Registered
Reports
Ask a Question
Review Literature
Formulate Hypothesis
Design/Run Study
Make Inferences
Report Results
Register study design, procedure, materials, data analytic plan
Open Notebook

25. The Open Research Process
Pre-registration
OSF
Aspredicted.org
Registered
Reports
Ask a Question
Review Literature
Formulate Hypothesis
Design/Run Study
Make Inferences
Report Results
Register study design, procedure, materials, data analytic plan
Open Notebook
Data Management Plan

27. The Open Research Process
Confirmatory and
Exploratory analyses
Pre-registration
OSF
Aspredicted.org
Registered
Reports
Ask a Question
Review Literature
Formulate Hypothesis
Design/Run Study
Make Inferences
Report Results
Register study design, procedure, materials, data analytic plan
Open Notebook
Data Management Plan

28. Dr. A.D. de Groot: The Meaning of “Significance” for Different Types of Research
Confirmatory Analyses
Exploratory Analyses

29. The Open Research Process
Confirmatory and
Exploratory analyses
Pre-registration
OSF
Aspredicted.org
Registered
Reports
Ask a Question
Review Literature
Formulate Hypothesis
Design/Run Study
Make Inferences
Report Results
Preprint Server
Psyarxiv
Socarxiv
Register study design, procedure, materials, data analytic plan
Open Notebook
Data Management Plan

31. The Open Research Process
Confirmatory and
Exploratory analyses
Pre-registration
OSF
Aspredicted.org
Registered
Reports
Ask a Question
Review Literature
Formulate Hypothesis
Design/Run Study
Make Inferences
Report Results
Preprint Server
Psyarxiv
Socarxiv
Register study design, procedure, materials, data analytic plan
Open Notebook
Data Management Plan
Journal
Traditional
Open Access

32. The Open Research Process
Confirmatory and
Exploratory analyses
Pre-registration
OSF
Aspredicted.org
Share data,
Code, and
Meta-data
Registered
Reports
Ask a Question
Review Literature
Formulate Hypothesis
Design/Run Study
Make Inferences
Report Results
Preprint Server
Psyarxiv
Socarxiv
Register study design, procedure, materials, data analytic plan
Open Notebook
Data Management Plan
Journal
Traditional
Open Access

33. Available Upon Request. Vanpaemel, Vermorgen, & Deriemaecker (2015)
Available Upon Request? Vanpaemel, Vermorgen, & Deriemaecker (2015). Are we wasting a good crisis? The availability of Psychological Research Data after the Storm. Collabra (link on image below)

34. Available Upon Request. Vanpaemel, Vermorgen, & Deriemaecker (2015)
Available Upon Request? Vanpaemel, Vermorgen, & Deriemaecker (2015). Are we wasting a good crisis? The availability of Psychological Research Data after the Storm. Collabra (link on image below)

35. “…we found that half of all published psychology papers that use NHST contained at least
one p-value that was inconsistent with its test statistic and degrees of freedom. One in
eight papers contained a grossly inconsistent p-value that may have affected the
statistical conclusion.” 

36. 25% of authors shared usable code.

37. But, Sharing Data and Code Does not Guarantee it is Usable
Data files can be massive
Variable names are not always intuitive
How variables are aggregated etc. not always clear
Popular statistical software programs cost a lot of money and thus not used by all (e.g., SPSS, SAS)
Free software may be perceived as too difficult to learn (e.g., R, Python)

38. At Minimum…
Share annotated analytic code to reproduce results presented
Share de-identified data (if possible) used in analyses presented
Share relevant meta-data for study variables, data structure, etc.

40. Complete Reproducibility with Code Ocean

41. There are Signs of Open Science Progress

42. Growth of Pre-Registrations at OSF

43. Open Science Badges

44. Badges Work

45. Data Sharing Citation Advantage

46. Preprint Citation Advantage

47. Open Access Citation Advantage
“…In a logistic regression model, controlling for potential confounders, OA articles compared to non-OA articles remained twice as likely to be cited (odds ratio = 2.1 [1.5–2.9]) in the first 4–10 mo after publication (April 2005), with the odds ratio increasing to 2.9 (1.5–5.5) 10–16 mo after publication (October 2005).”

48. Take Home Message
Open Science Practices are Necessary for properly evaluating scientific claims
Nullius in verba (Take nobody’s word for it)
Technology today allows for engaging in open science practices
Funding agencies, journals and societies are beginning to encourage open science practices
Incentive structures are changing to reward open science practices
So, embrace the future

49. Let’s not Fool Ourselves
Thank you for your time

50. Supplementary Slides

51. Key Terms for Using the Open Science Framework (OSF)
Types of Pages
Project
dynamic
Registration
fixed
Can be:
Private or Public
And can contain:
Components
Nested within higher order components and projects

52. Key Terms for Using the Open Science Framework (OSF)
Version Control
Wiki, commenting
Licensing and DOIs
Compatibility with other websites and/or apps
E.g., Dropbox, Google Drive, Github, Figshare, Dataverse, Box

53. Example: How My Lab Uses the OSF to Organize our Open Science

56. Disclosure Statements
Study Rationale & Hypotheses
Methods, Procedures and Study Scales
Data Analytic Plan
Participant Recruitment Plan (if applicable)
Post-Analytic Discussion

57. Sharing Analytic Code

58. Sharing Analytic Code

59. How to use the OSF
1) create an account:

60. How to use the OSF
2) Create a new project

61. New Project: Are we Scared More in the Dark?

64. Example: Using the Wiki

65. Example: Using the “Storage”

66. Example of Version Control

67. Different Types of “Contributors”
Add colleagues for different projects
Use “view only” links, including sharing
anonymous links for reviewers