1. Crisis Schmeisis: Introduction to the Open Science Framework for Scholarship and Teaching
Jon Grahe
In all my affiliations, there are projects that utilize the OSF. At PLU, my scholarship is housed there; At JSP, authors use it to apply for Open Science Badges; At Psi Chi, the Research Advisory Committee uses it to compile measurement scales for members to use.

2. Outline What Crisis and Who Cares? Review Tools Overall
Transparency and Openness Promotion (ToP) Guidelines
Practice: Next Research/Class Project
Review Open Science Framework (OSF) features
Recognize ways to employ the OSF
for personal scholarship
for scholarship in the classroom
Though there are only two goals, there are many benefits.

3. Crisis Schmeisis Festering Concerns about Methods
Since ??
Demonstration of Doubt
Ioannidis (2005)
Year of the Crisis (2011)
The Rebirth of Scientific Methods (2012)
Psychology Leads through Introspection

4. Open Science Framework
Responding to Crisis
Replication Debate -- Publication Debate
Perspectives on Psychological Science special issues
Social Media -- nasty discourse and new spaces
Open Invitation Meta-Science Projects
Center for Open Science and The Many Lab and
Psi Chi and the Next International Collaborative Exchange
New Publication Standards and Expectations
Transparency and Openness in Science
Confidence Intervals, Effect Sizes
“the New Statistics”
Center for Open Science
Open Science Framework

5. Who Cares? Assumptions of Inferential Statistics
Exploration vs. Confirmation
Researcher Degrees of Freedom
Questionable Research Practices
Simmons, J. P., Nelson, L. N., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22,
Choosing from DVs
Choosing sample size
Using covariates,
Reporting Subsets of experimental conditions
Alpha = .05, and therefore p = .05 assuming a random distribution. When we interference with randomness, that raises the alpha level.

6. Science Requires Transparency
Research Degrees of Freedom lead to
Questionable Research Practices
Scientific Evidence is not Immune to
Experimenter Bias
Expectancy Effects
Motivated Reasoning
Double Blind Designs not enough

7. Open Science as Antidote
Confirmatory Tests require forward planning
Preregistration and the “Pre-reg Challenge”
Replication requires access to materials
Open Materials
Reproducibility requires data reanalysis
Open Data
Open Transparency Badges and the Transparency and Openness in Publishing Guidelines (ToP guidelines)

8. Next Research/Class Project
Next Step in Research Agenda
Research Methods or Capstone Project
Group Class Project
Artistic Endeavor
Book for Scholarly or Public Audience

9. Exploratory Pre-registration
Date Stamping Ideas and Plans
Reduce Researcher Degrees of Freedom
Permanent Ink or Etched in Stone
Methods of Pre-registration
Open Science Framework
If there are 32 reseacher degrees of freedom. How many do you want to? Or are you able to claim from your Confirmatory Meta-Science Project?

10. Many Options

11. Preregistration as Protection
Confirmatory Meta-Science Step 2
Preregister your confirmatory meta-science project now to avoid anyone from stealing your idea!

12. Preregistration Challenge Template
Study Information
Title
Authorship
Research Questions
Hypotheses
Sampling Plan
Existing data
Explanation of existing data
Data collection procedures.
Sample size
Sample size rationale
Stopping rule
Variables
Manipulated variables
Measured variables
Indices
Design Plan
Study type.
Blinding.
Study design.
Randomization.
Analysis Plan
Statistical models
Transformations
Follow-up analyses
Inference criteria
Data exclusion
Missing data
Exploratory analysis (optional)
Script (Optional)
Analysis scripts (Optional)
Other
Handout worksheets

13. Personal Benefit from Open Science
Free, Easy to Access, Easy to Share, Permanent Storage of Research Materials.
Mechanism to disseminate Meta-Science
Does not apply to 
Researchers without collaborators
Researchers with only one computer
Researchers with specified and regularly updated back-up storage plan
Others?

14. Open Science Framework— Host your Meta-science Project
The OSF is a open-sourced computer software platform written and managed by the Center for Open Science (
OSF Project like an electronic file cabinet
With an interactive cover page;
unlimited files and folders;
That can be shared easily with some, all, or none
Imagine an electronic file cabinet; it is such a great metaphor. This file cabinet has drawers (called components on the OSF) and folders within those drawers. The entire system can be completely private, shared with a few, or shared for anyone who can find the link. Instead of a bland metal cabinet, the exterior contains details about which drawers to open, who contributed the work, and so much more.

15. The Open Science Framework as electronic file cabinet
Editable Wiki
Version Control
Drag & Drop
Commenting
Record Contributions
Motivated Support
ANYONE
Some People
Only Me

16. Open Science Framework Features
Internet page for scholarship (WIKI screen)
Components
Sharing with Contributors
Sharing with Public
File and Version Control
Connecting with add-ons (e.g. Dropbox)
Enabling Comments from contributors/public
These are the basic features (some are pretty valuable), and there is so much more than what I will cover in this slide show. However, these slides provide enough instruction so that researchers can employ the OSF for their personal scholarship or for purpose of developing a student researcher.

17. Center for Open Science OSF Curriculum
Center for Open Science OSF Curriculum
Introduction
Reproducibility, OSF and Reproducibility
Project setup
Creating, Contributors, Wiki, Components, Add Ons
Documentation
Registration, File and Version Control
Sharing
Contributors & Public Privacy,
Increasing Discoverability with tags
Conclusion
Take-aways, How to get more information
The Center for Open Science is a wonderful organization and they developed the OSF. They have their own curriculum. While this slide show breaks the process into very simple components, there are more details in these links

18. Other Resources CREP Instructions and Workflow Procedures
Collaborative Replications and Education Project
CREP Instructions and Workflow Procedures
Build a basic project for students.
The CREP Step-by-Step provides instructions on how to build a project (
This slide show ( is located on my Crisis Schmeisis Tour Page
The Collaborative Replications and Education Project developed resources to teach young researchers in our large scale collaborative research project where students replicate students as part of their learning experiences and those findings are collated into an estimate of the reproducibility of a given manuscript.
Of course, this slide show is intended to be a resource and when I give this talk, this is a very funny joke

19. Applying OSF features to Classroom
Train students to be transparent scientists and …
Student Projects are Research
Instructors are collaborators
Forked projects provide scaffolding
(copied from original)
Student Projects could be Shared
Comment features allow for direct, monitored peer-review feedback or other constituents
Capstone projects should be public
I am going to primarily use learning related examples in this slide show. I think of students as future scientists or consumers of science. In either case, employing the OSF as an electronic file cabinet ensures that the research will maintain its integrity across time. Valuable research becomes artifacts for future meta-analyses, even if the researcher pursues other agendas.
When students complete research activities as part of their courses, the instructor becomes a collaborator. The OSF enables this collaboration in a potentially public space (not possible with classroom based software). Further, copied (forked) projects can give students additional scaffolding. For instance, if all students are completing an observational study, then the instructor could create a template project with instructions that each student must fork (copy) and use to start building their own project.
Another reason that scholars should start using OSF is that student research should be shared. Though most of it will not make major contributions, the process of completing the research cycle helps students learn the research process.
At PLU, Capstone projects are required to be public; the OSF provides a platform to expand that dissemination. This type of research can be shared with graduate schools and employers where applicable. An OSF project is an easy link osf.io/6lette (too bad it isn’t 8 letters).

20. Exploratory and Confirmatory Meta-Science

21. An EAMMi2 Example Last Week
6 coauthors on a teaching poster for project.
Need easy way to share content and build our poster
First create a google.doc and share with coauthors

22. Put Resources on OSF
Blue Box top left – Name project better than “CAPSTONE499 project”
Purple Diamond – add a short description
Orange circle – Add contributors here
Purple circle – use this to configure settings
Yellow Circle – will expand the Wiki so that you can edit it.
Orange arrow middle – click here for your APA style citation
Green Arrow bottom – you can drag and drop files
Blue arrow– make a new component

23. Provide Instructions using Wiki
Red Rectangle –Editing Mode
Yellow Rectangle –Editing Tools
Blue Rectangle –Reader View
Purple Rectangle – Editing Space

24. Make the Wiki informative and connected
Description
Editing Tools Menu
Red Square—bold/italic
Blue Square—spell check
Green Square--hyperlink
Green Circle—block quote
Pink Circle—code data
Pink Rectangle—add picture
Beyond
Numbered Lists
Bulleted Lists
Heading
Horizontal Rule

25. Sharing Add collaborators to share information, workload, and credit.
With collaborators
With Public
Add collaborators to share information, workload, and credit.
Search for OSF members,
Add non-OSF members anyway
Add with varying permissions
Includes view-only links for projects under review
Use the Public Button
Make public when ready. Anyone with link can access.
It can be made private again, but once public there is not guarantee
Add Keywords and tags

26. Adding Contributors Orange Rectangle – find contributors, new window
Green Rectangle – admins can add or delete, others can read and/or write
Blue Rectangle – determines whether someone is added in references/listed as contributor
Pink Rectangle – make view only links for manuscripts under review

27. OSF provides help finding contributors

28. Don’t get distracted; Don’t take too long

29. File and Version Control
Types of control
Registration
Updating Wiki’s
Edits to the wiki are tracked by the OSF. Can revert to earlier version as administrator
Updating Files
OSF can track uploads of the same file as multiple versions (see COS curriculum)
With google drive
Use registration to date stamp an OSF project.
Cannot be revoked when finalized

30. Connecting with add-ons
For each component under parent project.
1st Block – basic info
2nd Block – select add-ons
3rd Block – config. add-ons
4th Block – enable Wiki
5th Block – commenting
Either contributors or all users
6th Block – Project Affiliation/Branding

31. Add-ons Explored
Use what you know
Here’s what I know
Dropbox – anything that is added to the dropbox folder is added to the OSF page(and reverse) and all computers attached to that dropbox
Google Drive – easy connect to google.docs. Like Dropbox in Google
OSF Storage—Drag and drop from hard-drive to OSF
External Link—Embed hyperlinks into the Wiki for easy connection to important resources
Mendelay/Zotero – These track citations as an alternative to Endnote

32. Select only one folder, not all of drop box or google drive

33. Commenting Explored Peers or Professors
Comments are links to a specific page, not all pages
Must have an OSF account to make posts (comments or public)
Activity is date/time stamped
Configure settings to receive notification of comments

34. Find Support
– they are highly motivated to improve scientific transparency by making sure your OSF experience is rewarding
Jon Grahe – invite me to your institution, department, or division for workshops
This Slide Show (
Available on P481-Reproducibility in Psychological Sciences - Spring16_PLU OSF page
Use slides to build course specific instructions or as basic introduction