Crisis Schmeisis: Introduction to the Open Science Framework for Scholarship and Teaching Jon Grahe In all my affiliations, there are projects that utilize the OSF. At PLU, my scholarship is housed there; At JSP, authors use it to apply for Open Science Badges; At Psi Chi, the Research Advisory Committee uses it to compile measurement scales for members to use.

Outline What Crisis and Who Cares? Review Tools Overall Transparency and Openness Promotion (ToP) Guidelines Practice: Next Research/Class Project Review Open Science Framework (OSF) features Recognize ways to employ the OSF for personal scholarship for scholarship in the classroom Though there are only two goals, there are many benefits.

Crisis Schmeisis Festering Concerns about Methods Since ?? Demonstration of Doubt Ioannidis (2005) Year of the Crisis (2011) The Rebirth of Scientific Methods (2012) Psychology Leads through Introspection

Open Science Framework Responding to Crisis Replication Debate -- Publication Debate Perspectives on Psychological Science special issues Social Media -- nasty discourse and new spaces Open Invitation Meta-Science Projects Center for Open Science and The Many Lab and Psi Chi and the Next International Collaborative Exchange New Publication Standards and Expectations Transparency and Openness in Science Confidence Intervals, Effect Sizes “the New Statistics” Center for Open Science Open Science Framework

Who Cares? Assumptions of Inferential Statistics Exploration vs. Confirmation Researcher Degrees of Freedom Questionable Research Practices Simmons, J. P., Nelson, L. N., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. Choosing from DVs Choosing sample size Using covariates, Reporting Subsets of experimental conditions Alpha = .05, and therefore p = .05 assuming a random distribution. When we interference with randomness, that raises the alpha level.

Science Requires Transparency Research Degrees of Freedom lead to Questionable Research Practices Scientific Evidence is not Immune to Experimenter Bias Expectancy Effects Motivated Reasoning Double Blind Designs not enough

Open Science as Antidote Confirmatory Tests require forward planning Preregistration and the “Pre-reg Challenge” Replication requires access to materials Open Materials Reproducibility requires data reanalysis Open Data Open Transparency Badges and the Transparency and Openness in Publishing Guidelines (ToP guidelines)

Next Research/Class Project Next Step in Research Agenda Research Methods or Capstone Project Group Class Project Artistic Endeavor Book for Scholarly or Public Audience

Exploratory Pre-registration Date Stamping Ideas and Plans Reduce Researcher Degrees of Freedom Permanent Ink or Etched in Stone Methods of Pre-registration Open Science Framework If there are 32 reseacher degrees of freedom. How many do you want to? Or are you able to claim from your Confirmatory Meta-Science Project?

Many Options

Preregistration as Protection Confirmatory Meta-Science Step 2 Preregister your confirmatory meta-science project now to avoid anyone from stealing your idea!

Preregistration Challenge Template Study Information Title Authorship Research Questions Hypotheses Sampling Plan Existing data Explanation of existing data Data collection procedures. Sample size Sample size rationale Stopping rule Variables Manipulated variables Measured variables Indices Design Plan Study type. Blinding. Study design. Randomization. Analysis Plan Statistical models Transformations Follow-up analyses Inference criteria Data exclusion Missing data Exploratory analysis (optional) Script (Optional) Analysis scripts (Optional) Other Handout worksheets

Personal Benefit from Open Science Free, Easy to Access, Easy to Share, Permanent Storage of Research Materials. Mechanism to disseminate Meta-Science Does not apply to  Researchers without collaborators Researchers with only one computer Researchers with specified and regularly updated back-up storage plan Others?

Open Science Framework— Host your Meta-science Project The OSF is a open-sourced computer software platform written and managed by the Center for Open Science (https://cos.io) OSF Project like an electronic file cabinet With an interactive cover page; unlimited files and folders; That can be shared easily with some, all, or none Imagine an electronic file cabinet; it is such a great metaphor. This file cabinet has drawers (called components on the OSF) and folders within those drawers. The entire system can be completely private, shared with a few, or shared for anyone who can find the link. Instead of a bland metal cabinet, the exterior contains details about which drawers to open, who contributed the work, and so much more.

The Open Science Framework as electronic file cabinet Editable Wiki Version Control Drag & Drop Commenting Record Contributions Motivated Support ANYONE Some People Only Me

Open Science Framework Features Internet page for scholarship (WIKI screen) Components Sharing with Contributors Sharing with Public File and Version Control Connecting with add-ons (e.g. Dropbox) Enabling Comments from contributors/public These are the basic features (some are pretty valuable), and there is so much more than what I will cover in this slide show. However, these slides provide enough instruction so that researchers can employ the OSF for their personal scholarship or for purpose of developing a student researcher.

Center for Open Science OSF Curriculum https://cos.io/our-services/training-services/cos-training-tutorials/ Center for Open Science OSF Curriculum Introduction Reproducibility, OSF and Reproducibility Project setup Creating, Contributors, Wiki, Components, Add Ons Documentation Registration, File and Version Control Sharing Contributors & Public Privacy, Increasing Discoverability with tags Conclusion Take-aways, How to get more information The Center for Open Science is a wonderful organization and they developed the OSF. They have their own curriculum. While this slide show breaks the process into very simple components, there are more details in these links

Other Resources CREP Instructions and Workflow Procedures Collaborative Replications and Education Project CREP Instructions and Workflow Procedures Build a basic project for students. The CREP Step-by-Step provides instructions on how to build a project (https://osf.io/srh4k/) This slide show (https://osf.io/zgrax/) is located on my Crisis Schmeisis Tour Page The Collaborative Replications and Education Project developed resources to teach young researchers in our large scale collaborative research project where students replicate students as part of their learning experiences and those findings are collated into an estimate of the reproducibility of a given manuscript. Of course, this slide show is intended to be a resource and when I give this talk, this is a very funny joke

Applying OSF features to Classroom Train students to be transparent scientists and … Student Projects are Research Instructors are collaborators Forked projects provide scaffolding (copied from original) Student Projects could be Shared Comment features allow for direct, monitored peer-review feedback or other constituents Capstone projects should be public I am going to primarily use learning related examples in this slide show. I think of students as future scientists or consumers of science. In either case, employing the OSF as an electronic file cabinet ensures that the research will maintain its integrity across time. Valuable research becomes artifacts for future meta-analyses, even if the researcher pursues other agendas. When students complete research activities as part of their courses, the instructor becomes a collaborator. The OSF enables this collaboration in a potentially public space (not possible with classroom based software). Further, copied (forked) projects can give students additional scaffolding. For instance, if all students are completing an observational study, then the instructor could create a template project with instructions that each student must fork (copy) and use to start building their own project. Another reason that scholars should start using OSF is that student research should be shared. Though most of it will not make major contributions, the process of completing the research cycle helps students learn the research process. At PLU, Capstone projects are required to be public; the OSF provides a platform to expand that dissemination. This type of research can be shared with graduate schools and employers where applicable. An OSF project is an easy link osf.io/6lette (too bad it isn’t 8 letters).

Exploratory and Confirmatory Meta-Science

An EAMMi2 Example Last Week 6 coauthors on a teaching poster for project. Need easy way to share content and build our poster First create a google.doc and share with coauthors

Put Resources on OSF Blue Box top left – Name project better than “CAPSTONE499 project” Purple Diamond – add a short description Orange circle – Add contributors here Purple circle – use this to configure settings Yellow Circle – will expand the Wiki so that you can edit it. Orange arrow middle – click here for your APA style citation Green Arrow bottom – you can drag and drop files Blue arrow– make a new component

Provide Instructions using Wiki Red Rectangle –Editing Mode Yellow Rectangle –Editing Tools Blue Rectangle –Reader View Purple Rectangle – Editing Space

Make the Wiki informative and connected Description Editing Tools Menu Red Square—bold/italic Blue Square—spell check Green Square--hyperlink Green Circle—block quote Pink Circle—code data Pink Rectangle—add picture Beyond Numbered Lists Bulleted Lists Heading Horizontal Rule

Sharing Add collaborators to share information, workload, and credit. With collaborators With Public Add collaborators to share information, workload, and credit. Search for OSF members, Add non-OSF members anyway Add with varying permissions Includes view-only links for projects under review Use the Public Button Make public when ready. Anyone with link can access. It can be made private again, but once public there is not guarantee Add Keywords and tags

Adding Contributors Orange Rectangle – find contributors, new window Green Rectangle – admins can add or delete, others can read and/or write Blue Rectangle – determines whether someone is added in references/listed as contributor Pink Rectangle – make view only links for manuscripts under review

OSF provides help finding contributors

Don’t get distracted; Don’t take too long

File and Version Control Types of control Registration Updating Wiki’s Edits to the wiki are tracked by the OSF. Can revert to earlier version as administrator Updating Files OSF can track uploads of the same file as multiple versions (see COS curriculum) With google drive Use registration to date stamp an OSF project. Cannot be revoked when finalized

Connecting with add-ons For each component under parent project. 1st Block – basic info 2nd Block – select add-ons 3rd Block – config. add-ons 4th Block – enable Wiki 5th Block – commenting Either contributors or all users 6th Block – Project Affiliation/Branding

Add-ons Explored Use what you know Here’s what I know Dropbox – anything that is added to the dropbox folder is added to the OSF page(and reverse) and all computers attached to that dropbox Google Drive – easy connect to google.docs. Like Dropbox in Google OSF Storage—Drag and drop from hard-drive to OSF External Link—Embed hyperlinks into the Wiki for easy connection to important resources Mendelay/Zotero – These track citations as an alternative to Endnote

Select only one folder, not all of drop box or google drive

Commenting Explored Peers or Professors Comments are links to a specific page, not all pages Must have an OSF account to make posts (comments or public) Activity is date/time stamped Configure settings to receive notification of comments

Find Support support@osf.io – they are highly motivated to improve scientific transparency by making sure your OSF experience is rewarding Jon Grahe – invite me to your institution, department, or division for workshops This Slide Show (https://osf.io/f5ses) Available on P481-Reproducibility in Psychological Sciences - Spring16_PLU OSF page Use slides to build course specific instructions or as basic introduction