1. Practical Steps for Increasing Openness and Reproducibility Courtney Soderberg Statistical and Methodological Consultant Center for Open Science

2. INFRASTRUCTURE COMMUNITY METASCIENCE

3. Scientific Ideals - Innovative ideas - Reproducible results - Accumulation of knowledge

5. What is reproducibility? Computation Reproducibility: – If we took your data and code/analysis scripts and reran it, we can reproduce the numbers/graphs in your paper Empirical Reproducibility: – We have enough information to rerun the experiment or survey the way it was originally conducted Replicability: – We use your exact methods and analyses, but collect new data, and we get the same statistical results

6. Search and discover Develop idea Design study Acquire materials Collect data Store data Analyze data Interpret findings Write report Publish report

7. Why should you care? Your own work less efficient – Hard to build off our own work, or work of others in our lab We may not have the knowledge we think we have – Hard to even check this if reproducibility low

8. Current Barriers ● Statistical o Low Power o Researcher degrees of freedom ● Transparency o Poor documentation o Lack of openness

9. Steps 1.Create a structured workspace

10. Open Science Framework https://osf.io

11. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration

12. Pre-registration Before conducting a study registering: – The what of the study: General information about what you are investigating and how Research question Population and sample size General design Variables you’ll be collecting, or dataset you’ll be using

13. Pre-registration Study pre-registration decreases file-drawer effects – Helps with discovery of unpublished, usually null findings

14. Figure 1. Positive Results by Discipline. Fanelli D (2010) “ Positive ” Results Increase Down the Hierarchy of the Sciences. PLoS ONE 5(4): e10068. doi:10.1371/journal.pone.0010068 http://127.0.0.1:8081/plosone/article?id=info:doi/10.1371/journal.pone.0010068

15. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power

16. Button et al. (2013) Power in Neuroscience

17. Low Power ● Low replicability due to power: o 16% chance of finding the effect twice ● Inflated effect size estimates ● Decreased likelihood of true positives

18. Low Power ● Low replicability due to power: o 16% chance of finding the effect twice ● Inflated effect size estimates

19. Distribution Shape 30% Power 90% Power

20. Significant effect sizes 30% Power 90% Power

21. True Cohen’s d =.50

22. Low Power ● Low replicability due to power: o 16% chance of finding the effect twice ● Inflated effect size estimates ● Decreased likelihood of true positives

23. Positive Predictive Value Curves Button, Ioannidis, Mokrysz, Nosek, Flint, Robinson, & Munafo (2011)

24. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research

25. Pre-analysis plan Like a pre-registration – Detail the analyses planned for confirmatory hypothesis testing Decrease researcher degrees of freedom

26. Researcher Degrees of Freedom ● All data processing and analytical choices made after seeing and interacting with your data Should I collect more data? Which observations should I exclude? Which conditions should I compare? What should be my main DV? Should I look for an interaction effect?

27. False positive inflation Simmons, Nelson, & Simonsohn (2012)

28. Solution: Pre-registered analyses ● Before data is collected, specify o Sample size o Data processing and cleaning procedures o Exclusion criterion o Statistical Analyses ● Registered in a read-only format so it can’t be changed

29. Exploratory vs. Confirmatory Analyses Exploratory – Interested in exploring possible patterns/relationships in data to develop hypotheses Confirmatory – Have a specific hypothesis you want to test Pre-registration of analyses clarifies which are exploratory and which are confirmatory

30. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study

31. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study 6.Analyze and document analyses

32. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study 6.Analyze and document analyses 7.Share study data, code, materials

33. Why you might want to share Journal/Funder mandates Increase impact of work Recognition of good research practices

34. Incentives for Researchers Badges for open practices

35. Signals: Making Behaviors Visible Promotes Adoption

37. Data Availability in Psychological Science

38. Incentives for Researchers Badges for open practices Registered Reports

39. Design Collect & Analyze ReportPublish PEER REVIE W

40. Registered Reports Design Collect & Analyze ReportPublish PEER REVIE W

41. Registered Reports AIMS Neuroscience Attention, Perception, & Psychophysics Cognition and Emotion Comprehensive Results in Social Psychology Cortex Drug and Alcohol Dependence eLife Euro Journal of Neuroscience Experimental Psychology Journal of Accounting Research Journal of Business and Psychology Journal of Personnel Psychology Journal of Media Psychology Nutrition and Food Science Journal Perspectives on Psych. Science Royal Society Open Science Social Psychology Stress and Health Work, Aging, and Retirement

42. Incentives for Researchers Badges for open practices Registered Reports Pre-Reg Challenge

43. The $1,000,000 Preregistration Challenge Endorse TOP Guidelines Badges for Open Practices Registered Reports

44. https://cos.io/prereg

45. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study 6.Analyze and document analyses 7.Share study data, code, materials

46. How to make this more efficient? Have conversations with collaborators early – What is our data management plan? – What/when will we share? Be consistent across studies – If an entire lab has the same structure, then it’s easier to find things Document from the beginning

47. Where to get help: ● Reproducible Research Practices? o stats-consulting@cos.io ● The OSF? o support@osf.io support@osf.io ● Have feedback for how we could support you more? o contact@cos.io contact@cos.io o feedback@cos.io feedback@cos.io

48. API Docs https://api.osf.io/v2/docs/

49. Now OSF

50. OpenSesame Soon OSF 29 grants to develop open tools and services: https://cos.io/pr/2015-09-24/https://cos.io/pr/2015-09-24/

51. Awaiting Instruction…