1. Practical Steps for Increasing Openness and Reproducibility Courtney Soderberg Statistical and Methodological Consultant Center for Open Science

2. INFRASTRUCTURE COMMUNITY METASCIENCE

3. Scientific Ideals - Innovative ideas - Reproducible results - Accumulation of knowledge

5. What is reproducibility? Computation Reproducibility: – If we took your data and code/analysis scripts and reran it, we can reproduce the numbers/graphs in your paper Empirical Reproducibility: – We have enough information to rerun the experiment or survey the way it was originally conducted Replicability: – We use your exact methods and analyses, but collect new data, and we get the same statistical results

6. Search and discover Develop idea Design study Acquire materials Collect data Store data Analyze data Interpret findings Write report Publish report

7. Why should you care? Your own work less efficient – Hard to build off our own work, or work of others in our lab We may not have the knowledge we think we have – Hard to even check this if reproducibility low

8. Current Barriers ● Statistical o Low Power o Researcher degrees of freedom ● Transparency o Poor documentation o Lack of openness

9. Steps 1.Create a structured workspace

10. Open Science Framework https://osf.io

11. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration

12. Pre-registration Before conducting a study registering: – The what of the study: General information about what you are investigating and how Research question Population and sample size General design Variables you’ll be collecting, or dataset you’ll be using

13. Pre-registration Study pre-registration decreases file-drawer effects – Helps with discovery of unpublished, usually null findings

14. Figure 1. Positive Results by Discipline. Fanelli D (2010) “ Positive ” Results Increase Down the Hierarchy of the Sciences. PLoS ONE 5(4): e10068. doi:10.1371/journal.pone.0010068 http://127.0.0.1:8081/plosone/article?id=info:doi/10.1371/journal.pone.0010068

15. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power

16. Button et al. (2013) Power in Neuroscience

17. Low Power ● Low replicability due to power: o 16% chance of finding the effect twice ● Inflated effect size estimates ● Decreased likelihood of true positives

18. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research

19. Pre-analysis plan Like a pre-registration – Detail the analyses planned for confirmatory hypothesis testing Decrease researcher degrees of freedom

20. Researcher Degrees of Freedom ● All data processing and analytical choices made after seeing and interacting with your data Should I collect more data? Which observations should I exclude? Which conditions should I compare? What should be my main DV? Should I look for an interaction effect?

21. False positive inflation Simmons, Nelson, & Simonsohn (2012)

22. Solution: Pre-registered analyses ● Before data is collected, specify o Sample size o Data processing and cleaning procedures o Exclusion criterion o Statistical Analyses ● Registered in a read-only format so it can’t be changed

23. Exploratory vs. Confirmatory Analyses Exploratory – Interested in exploring possible patterns/relationships in data to develop hypotheses Confirmatory – Have a specific hypothesis you want to test Pre-registration of analyses clarifies which are exploratory and which are confirmatory

24. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study

25. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study 6.Analyze and document analyses

26. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study 6.Analyze and document analyses 7.Share study data, code, materials

27. Why you might want to share Journal/Funder mandates Increase impact of work Recognition of good research practices

28. What to share Sharing is a continuum – Data underlying just results reported in a paper – Data underlying publication + information about other variables collected – Data underlying publication + embargo on full dataset – All data collected for that study

29. Steps 1.Create a structured workspace 2.Create a research plan i.Pre-registration 3.Determine sample size/power 4.Pre-analysis plan for confirmatory research 5.Archive materials from study 6.Analyze and document analyses 7.Share study data, code, materials

30. How to make this more efficient? Have conversations with collaborators early – What is our data management plan? – What/when will we share? Be consistent across studies – If an entire lab has the same structure, then it’s easier to find things Document from the beginning

31. Where to get help: ● Reproducible Research Practices? o stats-consulting@cos.io ● The OSF? o support@osf.io support@osf.io ● Have feedback for how we could support you more? o contact@cos.io contact@cos.io o feedback@cos.io feedback@cos.io

32. Registered Reports Committee

33. Early Adopters 1.AIMS Neuroscience 2.Attention, Perception & Psychophysics 3.Comparative Political Studies 4.Comprehensive Results in Social Psychology 5.Cortex 6.Drug and Alcohol Dependence 7.eLife 8.Experimental Psychology 9.Frontiers in Cognition 10.Journal of Business and Psychology 11.Nutrition and Food Science Journal 12.Perspectives on Psychological Science 13.Social Psychology 14.Working, Aging, and Retirement

34. Signals: Making Behaviors Visible Promotes Adoption

36. Data Availability in Psychological Science

37. Connecting silos to enable increased discovery

38. http://osf.io/share

39. API Docs https://api.osf.io/v2/docs/

40. http://osf.io/ free, open source

41. Put data, materials, and code on the OSF

42. Manage access and permissions

43. Automate versioning With hashes

44. Share your work

45. Now OSF

46. OpenSesame Soon OSF 29 grants to develop open tools and services: https://cos.io/pr/2015-09-24/https://cos.io/pr/2015-09-24/

47. Get a persistent identifier https://osf.io/tvyxz/

48. See the impact File downloads

49. http://osf.io/share

50. API Docs https://api.osf.io/v2/docs/

51. Awaiting Instruction…