1. Doing Open Science Lorne Campbell With inspiration and ideas from colleagues Etienne LeBel & Timothy Loving

2. I am Inclined to simply show this talk by Victoria Stodden: https://www.youtube.com/watch?v=sZIxzTsvWhw https://www.youtube.com/watch?v=sZIxzTsvWhw

3. Why Is Science Open? (title of first slide from Stodden’s talk) The only way to fully evaluate scientific conclusions is to fully understand the methods that underlie those conclusions Sharing study details allows for research to be reproducible Scientific community places more value in findings that are consistently reproduced with the same procedures

6. Donoho (via Stodden): The published paper is only an advertisement of the scholarship; it is not the scholarship itself

7. When and What to Share? When – Typically at the end of the research process in a manuscript submitted for peer review What – Details related to the particular results being presented Subset of conditions, measures, procedures, analytic plan (including analyses actually run) – As thorough as space limitations allow (example on next slide) Other details “available upon request”

8. M. Vess, 2012: “Warm Thoughts” paper in Psychological Science Method I recruited 56 individuals (32 females and 24 males) residing in the United States (mean age = 33.50 years, SD = 11.09 years) through Amazon’s Mechanical Turk (see Buhrmester, Kwang, & Gosling, 2011). Each participant received $0.35 as compensation for taking part in Study 1.Buhrmester, Kwang, & Gosling, 2011 Participants first completed a brief measure of adult attachment avoidance and attachment anxiety (Wei, Russell, Mallinckrodt, & Vogel, 2007). They were then randomly assigned to two conditions: In one condition, participants were asked to reflect on a past romantic breakup, whereas in the other condition, they were asked to reflect on an ordinary event. All participants then rated the desirability of warm-temperature refreshments (“hot tea/coffee,” “warm pie,” and “soup”) and neutral-temperature refreshments (“crackers,” “candy bar,” “potato chips,” and “pretzels”) on 11- point scales ranging from not at all desirable to extremely desirable. This measure was modeled after the one developed by Zhong and Leonardelli (2008). For each participant, ratings were averaged to create composite scores for warm-temperature desirability (α =.65) and neutral-temperature desirability (α =.66).Wei, Russell, Mallinckrodt, & Vogel, 2007Zhong and Leonardelli (2008)

9. Retrospective Reporting

10. Biased Recall A known problem with retrospective research methods Recalling Details of the Research Process: – What, specifically, was predicted before data was collected and/or analyses run? Why? – Many decisions made during the research process – Many initial hypotheses may be more abstract than specific – Data exclusion rules generated on the spot – What analyses were run, with what variables?

12. Another Feynman Quote Another thing I must point out is that you cannot prove a vague theory wrong….[For example] ‘A’ hates his mother. The reason is, of course, because she did not caress him or love him enough when he was a child. But if you investigate you find out that as a matter of fact she did love him very much…. Well then, it was because she was over-indulgent when he was a child! By having a vague theory it is possible to get either result. The cure for this one is the following. If it were possible to state exactly, ahead of time, how much love is not enough, and how much love is over-indulgent, then there would be a perfectly legitimate theory against which you could make tests. It is usually said when this is pointed out, ‘When you are dealing with psychological matters things can’t be defined so precisely’. Yes, but then you cannot claim to know anything about it. (Feynman, 1965, pp. 158-159)

13. Theory Building Requires testing “risky predictions” (Meehl, 1967, 1978) Risky prediction: hypothesis that stands a high chance of being wrong (Feynman, 1974; Popper, 1959)

14. Vagueness of hypothesis High Low Set of Possible Outcome From A Study Range of Outcomes Consistent with Hypothesis

15. But… “…the ‘derivations’ from most of the theories in social psychology are usually not unequivocal, or strictly logical, for they skip steps, they depend on unexpressed assumptions, and they rest on the criterion of intuitive reasonableness or plausibility rather than on formal logical criteria of consistency.” (p. 7) – Deutsch & Krauss (1965) Theory testing in psychology is hard: Our Theories do not dictate specific hypotheses Requires more truly confirmatory research (Schaller, 2015; Simpson, 2013)

16. Not all Research is Confirmatory We are not always testing specific hypotheses May have ideas of what to expect Or, patterns in the data may lead to the development of novel findings/hypotheses One process of discovery Agreed, this type of discovery is important – Simply share how the results were obtained – Can state at the beginning that specific hypotheses not being tested

17. Doing Open Science We receive a lot of training on research methods and statistical procedures (but likely not enough—another talk!) But, not much (if any) on how to do open science Technology today allows for open science practices

18. Why I Transitioned to Open Science Practices My support for the value of replication, and other activities occurring in the field in 2011-12, resulted in me conducting a direct replication with Etienne LeBel of research published in Psychological Science by Vess (2012) Created an OSF account, partly for this project Our results were published in PS in 2013 My OSF account at that time had 2 projects—the 2 replication attempts of Vess (2012) – Note: I had conducted other research during this time period. Hmm.

19. Decisions (1)I used open science practices with the replication studies (2) I was using closed research practices for my own original research So I asked myself if I only value openness for other people’s research? – I answered no. From that point on I was all in for open science But how?

20. What to do? A lot of uncertainties, unanswered questions, and not a lot of guidance John Lennon: “Well I tell them there’s no problem, only solutions” Wrote paper with Timothy Loving and Etienne LeBel (2014) attempting to find some solutions

28. Example

32. What Else? Blind Analysis? Blind Analysis Methods Videos? Methods Videos

35. “You Had an Option, Sir” We all have the option to adopt open, or closed, research practices; it is our choice. When deciding what option to choose, ask yourself if that is the best choice for advancing scientific discovery. We have the obligation of sharing our choice for open or closed research practices.