1. Safer science: making psychological science more Transparent & replicable
Simine Vazire
UC Davis

2. Make predictions and test them. Show your work.
What is science?
Make predictions and test them. Show your work.

3. overview
PART I: Values and norms in science PART II: Stats review PART III: Crisis PART IV: Solutions

4. Part i: values and norms
PART I: Values and norms in science PART II: Stats review PART III: Crisis PART IV: Solutions

5. Values and norms in science
What makes science different from other ways of knowing?
What makes science different from pseudoscience?
What are the core values of science?

6. Part i: values and norms
What makes science different from other ways of knowing?

7. Ways to Acquire Knowledge
Intuition: I feel that it is so
Authority: A trusted source says that it is so
Empiricism: I’ll believe it when I see it
Rationalism: Logically, it follows that… {
Scientific Knowledge
Authority – you get a parking ticket. Someone in authority is telling you a truth.

8. Part i: values and norms
What makes science different from other ways of knowing?
Focus on empiricism and rationality
What makes science different from pseudoscience?

9. What makes science different from pseudoscience?
What is “the scientific method”?
Positivism: Base conclusions on objective facts
Problem: No such thing (always requires some interpretation)
Falsificationism: Make predictions that can be wrong
Problem: All scientific theories are wrong
Successive approximation: Make predictions that are closer and closer to reality
Problem: How do we know if we’re getting closer?
Maybe there is no such thing as THE scientific method?

10. Some “rules” for science
Make predictions and test them. Show your work.

11. Part i: values and norms
Instead of THE scientific method, there are norms and values that are followed in science.
This helps distinguish science from pseudoscience
Robert Merton, a sociologist, studied the “ethos” of science
Came up with four “norms” that help define science
Can also help distinguish science from pseudoscience

12. Merton’s norms Universalism Communality Disinterestedness
The validity of a scientific claim does not depend on who is making it.
No hierarchy. Status should not matter.
Communality
The findings of science belong to everyone, they are not private property.
No secrecy. Open communication is key.
Disinterestedness
Scientists should be focused on finding the truth, not on their own success.
No self-interest. Report whatever you find, even if it makes you look bad.
Organized skepticism
Do not take things at face value. Verify others’ claims.
Nothing is sacred.

13. “What we call scientific knowledge today is a body of statements of varying degrees of certainty. Some of them are most unsure; some of them are nearly sure; but none is absolutely certain. Scientists are used to this. We know that it is consistent to be able to live and not know. Some people say, ‘How can you live without knowing?’ I do not know what they mean. I always live without knowing. That is easy. How you get to know is what I want to know.”
Organized skepticism

14. Transparency as a core value
“The credibility of scientific claims comes, in part, from the fact that their meaning is, at a minimum, available for other scholars to rigorously evaluate […] Such open access to the origins of others’ claims is the hallmark of scientific ways of knowing. Accordingly, when social scientists fail to document their assumptions, decisions, and actions […] it limits others’ ability to understand the meaning of scientists’ claims.” -Lupia & Elman, 2014, p. 20

15. Transparency leads to reproducible science
“Nullius in Verba”
Take no one’s word
Science means never having to say “Trust me”
Others should be able to reproduce my findings

16. The ethos of science
The overarching goal of science is to get closer and closer to the truth
We do this by finding out what is reproducible, and throwing out what is not
Self-correction is an active process, and requires (Merton’s norms):
No hierarchy – anyone can challenge anyone else’s finding
Transparency – everything is out in the open
No conflicts of interest – truth is more important than personal success
Skepticism – tearing things down is as valuable as finding new things
For Merton, these were moral values, not just habits
Scientists do not always (ever?) completely live up to these values

17. Self-correction in science

18. Some “RULES” for science
Make predictions and test them. Show your work.

19. Part i: values and norms
What makes science different from other ways of knowing?
Focus on empiricism and rationality
What makes science different from pseudoscience?
Testable predictions, falsifiability, verifiability
What are the core values of science?
Transparency, reproducibility, self-correction

20. So I have just one wish for you—the good luck to be somewhere where you are free to maintain the kind of integrity I have described, and where you do not feel forced by a need to maintain your position in the organization, or financial support, or so on, to lose your integrity.  May you have that freedom.
-Richard Feynman

21. Part Ii: stats review
PART I: Values and norms in science PART II: Stats review PART III: Crisis PART IV: Solutions

22. Errors in Science

23. not significant p < .05 Is there an effect? True state of the world
No effect
Effect
No effect
not significant
Our Conclusion
p < .05
Effect

24. not significant p < .05 Is there an effect? True state of the world
No effect
Effect
Correct
Rejection
No effect
not significant
Our Conclusion
Hit
p < .05
Effect

25. not significant p < .05 Two kinds of errors True state of the world
No effect
Effect
Correct
Rejection
False
Negative
No effect
not significant
Our Conclusion
Hit
p < .05
Effect

26. not significant p < .05 Two kinds of errors True state of the world
No effect
Effect
Correct
Rejection
False
Negative
No effect
not significant
Our Conclusion
False
Positive
Hit
p < .05
Effect

27. False positive rate True state of the world Correct Rejection False
No effect
Effect
Correct
Rejection
False
Negative
No effect
95%
Our Conclusion
False
Positive 5%
??%
Hit
??%
Effect

28. So what is the % of false positives?

29. Part IiI: CRisis
PART I: Values and norms in science PART II: Stats review PART III: Crisis PART IV: Solutions

30. 97%
37% xx
Open Science Collaboration, 2015, Science

33. How to turn anything into a significant result
Did you get the effect you predicted? No
Yes
Did you get ANY effect?
HARK!
Yes
Publish

34. HARKing

36. Make predictions and test them. Show your work.
What is science?
Make predictions and test them. Show your work.
[The same dataset cannot be used to generate the hypothesis and to test it]

37. How to turn anything into a significant result
Did you get the effect you predicted? No
Yes
Did you get ANY effect?
HARK!
Yes No
Publish
p-hack!
Can you dig around and find one?
Yes

38. Source: Leif Nelson’s slides

39. H1: The U.S. economy is affected by whether Republicans or Democrats are in office

40. H1: The U.S. economy is affected by whether Republicans or Democrats are in office

42. How to turn anything into a significant result
Did you get the effect you predicted? No
Yes
Did you get ANY effect?
HARK!
Yes No
Publish
p-hack!
Can you dig around and find one?
Yes No
File drawer

43. % significant (Psychology)
Fanelli (2012)

44. False positive rate True state of the world Correct Rejection False
No effect
Effect
Correct
Rejection
False
Negative
No effect
95%
Our Conclusion
False
Positive 5%
Hit
Effect

46. Make predictions and test them. Show your work.
Where we went wrong
Make predictions and test them. Show your work.

47. Why not just do science right?
We didn’t know better.
It’s hard.

49. What is wrong with people?
“The first principle is that you must not fool yourself – and you are the easiest person to fool.”
-Richard Feynman

50. What is wrong with people?
Students start out with the right ideas about science
That you should make predictions ahead of time
That you should try to prove yourself wrong
That you should report everything you find
Incentive structure encourages cutting corners
Overconfidence leads us to believe incredible results
Scientific community rewards scientists who can produce amazing results

51. Part IV: solutions
PART I: Values and norms in science PART II: Stats review PART III: Crisis PART IV: Solutions

52. How can we do better?

53. When is a finding likely to be true?
Large sample

54. Why are large samples better?

55. When is a finding likely to be true?
Large sample
Small p-value (well below p < .05)

56. Why p = .04 should be rare

57. Why p = .04 should be rare

58. When is a finding likely to be true?
Large sample
Small p-value (well below p < .05)
Pre-registered

59. What is pre-registration?
Source: Leif Nelson’s slides

60. When is a finding likely to be true?
Large sample
Small p-value (well below p < .05)
Pre-registered
Full disclosure (transparency)

61. What is transparency? Making materials publicly available Sharing data
Disclosing flexibility in data analysis
If you have a prediction, pre-register
If you don’t, don’t HARK

62. Why transparency?
“The credibility of scientific claims comes, in part, from the fact that their meaning is, at a minimum, available for other scholars to rigorously evaluate […]
Such open access to the origins of others’ claims is the hallmark of scientific ways of knowing.
Accordingly, when social scientists fail to document their assumptions, decisions, and actions […] it limits others’ ability to understand the meaning of scientists’ claims.”
-Lupia & Elman, 2014, p. 20

63. Why transparency?

64. When is a finding likely to be true?
Large sample
Small p-value (well below p < .05)
Pre-registered
Full disclosure (transparency)
Has been replicated

66. When is a finding likely to be true?
Large sample
Small p-value (well below p < .05)
Pre-registered
Full disclosure (transparency)
Has been replicated*
Believable

67. Dreber et al.,
2015, PNAS

68. Make predictions and test them. Show your work.
How can we do better?
Make predictions and test them. Show your work.

70. Science is hard www.improvingpsych.org
To make science more scientific, we need :
More transparency
More accountability
Lower expectations

71. The end