Download presentation
Presentation is loading. Please wait.
1
Three points 1. Scientists’ Conflict of Interest 2
Three points 1. Scientists’ Conflict of Interest 2. Role of Direct Replication 3. Psychology’s crisis? Brian Nosek University of Virginia -- Center for Open Science My general substantive interest in the gap between values and practices. The work that I am discussing today is a practical application of this interest to the gap between scientific values and practices. In particular, how can I best advance knowledge and my career at the same time? Challenges I face when working to advance scientific knowledge and my career at the same time. And, how my scientific practices can be adapted to meet my scientific values.
2
Norms Counternorms Communality Secrecy Open sharing Closed
Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume
3
Norms Counternorms Communality Universalism Secrecy Particularlism
Open sharing Universalism Evaluate research on own merit Secrecy Closed Particularlism Evaluate research by reputation Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume
4
Norms Counternorms Communality Universalism Disinterestedness Secrecy
Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume
5
Norms Counternorms Communality Universalism Disinterestedness
Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume
6
Norms Counternorms Communality Universalism Disinterestedness
Open sharing Universalism Evaluate research on own merit Disinterestedness Motivated by knowledge and discovery Organized skepticism Consider all new evidence, even against one’s prior work Quality Secrecy Closed Particularlism Evaluate research by reputation Self-interestedness Treat science as a competition Organized dogmatism Invest career promoting one’s own theories, findings Quantity Communality – open sharing with colleagues; Secrecy Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations Quality – seek quality contributions; Quantity – seek high volume
7
Anderson, Martinson, & DeVries, 2007
8
Incentives for individual success are focused on getting it published, not getting it right
Nosek, Spies, & Motyl, 2012
9
Scientists’ Conflict of Interest
What is good for me versus What is good for science Solutions: Nudge Incentives – Leslie’s talk
10
Direct Replication Attempting to reproduce a previously observed result with a procedure that provides no a priori reason to expect a different outcome
11
There is no such thing as exact replication
Direct replication theoretically confronts the original finding and simultaneously tests generalizability across sample, setting, and history (at minimum)
12
Direct Replication “SUCCEEDS” “FAILS”
Increases confidence in original finding Increases generalizability of finding Could be due to error Decreases confidence in original finding Decreases generalizability of finding Could be due to error A priori: direct replication is an opportunity for theoretical confrontation Success: : Evidence that presumed irrelevant factors are irrelevant Failure: : Evidence that presumed irrelevant factors are relevant Ex post: “successful” direct replication increase precision and advance generalizability – presumed irrelevant factors are in fact irrelevant “failed” direct replications introduces opportunity for theoretical innovation because there may be previously unrecognized boundary conditions or necessary features for obtaining the finding that differ between the original and the replication procedures. That innovation is hypothesized with the new evidence and then tested with a follow-up investigation
13
Consequence: Theoretical innovation, replicate again
Direct Replication “SUCCEEDS” “FAILS” Consequence: Theoretical innovation, replicate again Decreases confidence in original finding Decreases generalizability of finding Could be due to error
14
Conceptual Replication
Direct Replication Worse for explanation Better for theoretical confrontation Conceptual Replication Better for explanation Worse for theoretical confrontation
15
So is there a crisis? Nature survey.
Who are these respondents?
16
And why are they saying that – pervasive challenge across disciplines in reproducing prior results. A fundamental feature of science is that scientific claims gain credibility compared to other kinds of claims based on the potential to independently reproduce the evidence. But, of course, whether there is a reproducibility crisis or not isn’t really an a relevant question. The real question is whether there are cultural norms and practices that are undermining the efficiency and effectiveness of knowledge building, and are there ways that we can do better?
17
Conclusion Psychology is not under threat, it is leading the way to more open, reproducible, efficient science Start with conclusion…. Why such focus on psych? We are getting the attention not because we have bigger problems, but because we are bothering to face them and we have productive ideas and are taking steps to deal with them. Think about how much time you spend not caring about what is happening in physics, chemistry, earth sciences, and biology. Turns out, that is just about the same amount of time that physicists, chemists, earth scientists, and biologists spend not caring about us. That’s why the “reproducibility crisis” is actually an opportunity for us, not a threat. Other areas of science will care about us more and more not when we solve our challenges, but when we solve theirs. And, that’s already underway… Slides available at:
Similar presentations
