1. Questionable practices and misconduct in science
Marcel A.L.M. van Assen
Tilburg University & Utrecht University
Department of Methodology and Statistics & Department of Sociology

2. Who believes in the storybook image of the scientist?
(Veldkamp et al., 2017)
objective, rational, open-minded, intelligent, integer, cooperative

3. Who believes in the storybook image of the scientist?
(Veldkamp et al., 2017)
Lay people and scientists attributed considerably more objectivity, rationality, open-mindedness, intelligence, integrity, and cooperativeness to scientists than other highly educated people
Really?
Scientists are mere humans, and humans are fallible

4. Main messages/story
Current science is about results, process leading to results is largely hidden (black box)
Rewards for novel-nice-significant research
Questionable research practices (qrps) prevalent
Qrps and fraud largely undetected and not punished
Qrps and fraud are rational!
Science needs to be fundamentally restructered to limit problems (including qrps and fraud)

5. Overview How science works Definitions of qrps and fraud
Qrps and their prevalence
Fraud and prevalence
Qrps and fraud are rational!
Restructuring science

6. Which papers get published in journals?
(1) How science works
Which papers get published in journals?
Something novel
“Nice” story (good to read)
Statistically significant result
Note: all three are irrelevant to science
 Strong evidence on all three

7. (1) How science works Novel
Editor: “We don’t accept replication studies”
Nice story
HARKing (Kerr 1998)
Editor/reviewer: “Leave out/add this/that to obtain nicer story”
Significant
Publication bias
Failures to replicate

8. (1) How science works Publication bias Fanelli (2010)
Varies across disciplines
UK < (rest) Europe < US < Japan

9. What do we see of process leading to papers?
(1) How science works
What do we see of process leading to papers?
Initial theory/hypotheses?  no
Complete methods?  no
Data?  no
All analyses?  no
Distinction confirmatory/exploratory analyses?  no
Review process?  No
Final paper  yes (if you are lucky or rich)

10. (1) How science works Conclusion
Current science is a black box (omission process), focused on novel-nice-significant (irrelevant)

11. (2) Definitions of qrps and fraud
What is “fraud”?
No consensus
Fraud specific meaning in law  ‘scientific misconduct’ preferable terminology (Resnik 2012)
Intention to deceive is key (Fanelli 2009)
Scientific misconduct
(1) Fabrication: “invention” data
(2) Falsification: willful distortion data or results
(3) Plagiarism: copy without attribution

12. (2) Definitions of qrps and fraud
Questionable research practices (NAS ’92) =
“.. actions that violate traditional values of the research enterprise and that may be detrimental to the research process.. “
… Rather vague … (list of behaviors)
Working definition:
“detrimental to the research process”
 without qrps results would have been different, or interpretation of results would have been different

13. (2) Definitions of qrps and fraud
Questionable research practices
Results and interpretation of results depend on how data were collected, (how many) analyses were performed
“Patients in the treatment group score significantly higher on X than patients in the control group (p = .012)”
p-value only correct in case of ONE test

14. (2) Definitions of qrps and fraud
Questionable research practices: final comments
Distinction fraud and qrps: intention to deceive (really?)
Distinction and definition fraud, fabrication, falsification, QRP unclear and arbitrary (Fanelli 2009)
Many shades of grey between good conduct and fraud

15. (3) Qrps and their prevalence
Questionnaires
P-values
Process to article
(A) Questionnaires
Do researchers use qrp and find them defensible?
John et al. (2012, Tijdink et al. (2014), Bedeian et al. (2010), Feld et al. (2012)
Questionnaires research is disputed (Fiedler & Schwarz, 2015)

16. (3) Qrps and their prevalence
(A) Questionnaires
Do researchers use qrp and find them defensible?
Using anonymous elicitation procedure with incentives
Survey > 2,000 psychologists
Self-serving bias: own = collaborators < researchers own institute < researchers other institute

17. Adapted from John, Loewenstein, and Prelec (2012)
Ordering based on defenisibility rating (0-2)
Examples: 2, 3, 7, 8
(John, Loewenstein, & Prelec, 2012): Survey among more than 2000 psychologists

18. (3) Qrps and their prevalence
(B) P-values
Hartgerink et al (2016)
Inspect p-values reported as ‘p= .05’
67.5% > 0.05 and 32.5% lower than .05

19. (3) Qrps and their prevalence
(C) Process to paper
Chan et al (2004)

20. (3) Qrps and their prevalence
(C) Process to paper
But see also O’Boyle et al (2014)
Tracking differences between PhD theseses and articles resulting from them
Proquest Dissertations and Thesises (Management Sciences)
142 studies, 2,311 hypotheses

21. (3) Qrps and their prevalence
Conclusions
Qrps highly prevalent
Hard to detect when science is black box or process is not reviewed

22. (4) Fraud and prevalence
Questionnaire: 0.6% (John et al 2012)
Meta-analysis on questionnaires: ~2% (Fanelli)
3.8% figure manipulation (level of papers!)

23. (4) Fraud and prevalence
But...
Hard to detect (black box)
Reviewers do not pay attention to it
Scientists do not recognize it

24. (4) Fraud and prevalence
Conclusions
Prevalence unknown
Because qrps are much more prevalent, damage to science from qrps is higher than from misconduct

25. (5) Qrps and fraud are rational!
Benefits of qrps and fraud are high
“Helps” to get nice-significant papers, at low cost (small sample sizes, quick results)
Helps to obtain PhD, grants, tenure, to become a full professor
Qrps are accepted or recommended by many scientists, so hardly any costs  what about fraud?

26. (5) Qrps and fraud are rational!
Probability of getting caught is small
Undetected (black box, lack of attention, lack of expertise)

27. (5) Qrps and fraud are rational!
If caught, then unlikely to be punished
For many examples, see “Ontspoorde wetenschap” or “Derailed science” by Van Kolfschooten
Investigation in secrecy (if any)
Investigation amateuristic
“fraudster” often protected
If punished, punishment = small
Whistleblower punished
Fraudulous research not retracted

28. (5) Qrps and fraud are rational!
One specific case: Trivers (Brown)
Study in Nature (2005), a lot of publicity
Brown, W. M., Cronk, L., Grochow, K., Jacobson, A., Liu, C. K., Popović, Z., & Trivers, R. (2005). Dance reveals symmetry especially in young men. Nature,438(7071),
Trivers detects fraud
Brown, and Cronk deny
Horrible treatment of Trivers, and lots of work by Trivers
Finally, ten years later, study is retracted and fraud established

29. “Trivers” (Brown)

30. “Trivers” (Brown) [continue reading]

31. Cases of misconduct: from start to end
(1) Something ‘strange’ is noticed about research of Suspect by someone
 Mostly it ends here
(2) ‘Strangeness’ is communicated by someone to …
(i) Suspect
 Mostly it ends here; could have bad consequences for Accusor

32. Cases of misconduct: from start to end
(2) ‘Strangeness’ is communicated by someone to …
(ii) Colleague
 Mostly it ends here
(iii) (Integrity or ‘high’) person within organization
 nothing happens
 Accusor experiences negative consequences
 investigation by own institute 

33. Cases of misconduct: from start to end
(2) (iii) (Integrity or ‘high’) person within organization
 investigation by own institute
Investigation usually is …
Ad hoc
‘Amateuristic’
Opportunistic
 Result of investigation is confidential, usually in favor of Accused, Accusor gets reprimanded
 … Investigation is carried out well, by an independent committee (as in the case of Stapel)!

34. Cases of misconduct: from start to end
(2) ‘Strangeness’ is communicated by someone to
(iv) Integrity institute outside organization
Netherlands: LOWI
 Independent investigation
 Reports to organization of Accused, which determines what is going to happen
US: ORI
 Can sanction Accused and organization of Accused (and reward)

35. Cases of misconduct: from start to end
Conclusions
Whistleblower has a large disadvantage
Accused and Organization of Accused are protected
Investigations of fraud are often ad hoc, amateuristic (and opportunistic)
Highly confidential: bad for science
 Few retractions, reasons of retractions unknown
 Gap scientific and legal investigation

36. (5) Qrps and fraud are rational!
Conclusions
Expected benefits > expected costs of fraud, hence fraud is rational behavior
Partly because system, qrps and fraud will continue to exist (black box)

37. (6) Restructuring science
What do we see of process leading to papers?
Initial theory/hypotheses?  no
Complete methods?  no
Data?  no
All analyses?  no
Distinction confirmatory/exploratory analyses?  no
Review process?  No
Final paper  yes (if you are lucky or rich)

38. (6) Restructuring science
1. Theory, Method/Design
2. Preregistration of Paper
and Method/Design
R1. Pre-review
Open
Open
R1. Post-review
Open
Open
3. Data collection
4. Complete paper with
data-analysis (expl. and conf.)
and discussion
R2. Pre-review
Open
Open
R2. Post-review
Open

39. (6) Restructuring science
1. Theory, Method/Design
2. Preregistration of Paper
and Method/Design
R1. Pre-review
Open
Open
R1. Post-review
Open
Open
3. Data collection
4. Complete paper with
data-analysis (expl. and conf.)
and discussion
R2. Pre-review
Open
Open
R2. Post-review
Open

40. (6) Restructuring science
Only when science is restructured such that
Theory/hypotheses, method, and planned analyses are pre-registered
Reviewers check consistency between paper and pre-registration (and not novel-nice-significant)
All research flow is openly shared
QRPs will be limited
Fraud will be more likely detected

41. The End Main messages/story
Current science is about results, process leading to results is largely hidden (black box)
Rewards for novel-nice-significant research
Questionable research practices (qrps) prevalent
Qrps and fraud largely undetected and not punished
Qrps and fraud are rational!
Science needs to be fundamentally restructered to limit problems (including qrps and fraud)