Unreliable research Trouble at the lab 1

Patrik Ahlberg Magnus Åhberg Syeda Rabab Naqvi Thawatachart Chulapakorn Group Members:

Outline: Introduction Understanding Insignificance Not Even Wrong Blame the Ref Harder to Clone than you wish Making the Paymaster´s Care Personal Reflections 3

Introduction Irreproducibility is on rampage Self corrections are not working IRL “There is no cost to getting things wrong, The cost is not getting them published.” Brian Nosek, University of Virginia 4

Understanding Insignificance False Positives + False Negatives = Problem We are not using statistics Not possible with power of 0.8 5

Not even wrong Research is commonly not thought through or not well executed Examples: – The ”pentaquark” saga (experiment not properly blinded) – Risks with computer model ”tuning” – Longevity associated genetic variations due to inproper handling of research samples 6

Blame the ref Peer reviewers are not detecting errors – John Bohannon´s made-up paper was published in 157 out of 304 journals – Fiona Godlee yielded similar results when sending an 8-error paper to the 200 reviewers of BMJ Errors are explained by incompetence rather than fraud The replication mechanism for scientific self- correction is not functioning well 7

Harder to Clone than you wish : Replication is hard to meet our standard To find errors in the publications, we need to replicate the work of others, but process is not according to our wish. Replication Lack of interest by Journals and Academic Researchers 8

Lack of Interest  Journals thirsty of novelty, show little interest in it.  Academic researchers mostly spend time on work which is more likely to enhance their careers. Replication is Hard  Original methods and data required  Unpublished Research  Clinical trials are very costly to rerun  Software may be different during the replication  Failure of replication due to tacit knowledge  Expermenter’s regress 9

Finally, we would say that… Making the paymasters care SituationPaymastersResearcher Problem-Prefer pioneer -Deny failed result -Paper machine -Bias discussion -Blame the previous error Purpose-Emphasize replication as same as others -Don’t serious the failed result (Hard) -Develop technical and statistical skill -Honest result -Enforce standard (Journal) 10

Patriks work Unethical publishing cancels this problem – No reproducibility possible – Small feasibility tests The system has given up on the system – Seniors warning juniors – Reliability comes from people not publications 11

Own reflections The possibility to ”tune” computer models until the desired results appear relates to my own research It is easy to percieve patterns and get results that are expected and might not even exist Responsibility as a reviewer: – When do I accept a reviewer invitation? – Do I expect to find errors? What am I looking for as a reviewer? 12

Own Reflections If some one tries to repeat my experiment, he must have  Access to original methods and data  Awareness of simulation procedure  Similar samples for study 13

Related to my (future) research Replication – Same or not? Why? Scrutiny – Procedure – Measurement Bias discussion Raw data Log book!!! 14

15