What counts as good evidence? Dylan Evans Department of Mechanical Engineering University of Bath

It’s hard to know what is bogus! NASA invests $1m in antigravity machine US Patent 6,362,718 issued for a motionless electromagnetic generator Both of these machines are physically impossible But scientists have advocated both!

Seven warning signs Robert L Park is a Professor of Physics at the University of Maryland He has identified seven warning signs that indicate that a scientific claim is bogus. Remember, these are only warning signs – they are not infallible.

1. Pitching to the media The integrity of science rests on the willingness of scientists to expose new ideas and findings to the scrutiny of other scientists before anyone else. This is called ‘peer review’. Going to the media first suggests that the work is unlikely to stand up to peer review. Eg. Pons & Fleischmann, 1989.

2. Conspiracy and cover-ups The discoverer says that a powerful establishment is trying to suppress his or her work. Mainstream science is made out to be a part of a larger conspiracy that includes industry and the government. Examples: GM potatoes, cold fusion.

3. At the limit of detection The scientific effect is said to be very hard to detect. All scientific measurements must contend with some level of background noise. But if the signal-to-noise ratio cannot be improved, even in principle, the effect is probably not real. Examples: parapsychology.

4. Evidence is anecdotal Anecdotes and narrative have a very strong emotional impact. Statistics have a weak emotional impact. Frauds tend to use anecdotes and not statistics such as RCTs. Scientists have to learn to put statistics above anecdote. Eg. bloodletting.

5. Ancient wisdom The discoverer says a belief is credible because it has endured for centuries. ‘Our ancestors possessed wonderful advanced knowledge that has been forgotten.’ Eg. crystal healing. Science was born when people began to reject blind tradition in favour of experiment and observation (eg. Bacon)

6. Splendid isolation The image of the lone genius who struggles in secrecy in an attic laboratory is rarely accurate. Scientific breakthroughs nowadays are almost always syntheses of the work of many scientists.

7. New laws of nature The discoverer proposes a new law of nature that conflicts with what is already well-established. Weigh up the evidence for each alternative: eg. homeopathy and the meta-analysis in the BMJ.

On the other hand… Sometimes a scientist MAY go to the media first! Sometimes there ARE powerful forces trying to suppress a good idea! Some scientific effects ARE very hard to detect!

On the other hand… (contd) Anecdotal evidence is better than NO evidence at all! Some ancient knowledge HAS been forgotten! Some scientists DO work in isolation! Sometimes a new law of nature IS correct!

So how can we tell? No ‘recipe’ or ‘algorithm’ for assessing science. No substitute for thinking! But there are some general guidelines on how to be critical. Apply them to your own work and to that of others.

Whenever you read a paper…..(even your own!).. What is the hypothesis being tested? Read the methods section – is there enough information to allow the experiment or observation to be repeated? Read the results section – is there a proper statistical analysis?