CAMARADES: Bringing evidence to translational medicine Study Quality and Publication Bias in Experimental Studies of Neurological Diseases Emily S Sena, Hanna M Vesterinen, Kieren J Egan and Malcolm R Macleod Centre for Clinical Brain Sciences, University of Edinburgh

CAMARADES: Bringing evidence to translational medicine CAMARADES Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies Look systematically across the modelling of a range of conditions Data Repository –7 Neurological Diseases –2500 studies –4700 in vivo experiments –from over 60,000 animals

CAMARADES: Bringing evidence to translational medicine Hypotheses: In the life sciences there are perverse incentives (publication, funding, promotion) to produce positive results with little attention paid to their validity In the use of animal disease models, pressure to reduce the number of animals (cost, time, ethics, feasibility) results in studies either being underpowered or of unknown power These factors combine to compromise the utility of animal models and contribute to translational failure

CAMARADES: Bringing evidence to translational medicine Outline Translational failure Internal validity of experiments –Reporting of measures to minimise bias External validity of experiments Publication bias

CAMARADES: Bringing evidence to translational medicine Outline Translational failure Internal validity of experiments –Reporting of measures to minimise bias External validity of experiments Publication bias

CAMARADES: Bringing evidence to translational medicine interventions in experimental stroke Tested in experiments OCollins et al, 2006

CAMARADES: Bringing evidence to translational medicine interventions in experimental stroke Tested in focal ischaemia OCollins et al, 2006

CAMARADES: Bringing evidence to translational medicine interventions in experimental stroke Effective in focal ischaemia OCollins et al, 2006

CAMARADES: Bringing evidence to translational medicine interventions in experimental stroke Tested in clinical trial OCollins et al, 2006

CAMARADES: Bringing evidence to translational medicine interventions in experimental stroke Effective in clinical trial OCollins et al, 2006

CAMARADES: Bringing evidence to translational medicine Animal data in stroke There are huge amounts of often confusing data Systematic review can help to make sense of it If you select extreme bits of the evidence you can prove either harm or substantial benefit Investigating the sources behind this variation may be helpful in translation Hypothermia: a systematic search identified 222 experiments in 3353 animals Better Worse Van der Worp et al Brain 2007

CAMARADES: Bringing evidence to translational medicine Outline Translational failure Internal validity of experiments –Reporting of measures to minimise bias External validity of experiments Publication bias

CAMARADES: Bringing evidence to translational medicine Potential sources of bias in animal studies Internal validity External validity –Are the models we use good models? –Publication bias ProblemSolution Selection BiasRandomisation Performance BiasAllocation Concealment Detection BiasBlinded outcome assessment Attrition biasReporting drop-outs/ ITT analysis

CAMARADES: Bringing evidence to translational medicine Internal Validity NXY-059 in Stroke Infarct Volume –11 publications –408 animals –Improved outcome by 44% (35-53%) Macleod et al, 2008

CAMARADES: Bringing evidence to translational medicine Internal validity in PD models Blinded outcome assessment Rooke et al, Publications 601 Experiments 4181 Animals

CAMARADES: Bringing evidence to translational medicine 1117 publications 388 report neurobehavioural outcome for 38 interventions tested >5 127 included in meta-analysis for 36 interventions Study quality Experimental Autoimmune Encephamyelitis (EAE) (Vesterinen et al, 2010) 255 missing essential data

CAMARADES: Bringing evidence to translational medicine EAE : Sample Size

CAMARADES: Bringing evidence to translational medicine Chances that data from any given animal will be non-contributory Number of animalsPower% animals wasted 418.6%81.4% 832.3%67.7% %43.6% %14.9% assume simple two group experiment seeking 30% reduction in infarct volume, observed SD 40% of control infarct volume

CAMARADES: Bringing evidence to translational medicine How do models of neurological disorders compare? Randomisation Blinded Outcome Assessment Sample Size calculation Stroke 36%29%3% MND 31%20%<1% AD 15%25%0% PD 12%15%0% EAE 8%15%<1% Glioma 14%0% Sena et al TiNS 2007

CAMARADES: Bringing evidence to translational medicine Reporting of measures to avoid bias Sample size calculation 2/311 (1%) Randomisation 46/292 (16%) Blinding 46/312 (15%)

CAMARADES: Bringing evidence to translational medicine

Outline Translational failure Internal validity of experiments –Reporting of measures to minimise bias External validity of experiments Publication bias

CAMARADES: Bringing evidence to translational medicine External Validity Hypertension in studies of tPA in experimental stroke Comorbidity Normal BP Efficacy -2% 25% Sena et al JCBFM 2010 High prevalence of hypertension in patients with stroke tPA is substantially less effective in hypertensive animals

CAMARADES: Bringing evidence to translational medicine External Validity Simulation of the asymptomatic phase of AD

CAMARADES: Bringing evidence to translational medicine External Validity: Time to Treatment in EAE Median: 0 days (IQR -11 to 4) 1% did not report time of administration Before EAE48% Day of Induction22% After EAE30% Day of Symptom Onset2%

CAMARADES: Bringing evidence to translational medicine Outline Translational failure Internal validity of experiments –Reporting of measures to minimise bias External validity of experiments Publication bias

CAMARADES: Bringing evidence to translational medicine Publication Bias Neutral and negative studies –remain unpublished –less likely to be identified in systematic review –leads to the overstatement of efficacy in meta-analysis.

CAMARADES: Bringing evidence to translational medicine

Publication bias in experimental stroke Trim and Fill suggested 16% of experiments remain unpublished Best estimate of magnitude of problem –Overstatement of efficacy 31% Only 2% publications reported no significant treatment effects

CAMARADES: Bringing evidence to translational medicine Hypotheses: In the life sciences there are perverse incentives (publication, funding, promotion) to produce positive results with little attention paid to their validity In the use of animal disease models, pressure to reduce the number of animals (cost, time, ethics, feasibility) results in studies either being underpowered or of unknown power These factors combine to compromise the utility of animal models and contribute to translational failure

CAMARADES: Bringing evidence to translational medicine What happens … Small (underpowered), poorly conducted (randomisation, blinding) studies reach spurious (falsely positive) conclusions but are published because they are seen to be interesting. Small (perhaps) poorly conducted (sometimes) studies not reaching the same conclusions are not published. Investigators become conditioned by the apparent success that comes from conducting small underpowered studies Investigators keep trying to replicate the positive studies

CAMARADES: Bringing evidence to translational medicine What should we do? Be rigorous in demanding the highest quality standards in the conduct and reporting of studies Develop model specific GLP guidelines Develop a registry of animal studies to prevent unnecessary replication Where effect sizes are small (preclinical testing), develop tools for multicentre animal studies

CAMARADES: Bringing evidence to translational medicine Thanks to … Edinburgh –Peter Sandercock Utrecht –Bart van der Worp Melbourne –David Howells –Geoff Donnan Nottingham –Philip Bath