Summarising the evidence from animal models of neurological disease: Publication bias, poor internal validity, and (perhaps) some efficacy Malcolm Macleod University of Edinburgh

Lots of drugs seem to work in animal models of stroke Some of these drugs have been around for ages, and are probably safe Rather than spend billions developing a novel drug, why not test something we already know works in animals? Where we started … IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

What do we already know works in animals? Lots of individual promising studies Little in the way of research synthesis So … Review the field systematically Pick a target and synthesise the evidence IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

interventions in experimental stroke IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

interventions in experimental stroke Tested in focal ischaemia IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

interventions in experimental stroke Effective in focal ischaemia IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

interventions in experimental stroke Tested in clinical trial IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

interventions in experimental stroke Effective in clinical trial IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Outline Review the internal validity of animal studies Review the external validity of animal studies Explore the potential uses of this approach in mechanisms research IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Methodological approach Written review protocol – hypothesis, inclusion and exclusion criteria, analyses prespecified Standardised search strategy – 3 online databases, conference abstracts, dual screening Data extraction to bespoke database/analysis tool Random effects weighted mean difference stratified meta-analysis or meta-regression Publication bias by Funnel plot, Egger regression, Trim and Fill IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary There is nothing new under the sun … …you will meet with several observations and experiments which, though communicated for true by candid authors or undistrusted eye-witnesses, or perhaps recommended by your own experience, may, upon further trial, disappoint your expectation, either not at all succeeding, or at least varying much from what you expected Robert Boyle (1693) Concerning the Unsuccessfulness of Experiments

Internal validity … the extent to which an experiment accurately describes what happened in that model system may be confounded by Selection Bias(Randomisation) Performance Bias(Allocation Concealment) Detection Bias(Blinded outcome assessment) Attrition bias(Reporting drop-outs/ ITT analysis) False positive report bias(Adequate sample sizes) IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

NXY-059 in animal stroke models 9 publications, 29 experiments, 408 animals 44% (35-53%) improvement in outcome IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

A widespread problem … Dopamine Agonists in 6-OH model of Parkinsons Disease IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Internal validity and Journal Impact Factors Regression model offered: –Effect size –Variance –Randomisation –Allocation concealment –Blinded assessment of outcome –Sample size calculation –Statement of possible conflict of interest Univariate: –Allocation concealment –Blinded assessment of outcome –Statement of possible conflict of interest Multivariate: IF = (Conflict of Interest Statement) + 1.2(Allocation Concealment): adjusted r 2 = 0.06 IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

External validity Has the intervention been tested under a range of circumstances similar to those which might be encountered in clinical practice? Are the data you know about representative of all data? (what is the likelihood of publication bias) IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Hypertension in studies of NXY-059 in experimental stroke Hypertension: – 7% of animal studies – 77% of patients in the (neutral) SAINT II study IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Hypertension in studies of tPA in experimental stroke Comorbidity Normal BP Efficacy -2% 25% Infarct Volume: –113 publications –212 experiments –3301 animals –Improved outcome by 24% (20-28) Hypertension: –9% of animal studies –Specifically exclusion criterion in (positive) NINDS study IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

The importance of Time to Treatment Both tPA and tirilazad appear to work in animals tPA works in humans but tirilazad doesnt Time to treatment: tPA: –Animals– median 90 minutes –Clinical trial– median 90 minutes Time to treatment: tirilazad –Animals– median 10 minutes –Clinical trial- >3 hrs for >75% of patients IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Publication bias in experimental stroke Only 11/525 publications (2.2%) reported no significant treatment effects Trim and Fill suggested ~16% (214/1573) of experiments remain unpublished Best estimate of magnitude of problem –Observed efficacy31.3% ( ) –Adjusted efficacy 23.8% ( ) IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Publication bias RandomisationCo-morbidity bias Reported efficacy 24% 32% 18% 4% IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

There are multiple drivers of bias IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Summarising data from more fundamental research The interval validity of findings –Are they the product of bias? The validity of research summaries –Have the authors of pivotal reviews considered all relevant data? The external validity of findings –Do these findings hold only in limited highly controlled situations, or only one species, or are they generally applicable across biology? IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Some examples from EAE … External validity Do different endpoints measure different things?

Publications identified in review

Outcomes reported

Efficacy – for what its worth

Internal validity

External validity

Demyelination Axon loss Neurobehaviour 31% 32% 14% Relationship between endpoints

IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

The added value of a systematic approach … Identifying, and explaining, heterogeneity Developing a systematic evidence base for understanding biological pathways Providing the components for mathematical models predicting the response to interventions IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

The role of Th17 cells in the development of EAE IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Immunisation EAE Th17 Th1 CD4+ -=+ -6 =3 +61 EAE Th1 -= =1 +22 EAE Th17 IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

The internal and external validity of animal experiments is limited Conclusions from such experiments are confounded by many biases There is no reason to believe that more pathophysiological experiments are any better IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Future directions Qualitative systematic research Quantitative systematic research –Study quality issues –Evidence based pathways –Strategic research targeting IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary