Summarising the evidence from animal models of neurological disease: Are there any babies in the bathwater? Malcolm Macleod University of Edinburgh

Lots of drugs seem to work in animal models of stroke Some of these drugs have been around for ages, and are probably safe Rather than spend billions developing a novel drug, why not test something we already know works in animals? Where we started …

What do we already know works in animals? Lots of individual promising studies Little in the way of research synthesis So … Review the field systematically Pick a target and synthesise the evidence

interventions in experimental stroke

interventions in experimental stroke Tested in focal ischaemia

interventions in experimental stroke Effective in focal ischaemia

interventions in experimental stroke Tested in clinical trial

interventions in experimental stroke Effective in clinical trial

Outline Review the internal validity of animal studies Review the external validity of animal studies Explore the potential uses of this approach –Mechanisms research –Biomarker validation –Bibliometrics

Methodological approach Written review protocol – hypothesis, inclusion and exclusion criteria, analyses prespecified Standardised search strategy – 3 online databases, conference abstracts, dual screening Data extraction to bespoke database/analysis tool Random effects weighted mean difference stratified meta-analysis or meta-regression Publication bias by Funnel plot, Egger regression, Trim and Fill

There is nothing new under the sun … …you will meet with several observations and experiments which, though communicated for true by candid authors or undistrusted eye-witnesses, or perhaps recommended by your own experience, may, upon further trial, disappoint your expectation, either not at all succeeding, or at least varying much from what you expected Robert Boyle (1693) Concerning the Unsuccessfulness of Experiments

Focal cerebral ischaemia Emily Sena David Howells Tori OCollins Bart van der Worp Philip Bath

Internal validity … the extent to which an experiment accurately describes what happened in that model system may be confounded by Selection Bias(Randomisation) Performance Bias(Allocation Concealment) Detection Bias(Blinded outcome assessment) Attrition bias(Reporting drop-outs/ ITT analysis) False positive report bias(Adequate sample sizes)

NXY-059 in animal stroke models 9 publications, 29 experiments, 408 animals 44% (35-53%) improvement in outcome

External validity Has the intervention been tested under a range of circumstances similar to those which might be encountered in clinical practice? Are the data you know about representative of all data? (what is the likelihood of publication bias)

Hypertension in studies of NXY-059 in experimental stroke Hypertension: – 7% of animal studies – 77% of patients in the (neutral) SAINT II study

Hypertension in studies of tPA in experimental stroke Comorbidity Normal BP Efficacy -2% 25% Infarct Volume: –113 publications –212 experiments –3301 animals –Improved outcome by 24% (20-28) Hypertension: –9% of animal studies –Specifically exclusion criterion in (positive) NINDS study

The importance of Time to Treatment Both tPA and tirilazad appear to work in animals tPA works in humans but tirilazad doesnt Time to treatment: tPA: –Animals– median 90 minutes –Clinical trial– median 90 minutes Time to treatment: tirilazad –Animals– median 10 minutes –Clinical trial- >3 hrs for >75% of patients

Publication bias in experimental stroke Only 11/525 publications (2.2%) reported no significant treatment effects Trim and Fill suggested ~16% (214/1573) of experiments remain unpublished Best estimate of magnitude of problem –Observed efficacy31.3% ( ) –Adjusted efficacy 23.8% ( )

Publication biasRandomisationCo-morbidity bias Reported efficacy 24% 32% 18% 4%

There are multiple drivers of bias

Dopamine agonists in PD models Evelien Rooke Hanna Vesterinen Kieren Egan Emily Sena

Systematic review

Internal validity in PD models Blinded outcome assessmentComposite quality

Experimental allergic encaphalomyelitis Hanna Vesterinen Emily Sena

Publications identified in review

Outcomes reported

Efficacy – for what its worth

Internal validity

External validity

Demyelination Axon loss Neurobehaviour 31% 32% 14% Relationship between endpoints

Transgenic models of AD Kieren Egan

Systematic Review Full publications 284 papers have histological data extracted 69 Papers have data on the MWM extracted 28 Abstracts 8086 Excluded

Study Quality Randomisation Blinded Outcome Assessment Sample Size calculation Stroke36%29%3% MND31%20%<1% AD14%22%0% PD16%15%<1% EAE8%16%<1% Glioma14%0%

Is there a relationship between efficacy for Aß 40 and Aß 42? Amyloid beta 42 = x amyloid beta

Efficacy is higher in younger animals

Summarising data from more fundamental research The interval validity of findings –Are they the product of bias? The validity of research summaries –Have the authors of pivotal reviews considered all relevant data? The external validity of findings –Do these findings hold only in limited highly controlled situations, or only one species, or are they generally applicable across biology?

Perplexing pathways

The added value of a systematic approach … Identifying, and explaining, heterogeneity Developing a systematic evidence base for understanding biological pathways Providing the components for mathematical models predicting the response to interventions

The role of Th17 cells in the development of EAE

Immunisation EAE Th17 Th1 CD4+ -=+ -6 =3 +61 EAE Th1 -= =1 +22 EAE Th17

Surrogate outcomes and biomarkers

Surrogate outcome Functional outcome

Can animal models help? In animal studies –Does structural outcome predict functional outcome? –Does this relationship hold across interventions? Inclusion criteria: –Reports of the efficacy of a candidate stroke drug in an animal model of focal cerebral ischaemia –Structural and functional outcome reported from the same cohort of animals –Simultaneous measurement of structural and functional outcome

Data Experiments reporting any outcome 299TOTAL 58Stem Cells 11Other Thrombolytics 20Nicotinamide 54Thrombolytics 16NOS Inhibitors 8FK506 0NOS Donors 42Hypothermia 4Melatonin 51Growth Factors 3NXY-059 7Enriched Environment 16Tirilazad 8Minocycline 1IL1-RA Experiments reporting structural and functional outcomes Drug Group

Raw correlation Raw correlation coefficient = adjusted r 2 =0.301 Functional outcome Structural outcome Better Worse

Structural Outcome Coefficient95% CI Constant (tPA) Other lytics FK Tirilazad Nicotinamide Hypothermia NXY Delay to assessment (days) % improvement (SO) = f(drug) -0.01*(days to assessment) Adjusted r 2 = 0.24

Functional Outcome Coefficient95% CI Structural outcome (Infarct Volume) to 0.56 Stem cells to 18.1 Hypothermia to 19.9 FK to 32.6 Minocycline to 31.9 Nicotinamide to 29.7 NOS Inhibitors to 30.9 Tirilazad to 37.9 Delay to assessment (days) to 0.82 Delay to treatment (hours) to % improvement (FO) = 0.47*SO + f(drug) *(days to assessment) – 0.96*(hours to drug administration) Adjusted r 2 = 0.56

Summary Structural OutcomeFunctional OutcomeIndependent effects on both tPAStem cellsFK506 Other lyticsMinocyclineTirilazad NXY 059NOS InhibitionNicotinamide Hypothermia Delay to treatment (-ve) Delay to assessment (-ve SO, +ve FO) Lesion Structural Outcome Functional Outcome

Summary Structural OutcomeFunctional OutcomeIndependent effects on both tPAStem cellsFK506 Other lyticsMinocyclineTirilazad NXY 059NOS InhibitionNicotinamide Hypothermia Delay to treatment (-ve) Delay to assessment (-ve SO, +ve FO) Lesion Structural Outcome Functional Outcome

Summary Structural OutcomeFunctional OutcomeIndependent effects on both tPAStem cellsFK506 Other lyticsMinocyclineTirilazad NXY 059NOS InhibitionNicotinamide Hypothermia Delay to treatment (-ve) Delay to assessment (-ve SO, +ve FO) Lesion Structural Outcome Functional Outcome

Summary Structural OutcomeFunctional OutcomeIndependent effects on both tPAStem cellsFK506 Other lyticsMinocyclineTirilazad NXY 059NOS InhibitionNicotinamide Hypothermia Delay to treatment (-ve) Delay to assessment (-ve SO, +ve FO) Lesion Structural Outcome Functional Outcome

Key findings Structural outcome explains around 30% of the variation in functional outcome Crucially, this relationship is different for different interventions Surrogate outcome measures in Phase II are likely to be intervention specific rather than disease specific Nonetheless, this approach might help with rational selection of combination therapies

Summary The internal and external validity of animal experiments is limited Conclusions from such experiments are confounded by many biases There is no reason to believe that more pathophysiological experiments are any better

Does journal impact factor reflect validity of the presented work? 500 publications in focal cerebral ischaemia Modelling of Impact factor IF = (Conflict of Interest Statement) + 1.2(Allocation Concealment): adjusted r 2 = 0.06

Future directions Qualitative systematic research Quantitative systematic research –Study quality issues –Evidence based pathways –Strategic research targeting

Are there any babies in the bathwater? MRC Trials Methodology Hub Chief Scientist Office MS Society NHS R&D Methodology Program