1. Summarising the evidence from animal models of neurological disease: Are there any babies in the bathwater? Malcolm Macleod University of Edinburgh

2. Lots of drugs seem to work in animal models of stroke Some of these drugs have been around for ages, and are probably safe Rather than spend billions developing a novel drug, why not test something we already know works in animals? Where we started …

3. What do we already know works in animals? Lots of individual promising studies Little in the way of research synthesis So … Review the field systematically Pick a target and synthesise the evidence

4. 1026 1026 interventions in experimental stroke

5. 1026 603 1026 interventions in experimental stroke Tested in focal ischaemia

6. 1026 883 374 1026 interventions in experimental stroke Effective in focal ischaemia

7. 1026 883 550 9718 1026 interventions in experimental stroke Tested in clinical trial

8. 1026 883 550 9717 13 1026 interventions in experimental stroke Effective in clinical trial

9. Outline Review the internal validity of animal studies Review the external validity of animal studies Explore the potential uses of this approach –Mechanisms research –Biomarker validation –Bibliometrics

10. Methodological approach Written review protocol – hypothesis, inclusion and exclusion criteria, analyses prespecified Standardised search strategy – 3 online databases, conference abstracts, dual screening Data extraction to bespoke database/analysis tool Random effects weighted mean difference stratified meta-analysis or meta-regression Publication bias by Funnel plot, Egger regression, Trim and Fill

11. There is nothing new under the sun … …you will meet with several observations and experiments which, though communicated for true by candid authors or undistrusted eye-witnesses, or perhaps recommended by your own experience, may, upon further trial, disappoint your expectation, either not at all succeeding, or at least varying much from what you expected Robert Boyle (1693) Concerning the Unsuccessfulness of Experiments

12. Focal cerebral ischaemia Emily Sena David Howells Tori OCollins Bart van der Worp Philip Bath

13. Internal validity … the extent to which an experiment accurately describes what happened in that model system may be confounded by Selection Bias(Randomisation) Performance Bias(Allocation Concealment) Detection Bias(Blinded outcome assessment) Attrition bias(Reporting drop-outs/ ITT analysis) False positive report bias(Adequate sample sizes)

14. NXY-059 in animal stroke models 9 publications, 29 experiments, 408 animals 44% (35-53%) improvement in outcome

15. External validity Has the intervention been tested under a range of circumstances similar to those which might be encountered in clinical practice? Are the data you know about representative of all data? (what is the likelihood of publication bias)

16. Hypertension in studies of NXY-059 in experimental stroke Hypertension: – 7% of animal studies – 77% of patients in the (neutral) SAINT II study

17. Hypertension in studies of tPA in experimental stroke Comorbidity Normal BP Efficacy -2% 25% Infarct Volume: –113 publications –212 experiments –3301 animals –Improved outcome by 24% (20-28) Hypertension: –9% of animal studies –Specifically exclusion criterion in (positive) NINDS study

18. The importance of Time to Treatment Both tPA and tirilazad appear to work in animals tPA works in humans but tirilazad doesnt Time to treatment: tPA: –Animals– median 90 minutes –Clinical trial– median 90 minutes Time to treatment: tirilazad –Animals– median 10 minutes –Clinical trial- >3 hrs for >75% of patients

20. Publication bias in experimental stroke Only 11/525 publications (2.2%) reported no significant treatment effects Trim and Fill suggested ~16% (214/1573) of experiments remain unpublished Best estimate of magnitude of problem –Observed efficacy31.3% (29.7-32.8) –Adjusted efficacy 23.8% (22.2-25.5)

21. Publication biasRandomisationCo-morbidity bias Reported efficacy 24% 32% 18% 4%

22. There are multiple drivers of bias

23. Dopamine agonists in PD models Evelien Rooke Hanna Vesterinen Kieren Egan Emily Sena

24. Systematic review

25. Internal validity in PD models Blinded outcome assessmentComposite quality

26. Experimental allergic encaphalomyelitis Hanna Vesterinen Emily Sena

27. Publications identified in review

28. Outcomes reported

29. Efficacy – for what its worth

30. Internal validity

31. External validity

32. Demyelination Axon loss Neurobehaviour 31% 32% 14% Relationship between endpoints

33. Transgenic models of AD Kieren Egan

34. Systematic Review 8495 409 381 Full publications 284 papers have histological data extracted 69 Papers have data on the MWM extracted 28 Abstracts 8086 Excluded

35. Study Quality Randomisation Blinded Outcome Assessment Sample Size calculation Stroke36%29%3% MND31%20%<1% AD14%22%0% PD16%15%<1% EAE8%16%<1% Glioma14%0%

36. Is there a relationship between efficacy for Aß 40 and Aß 42? Amyloid beta 42 = 0.5321 x amyloid beta 40 + 0.3685

38. Efficacy is higher in younger animals

39. Summarising data from more fundamental research The interval validity of findings –Are they the product of bias? The validity of research summaries –Have the authors of pivotal reviews considered all relevant data? The external validity of findings –Do these findings hold only in limited highly controlled situations, or only one species, or are they generally applicable across biology?

40. Perplexing pathways

41. The added value of a systematic approach … Identifying, and explaining, heterogeneity Developing a systematic evidence base for understanding biological pathways Providing the components for mathematical models predicting the response to interventions

42. The role of Th17 cells in the development of EAE

43. Immunisation EAE Th17 Th1 CD4+ -=+ -6 =3 +61 EAE Th1 -=+ -111 =1 +22 EAE Th17

44. Surrogate outcomes and biomarkers

45. Surrogate outcome Functional outcome

46. Can animal models help? In animal studies –Does structural outcome predict functional outcome? –Does this relationship hold across interventions? Inclusion criteria: –Reports of the efficacy of a candidate stroke drug in an animal model of focal cerebral ischaemia –Structural and functional outcome reported from the same cohort of animals –Simultaneous measurement of structural and functional outcome

47. Data 1047 257 16 71 131 71 21 25 110 13 110 12 24 18 26 9 Experiments reporting any outcome 299TOTAL 58Stem Cells 11Other Thrombolytics 20Nicotinamide 54Thrombolytics 16NOS Inhibitors 8FK506 0NOS Donors 42Hypothermia 4Melatonin 51Growth Factors 3NXY-059 7Enriched Environment 16Tirilazad 8Minocycline 1IL1-RA Experiments reporting structural and functional outcomes Drug Group

48. Raw correlation Raw correlation coefficient = 0.439 adjusted r 2 =0.301 Functional outcome Structural outcome Better Worse

49. Structural Outcome Coefficient95% CI Constant (tPA)23.917.1-30.7 Other lytics19.32.93-35.7 FK50619.91.62-38.1 Tirilazad21.47.91-34.9 Nicotinamide23.46.75-40.2 Hypothermia26.115.9-36.4 NXY 05929.10.43-57.8 Delay to assessment (days)-0.01-0.02-0.00 % improvement (SO) = 23.9 + f(drug) -0.01*(days to assessment) Adjusted r 2 = 0.24

50. Functional Outcome Coefficient95% CI Structural outcome (Infarct Volume)0.470.37 to 0.56 Stem cells10.01.9 to 18.1 Hypothermia11.22.4 to 19.9 FK50618.44.2 to 32.6 Minocycline18.55.1 to 31.9 Nicotinamide19.910.1 to 29.7 NOS Inhibitors20.710.4 to 30.9 Tirilazad22.98.1 to 37.9 Delay to assessment (days)0.580.36 to 0.82 Delay to treatment (hours)-0.96-1.44 to -0.48 % improvement (FO) = 0.47*SO + f(drug) + 0.58*(days to assessment) – 0.96*(hours to drug administration) Adjusted r 2 = 0.56

51. Summary Structural OutcomeFunctional OutcomeIndependent effects on both tPAStem cellsFK506 Other lyticsMinocyclineTirilazad NXY 059NOS InhibitionNicotinamide Hypothermia Delay to treatment (-ve) Delay to assessment (-ve SO, +ve FO) Lesion Structural Outcome Functional Outcome

52. Summary Structural OutcomeFunctional OutcomeIndependent effects on both tPAStem cellsFK506 Other lyticsMinocyclineTirilazad NXY 059NOS InhibitionNicotinamide Hypothermia Delay to treatment (-ve) Delay to assessment (-ve SO, +ve FO) Lesion Structural Outcome Functional Outcome

53. Summary Structural OutcomeFunctional OutcomeIndependent effects on both tPAStem cellsFK506 Other lyticsMinocyclineTirilazad NXY 059NOS InhibitionNicotinamide Hypothermia Delay to treatment (-ve) Delay to assessment (-ve SO, +ve FO) Lesion Structural Outcome Functional Outcome

54. Summary Structural OutcomeFunctional OutcomeIndependent effects on both tPAStem cellsFK506 Other lyticsMinocyclineTirilazad NXY 059NOS InhibitionNicotinamide Hypothermia Delay to treatment (-ve) Delay to assessment (-ve SO, +ve FO) Lesion Structural Outcome Functional Outcome

55. Key findings Structural outcome explains around 30% of the variation in functional outcome Crucially, this relationship is different for different interventions Surrogate outcome measures in Phase II are likely to be intervention specific rather than disease specific Nonetheless, this approach might help with rational selection of combination therapies

56. Summary The internal and external validity of animal experiments is limited Conclusions from such experiments are confounded by many biases There is no reason to believe that more pathophysiological experiments are any better

57. Does journal impact factor reflect validity of the presented work? 500 publications in focal cerebral ischaemia Modelling of Impact factor IF = 3.7 + 2.4(Conflict of Interest Statement) + 1.2(Allocation Concealment): adjusted r 2 = 0.06

58. Future directions Qualitative systematic research Quantitative systematic research –Study quality issues –Evidence based pathways –Strategic research targeting

59. Are there any babies in the bathwater? MRC Trials Methodology Hub Chief Scientist Office MS Society NHS R&D Methodology Program