1. Summarising the evidence from animal models of neurological disease: Publication bias, poor internal validity, and (perhaps) some efficacy Malcolm Macleod University of Edinburgh

2. Lots of drugs seem to work in animal models of stroke Some of these drugs have been around for ages, and are probably safe Rather than spend billions developing a novel drug, why not test something we already know works in animals? Where we started … IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

3. What do we already know works in animals? Lots of individual promising studies Little in the way of research synthesis So … Review the field systematically Pick a target and synthesise the evidence IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

4. 1026 1026 interventions in experimental stroke IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

5. 1026 603 1026 interventions in experimental stroke Tested in focal ischaemia IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

6. 1026 883 374 1026 interventions in experimental stroke Effective in focal ischaemia IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

7. 1026 883 550 9718 1026 interventions in experimental stroke Tested in clinical trial IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

8. 1026 883 550 9717 13 1026 interventions in experimental stroke Effective in clinical trial IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

9. Outline Review the internal validity of animal studies Review the external validity of animal studies Explore the potential uses of this approach in mechanisms research IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

10. Methodological approach Written review protocol – hypothesis, inclusion and exclusion criteria, analyses prespecified Standardised search strategy – 3 online databases, conference abstracts, dual screening Data extraction to bespoke database/analysis tool Random effects weighted mean difference stratified meta-analysis or meta-regression Publication bias by Funnel plot, Egger regression, Trim and Fill IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

11. IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary There is nothing new under the sun … …you will meet with several observations and experiments which, though communicated for true by candid authors or undistrusted eye-witnesses, or perhaps recommended by your own experience, may, upon further trial, disappoint your expectation, either not at all succeeding, or at least varying much from what you expected Robert Boyle (1693) Concerning the Unsuccessfulness of Experiments

12. Internal validity … the extent to which an experiment accurately describes what happened in that model system may be confounded by Selection Bias(Randomisation) Performance Bias(Allocation Concealment) Detection Bias(Blinded outcome assessment) Attrition bias(Reporting drop-outs/ ITT analysis) False positive report bias(Adequate sample sizes) IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

13. NXY-059 in animal stroke models 9 publications, 29 experiments, 408 animals 44% (35-53%) improvement in outcome IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

14. A widespread problem … Dopamine Agonists in 6-OH model of Parkinsons Disease IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

15. Internal validity and Journal Impact Factors Regression model offered: –Effect size –Variance –Randomisation –Allocation concealment –Blinded assessment of outcome –Sample size calculation –Statement of possible conflict of interest Univariate: –Allocation concealment –Blinded assessment of outcome –Statement of possible conflict of interest Multivariate: IF = 3.7 + 2.4(Conflict of Interest Statement) + 1.2(Allocation Concealment): adjusted r 2 = 0.06 IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

16. External validity Has the intervention been tested under a range of circumstances similar to those which might be encountered in clinical practice? Are the data you know about representative of all data? (what is the likelihood of publication bias) IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

17. Hypertension in studies of NXY-059 in experimental stroke Hypertension: – 7% of animal studies – 77% of patients in the (neutral) SAINT II study IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

18. Hypertension in studies of tPA in experimental stroke Comorbidity Normal BP Efficacy -2% 25% Infarct Volume: –113 publications –212 experiments –3301 animals –Improved outcome by 24% (20-28) Hypertension: –9% of animal studies –Specifically exclusion criterion in (positive) NINDS study IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

19. The importance of Time to Treatment Both tPA and tirilazad appear to work in animals tPA works in humans but tirilazad doesnt Time to treatment: tPA: –Animals– median 90 minutes –Clinical trial– median 90 minutes Time to treatment: tirilazad –Animals– median 10 minutes –Clinical trial- >3 hrs for >75% of patients IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

20. IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

21. Publication bias in experimental stroke Only 11/525 publications (2.2%) reported no significant treatment effects Trim and Fill suggested ~16% (214/1573) of experiments remain unpublished Best estimate of magnitude of problem –Observed efficacy31.3% (29.7-32.8) –Adjusted efficacy 23.8% (22.2-25.5) IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

23. Publication bias RandomisationCo-morbidity bias Reported efficacy 24% 32% 18% 4% IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

24. There are multiple drivers of bias IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

25. Summarising data from more fundamental research The interval validity of findings –Are they the product of bias? The validity of research summaries –Have the authors of pivotal reviews considered all relevant data? The external validity of findings –Do these findings hold only in limited highly controlled situations, or only one species, or are they generally applicable across biology? IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

26. Some examples from EAE … External validity Do different endpoints measure different things?

27. Publications identified in review

28. Outcomes reported

29. Efficacy – for what its worth

30. Internal validity

31. External validity

32. Demyelination Axon loss Neurobehaviour 31% 32% 14% Relationship between endpoints

33. IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

34. The added value of a systematic approach … Identifying, and explaining, heterogeneity Developing a systematic evidence base for understanding biological pathways Providing the components for mathematical models predicting the response to interventions IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

35. The role of Th17 cells in the development of EAE IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

36. Immunisation EAE Th17 Th1 CD4+ -=+ -6 =3 +61 EAE Th1 -=+ -111 =1 +22 EAE Th17 IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

37. IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

38. The internal and external validity of animal experiments is limited Conclusions from such experiments are confounded by many biases There is no reason to believe that more pathophysiological experiments are any better IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

39. Future directions Qualitative systematic research Quantitative systematic research –Study quality issues –Evidence based pathways –Strategic research targeting IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary