1. Modelling Stroke in the Laboratory - Separating Fact from Artefact The impact of sources of bias in animal models of neurological disease, and what we should do about it Malcolm Macleod Senior Lecturer, Centre for Clinical Brain Sciences University of Edinburgh

2. Declaration of Interests

3. 1026 1026 interventions in experimental stroke OCollins et al Ann Neurol 2006

4. 1026 603 1026 interventions in experimental stroke Tested in focal ischaemia OCollins et al Ann Neurol 2006

5. 1026 883 374 1026 interventions in experimental stroke Effective in focal ischaemia OCollins et al Ann Neurol 2006

6. 1026 883 550 9718 1026 interventions in experimental stroke Tested in clinical trial OCollins et al Ann Neurol 2006

7. 1026 883 550 9717 13 1026 interventions in experimental stroke Effective in clinical trial OCollins et al Ann Neurol 2006

8. Where are we going wrong? Are animal experiments falsely positive? Have clinical trials tested the conditions of maximum efficacy? … and what, if anything, does this mean for models of other diseases? Background

9. 10 -120 M10 -60 M

10. Animal data in stroke There are huge amounts of often confusing data Systematic review can help to make sense of it If you select extreme bits of the evidence you can prove either harm or substantial benefit However, if you have a precise and highly significant overall effect, then it is probably real Hypothermia 101 publications 277 experiments 3353 animals Better Worse van der Worp et al Brain 2007

11. Potential sources of bias in animal studies Internal validity –Low sample size External validity –Publication bias –Are the models we use good models? Co-morbidities ProblemSolution Selection BiasRandomisation Performance BiasAllocation Concealment Detection BiasBlinded outcome assessment Attrition biasReporting drop-outs/ ITT analysis Crossley et al, Stroke 2008

12. Caviat

13. Internal Validity NXY-059 Macleod et al, Stroke 2008 9 publications 29 experiments 408 animals Improved outcome by 44% (35-53%)

14. Internal Validity Hypothermia van der Worp et al Brain 2007 Randomisation Yes No Blinded outcome assessment Yes No 101 publications 277 experiments 3353 animals Improved outcome by 44% (35-53%)

15. Internal Validity Stem Cell based therapies Infarct Volume Neurobehavioural score: Jen Lees, unpublished 54 publications 127 experiments 2012 animals Improved outcome by 29% (25-33%) 72 publications 111 experiments 1876 animals Improved outcome by 34% (30-39%)

16. Randomisation Stem Cell based therapies Jen Lees, unpublished

17. Blinded outcome assessment Stem Cell based therapies Jen Lees, unpublished

18. Internal Validity Meta-analysis of meta-analyses Crossley et al, Stroke, 2008 Blinded induction of ischaemia (Allocation Concealment)

19. Internal Validity Meta-analysis of meta-analyses Crossley et al, Stroke 2008 Co-morbidity

20. What does this mean? Modelling the efficacy of tPA Standard Healthy Male Rat No randomisation Halothane anaesthesia Quantify infarct volume with TTC 25% Co-morbid Hypertensive Male Rat No randomisation Halothane anaesthesia Quantify infarct volume with TTC 12% Randomised Hypertensive Male Rat Randomised Halothane anaesthesia Quantify infarct volume with TTC 0% Emily Sena, In Preparation

21. Reported Efficacy36% Corrected Efficacy<0%

22. Comparing interventions Modelling efficacy under standard conditions

23. MCA Occlusion in Cats 0 1000 2000 3000 4000 Volume(mm 3 ) Cerebral Hemisphere Cerebral Cortex Caudate Nucleus MK-801 Vehicle ** ** p < 0.01 Representations of uncertainty

24. … are somewhat cosmetic 95% CI95%5% MeasureInsideOutside s.e.m.54%46% SD72%28%

25. Sample size calculations If (if) our data are normally distributed, and If we have a simple two group comparison, and If we know our variance then There is a predictable relationship between –the number of animals included; –the Type I error (α) we are prepared to accept; –the expected difference between groups; and –the risk of falsely concluding that there is no such effect (Type II error, β, 1-power)

26. Type II error, β, (1-power) The risk of falsely concluding that a biological effect is not present because the study was not large enough reliably to detect such differences. The smaller the experiment, the greater the risk of a Type II error Small studies with increased risk of Type II errors –waste animals, time and money –may lead to avenues of research being closed down inappropriately.

27. Chances that data from any given animal will be non-contributory Number of animalsPower% animals wasted 418.6%81.4% 832.3%67.7% 1656.4%43.6% 3285.1%14.9% assume simple two group experiment seeking 30% reduction in infarct volume, observed SD 40% of control infarct volume

28. Chances of wasting an animal

29. How does stroke compare? Randomisation Blinded Outcome Assessment Sample Size calculation Stroke36%29%3% MND31%20%<1% PD12%15%0% EAE8%15%<1% Glioma14%0% Sena et al 2007 TiNS; Amarasingh et al, J NOnc in press

30. Efficacy of Dopamine agonists in suppressing induced rotational activity following unilateral 6-OH-DA lesioning Ferguson et al, in preparation Suppression of rotational activity Quality items scored

31. BetterWorse Precision 29 publications 109 experiments 1596 animals Improved outcome by 31% (27-35%) External Validity Publication Bias for FK506 Macleod et al, JCBFM 2005

32. External Validity Publication bias Sena et al, accepted for ESC 2008 991 publications

33. External Validity Hypertension in studies of NXY-059 Macleod et al, Stroke in press 7% of animals studied had hypertension 77% of patients in SAINT II had a history of hypertension at study entry

34. External Validity Hypertension in studies of tPA in experimental stroke Perel et al BMJ 2007 Comorbidity Normal HBP Efficacy -2% 25% 113 publications 212 experiments 3301 animals Improved outcome by 24% (20-28)

35. Summary Certain aspects of the design of animal experiments probably do lead to the over- statement of neuroprotective efficacy A substantial publication bias is present Neuroprotective efficacy may be substantially lower in animals with relevant co-morbidities

36. 26% Publication bias RandomisationCo-morbidity bias 32% Reported efficacy How much efficacy is left? 20% 5%

37. Quality of Translation tPA and tirilazad Both appear to work in animals tPA works in humans but tirilazad doesnt Time to treatment: tPA: –Animals– median 90 minutes –Clinical trial– median 90 minutes Time to treatment: tirilazad –Animals– median 10 minutes –Clinical trial- >3 hrs for >75% of patients Sena et al, Stroke 2007; Perel et al BMJ 2007

38. tPA: Effect of time to treatment on efficacy Perel et al BMJ 2007; Lancet 2004

39. Animal Studies Systematic Review And Meta-analysis how powerful is the treatment? what is the quality of evidence? what is the range of evidence? is there evidence of a publication bias? What are the conditions of maximum efficacy? Clinical Trial Summarising data from animal experiments STAIR VI: possible developments

40. Reverse translation Efficacy of stem cells on different outcome measures Jen Lees, MS in preparation Infarct Volume Neurobehavioural Score

41. Animal Studies Clinical Trial Systematic Review And Meta-analysis

42. Design of a hypothermia RCT Is meta-analysis enough?

43. A toolkit for effective translation Clear, rigorous SOPs for all aspects of experimental design On-line tools for –Sample size calculation –Random allocation to group Curated data repository –? List of interventions tested (~ stroketrials.org) –? Details of individual experiments (~ GENBANK) Development of experimental methods and funding streams to support multi-centre animal studies Adoption of CONSORT statement for animal stroke studies STAIR VI: possible developments

44. Macleod et al, Stroke/JCBFM/IJS in press Sample size calculation Animals used Inclusion and exclusion criteria Randomisation Allocation concealment Reporting of animals excluded from analysis Blinded assessment of outcome Reporting potential conflicts of interest and study funding

45. Reporting standards for animal stroke studies Sample size calculation: The manuscript should describe how the size of the experiment was planned. Animals: The precise species, strain, substrain and source of animals used should be stated. Inclusion and exclusion criteria: Where the severity of ischemia has to reach a certain threshold for inclusion this should be stated clearly. Randomisation: The manuscript should describe the method by which animals were allocated to experimental groups. Allocation concealment: The method of allocation concealment should be described. Reporting of animals excluded from analysis: All randomised animals should be accounted for in the data presented. The criteria for excluding animals from analysis and the number of animals excluded should be reported. Blinded assessment of outcome: The investigator responsible for measuring infarct volume, for scoring neurobehavioural outcome and for determining any other outcome measures should have no knowledge of the experimental group to which an animal belongs. The method of blinding the assessment of outcome should be described. Reporting potential conflicts of interest and study funding: Any relationship which might be perceived to introduce a potential conflict of interest, or the absence of such a relationship, should be disclosed in an acknowledgements section, along with information on study funding and for instance supply of drug or of equipment. Stroke, JCBFM

46. Acknowledgements Emily Sena Peter Sandercock H Bart van der Worp David Howells Tori OCollins Geoff Donnan Nicolas Crossley Ulrich Dirnagl Laura Gray Philip Bath Pablo Perel Ian Roberts

47. Financial support Funding bodyOutcome NHS R&D Methodology Program£12,000 Chief Scientist Office, Scotland£63,000 MS Society£20,000 UK MRCUnsuccessful application (x2) UK NC3RsUnsuccessful application (x2)