Modelling Stroke in the Laboratory - Separating Fact from Artefact The impact of sources of bias in animal models of neurological disease, and what we should do about it Malcolm Macleod Senior Lecturer, Centre for Clinical Brain Sciences University of Edinburgh

Declaration of Interests

interventions in experimental stroke OCollins et al Ann Neurol 2006

interventions in experimental stroke Tested in focal ischaemia OCollins et al Ann Neurol 2006

interventions in experimental stroke Effective in focal ischaemia OCollins et al Ann Neurol 2006

interventions in experimental stroke Tested in clinical trial OCollins et al Ann Neurol 2006

interventions in experimental stroke Effective in clinical trial OCollins et al Ann Neurol 2006

Where are we going wrong? Are animal experiments falsely positive? Have clinical trials tested the conditions of maximum efficacy? … and what, if anything, does this mean for models of other diseases? Background

M M

Animal data in stroke There are huge amounts of often confusing data Systematic review can help to make sense of it If you select extreme bits of the evidence you can prove either harm or substantial benefit However, if you have a precise and highly significant overall effect, then it is probably real Hypothermia 101 publications 277 experiments 3353 animals Better Worse van der Worp et al Brain 2007

Potential sources of bias in animal studies Internal validity –Low sample size External validity –Publication bias –Are the models we use good models? Co-morbidities ProblemSolution Selection BiasRandomisation Performance BiasAllocation Concealment Detection BiasBlinded outcome assessment Attrition biasReporting drop-outs/ ITT analysis Crossley et al, Stroke 2008

Caviat

Internal Validity NXY-059 Macleod et al, Stroke publications 29 experiments 408 animals Improved outcome by 44% (35-53%)

Internal Validity Hypothermia van der Worp et al Brain 2007 Randomisation Yes No Blinded outcome assessment Yes No 101 publications 277 experiments 3353 animals Improved outcome by 44% (35-53%)

Internal Validity Stem Cell based therapies Infarct Volume Neurobehavioural score: Jen Lees, unpublished 54 publications 127 experiments 2012 animals Improved outcome by 29% (25-33%) 72 publications 111 experiments 1876 animals Improved outcome by 34% (30-39%)

Randomisation Stem Cell based therapies Jen Lees, unpublished

Blinded outcome assessment Stem Cell based therapies Jen Lees, unpublished

Internal Validity Meta-analysis of meta-analyses Crossley et al, Stroke, 2008 Blinded induction of ischaemia (Allocation Concealment)

Internal Validity Meta-analysis of meta-analyses Crossley et al, Stroke 2008 Co-morbidity

What does this mean? Modelling the efficacy of tPA Standard Healthy Male Rat No randomisation Halothane anaesthesia Quantify infarct volume with TTC 25% Co-morbid Hypertensive Male Rat No randomisation Halothane anaesthesia Quantify infarct volume with TTC 12% Randomised Hypertensive Male Rat Randomised Halothane anaesthesia Quantify infarct volume with TTC 0% Emily Sena, In Preparation

Reported Efficacy36% Corrected Efficacy<0%

Comparing interventions Modelling efficacy under standard conditions

MCA Occlusion in Cats Volume(mm 3 ) Cerebral Hemisphere Cerebral Cortex Caudate Nucleus MK-801 Vehicle ** ** p < 0.01 Representations of uncertainty

… are somewhat cosmetic 95% CI95%5% MeasureInsideOutside s.e.m.54%46% SD72%28%

Sample size calculations If (if) our data are normally distributed, and If we have a simple two group comparison, and If we know our variance then There is a predictable relationship between –the number of animals included; –the Type I error (α) we are prepared to accept; –the expected difference between groups; and –the risk of falsely concluding that there is no such effect (Type II error, β, 1-power)

Type II error, β, (1-power) The risk of falsely concluding that a biological effect is not present because the study was not large enough reliably to detect such differences. The smaller the experiment, the greater the risk of a Type II error Small studies with increased risk of Type II errors –waste animals, time and money –may lead to avenues of research being closed down inappropriately.

Chances that data from any given animal will be non-contributory Number of animalsPower% animals wasted 418.6%81.4% 832.3%67.7% %43.6% %14.9% assume simple two group experiment seeking 30% reduction in infarct volume, observed SD 40% of control infarct volume

Chances of wasting an animal

How does stroke compare? Randomisation Blinded Outcome Assessment Sample Size calculation Stroke36%29%3% MND31%20%<1% PD12%15%0% EAE8%15%<1% Glioma14%0% Sena et al 2007 TiNS; Amarasingh et al, J NOnc in press

Efficacy of Dopamine agonists in suppressing induced rotational activity following unilateral 6-OH-DA lesioning Ferguson et al, in preparation Suppression of rotational activity Quality items scored

BetterWorse Precision 29 publications 109 experiments 1596 animals Improved outcome by 31% (27-35%) External Validity Publication Bias for FK506 Macleod et al, JCBFM 2005

External Validity Publication bias Sena et al, accepted for ESC publications

External Validity Hypertension in studies of NXY-059 Macleod et al, Stroke in press 7% of animals studied had hypertension 77% of patients in SAINT II had a history of hypertension at study entry

External Validity Hypertension in studies of tPA in experimental stroke Perel et al BMJ 2007 Comorbidity Normal HBP Efficacy -2% 25% 113 publications 212 experiments 3301 animals Improved outcome by 24% (20-28)

Summary Certain aspects of the design of animal experiments probably do lead to the over- statement of neuroprotective efficacy A substantial publication bias is present Neuroprotective efficacy may be substantially lower in animals with relevant co-morbidities

26% Publication bias RandomisationCo-morbidity bias 32% Reported efficacy How much efficacy is left? 20% 5%

Quality of Translation tPA and tirilazad Both appear to work in animals tPA works in humans but tirilazad doesnt Time to treatment: tPA: –Animals– median 90 minutes –Clinical trial– median 90 minutes Time to treatment: tirilazad –Animals– median 10 minutes –Clinical trial- >3 hrs for >75% of patients Sena et al, Stroke 2007; Perel et al BMJ 2007

tPA: Effect of time to treatment on efficacy Perel et al BMJ 2007; Lancet 2004

Animal Studies Systematic Review And Meta-analysis how powerful is the treatment? what is the quality of evidence? what is the range of evidence? is there evidence of a publication bias? What are the conditions of maximum efficacy? Clinical Trial Summarising data from animal experiments STAIR VI: possible developments

Reverse translation Efficacy of stem cells on different outcome measures Jen Lees, MS in preparation Infarct Volume Neurobehavioural Score

Animal Studies Clinical Trial Systematic Review And Meta-analysis

Design of a hypothermia RCT Is meta-analysis enough?

A toolkit for effective translation Clear, rigorous SOPs for all aspects of experimental design On-line tools for –Sample size calculation –Random allocation to group Curated data repository –? List of interventions tested (~ stroketrials.org) –? Details of individual experiments (~ GENBANK) Development of experimental methods and funding streams to support multi-centre animal studies Adoption of CONSORT statement for animal stroke studies STAIR VI: possible developments

Macleod et al, Stroke/JCBFM/IJS in press Sample size calculation Animals used Inclusion and exclusion criteria Randomisation Allocation concealment Reporting of animals excluded from analysis Blinded assessment of outcome Reporting potential conflicts of interest and study funding

Reporting standards for animal stroke studies Sample size calculation: The manuscript should describe how the size of the experiment was planned. Animals: The precise species, strain, substrain and source of animals used should be stated. Inclusion and exclusion criteria: Where the severity of ischemia has to reach a certain threshold for inclusion this should be stated clearly. Randomisation: The manuscript should describe the method by which animals were allocated to experimental groups. Allocation concealment: The method of allocation concealment should be described. Reporting of animals excluded from analysis: All randomised animals should be accounted for in the data presented. The criteria for excluding animals from analysis and the number of animals excluded should be reported. Blinded assessment of outcome: The investigator responsible for measuring infarct volume, for scoring neurobehavioural outcome and for determining any other outcome measures should have no knowledge of the experimental group to which an animal belongs. The method of blinding the assessment of outcome should be described. Reporting potential conflicts of interest and study funding: Any relationship which might be perceived to introduce a potential conflict of interest, or the absence of such a relationship, should be disclosed in an acknowledgements section, along with information on study funding and for instance supply of drug or of equipment. Stroke, JCBFM

Acknowledgements Emily Sena Peter Sandercock H Bart van der Worp David Howells Tori OCollins Geoff Donnan Nicolas Crossley Ulrich Dirnagl Laura Gray Philip Bath Pablo Perel Ian Roberts

Financial support Funding bodyOutcome NHS R&D Methodology Program£12,000 Chief Scientist Office, Scotland£63,000 MS Society£20,000 UK MRCUnsuccessful application (x2) UK NC3RsUnsuccessful application (x2)