Summarising the evidence from animal models of neurological disease: Publication bias, poor internal validity, and (perhaps) some efficacy Malcolm Macleod.

Slides:



Advertisements
Similar presentations
Study Quality and Publication Bias in Experimental Stroke
Advertisements

Design of Dose Response Clinical Trials
Interim Analysis in Clinical Trials: A Bayesian Approach in the Regulatory Setting Telba Z. Irony, Ph.D. and Gene Pennello, Ph.D. Division of Biostatistics.
The Application of Propensity Score Analysis to Non-randomized Medical Device Clinical Studies: A Regulatory Perspective Lilly Yue, Ph.D.* CDRH, FDA,
CAMARADES: Bringing evidence to translational medicine Study Quality and Publication Bias in Experimental Studies of Neurological Diseases Emily S Sena,
Summarising what we already know – the pivotal role of systematic reviews Malcolm Macleod.
Structural and functional outcomes in animal models of stroke: What do they measure? Malcolm Macleod University of Edinburgh.
Summarising the evidence from animal models of neurological disease: Publication bias, poor internal validity, and (perhaps) some efficacy Malcolm Macleod.
CAMARADES: Bringing evidence to translational medicine Stanford visit 8 Feb 2011.
Modelling Stroke in the Laboratory - Separating Fact from Artefact The impact of sources of bias in animal models of neurological disease, and what we.
Summarising the evidence from animal models of neurological disease: Are there any babies in the bathwater? Malcolm Macleod University of Edinburgh.
Improving the internal validity of experiments in focal ischaemia
Preventing introduction of bias at the bench: from randomizing to experimental results meta-analysis Malcolm Macleod Centre for Clinical Brain Sciences,
Stem cells: (How) do they work? Malcolm Macleod, Jen Lees, Emily Sena, Hanna Vesterinen, Simon Koblar, David Howells.
CAMARADES: Bringing evidence to translational medicine The failure to translate the basic science into therapy is due primarily to inadequacies in the.
CS1512 Foundations of Computing Science 2 Week 3 (CSD week 32) Probability © J R W Hunter, 2006, K van Deemter 2007.
Critical appraisal of research Sarah Lawson
Chapter 7 Sampling and Sampling Distributions
Cost Control and the Menu—Determining Selling Prices and Product Mix
1 Cell-Free Hemoglobin-Based Blood Substitutes and Risk of Myocardial Infarction and Death Natason et al., JAMA, Prepublished online April 28, 2008 at.
AADAPT Workshop South Asia Goa, December 17-21, 2009 Kristen Himelein 1.
CAMARADES: Bringing evidence to translational medicine The effect of anaesthetics on the developing neonatal brain: A systematic review and meta-analysis.
The Bahrain Branch of the UK Cochrane Centre In Collaboration with Reyada Training & Management Consultancy, Dubai-UAE Cochrane Collaboration and Systematic.
PSSA Preparation.
Secondary Data Analysis: Systematic Reviews & Associated Databases
Simple Linear Regression Analysis
Multiple Regression and Model Building
1 A Systematic Review of Cross- vs. Within-Company Cost Estimation Studies Barbara Kitchenham Emilia Mendes Guilherme Travassos.
EVAL 6970: Meta-Analysis Vote Counting, The Sign Test, Power, Publication Bias, and Outliers Dr. Chris L. S. Coryn Spring 2011.
Designs to Estimate Impacts of MSP Projects with Confidence. Ellen Bobronnikov March 29, 2010.
8. Evidence-based management Step 3: Critical appraisal of studies
Systematic reviews of animal studies Malcolm Macleod.
CAMARADES: Bringing evidence to translational medicine Optimizing the Predictive Value of Pre-Clinical Research Session 3: Reviewer Perspective Malcolm.
ODAC May 3, Subgroup Analyses in Clinical Trials Stephen L George, PhD Department of Biostatistics and Bioinformatics Duke University Medical Center.
Estimation and Reporting of Heterogeneity of Treatment Effects in Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare.
Writing a Research Protocol Michael Aronica MD Program Director Internal Medicine-Pediatrics.
Concept of Measurement
Statistics By Z S Chaudry. Why do I need to know about statistics ? Tested in AKT To understand Journal articles and research papers.
EVIDENCE BASED MEDICINE
Making all research results publically available: the cry of systematic reviewers.
Are the results valid? Was the validity of the included studies appraised?
STrengthening the Reporting of OBservational Studies in Epidemiology
Funded through the ESRC’s Researcher Development Initiative
EBD for Dental Staff Seminar 2: Core Critical Appraisal Dominic Hurst evidenced.qm.
Lecture 16 (Oct 28, 2004)1 Lecture 16: Introduction to the randomized trial Introduction to intervention studies The research question: Efficacy vs effectiveness.
CAMARADES: Bringing evidence to translational medicine Evidence based translational medicine Experimental Studies Systematic review and meta-analysis how.
Study design P.Olliaro Nov04. Study designs: observational vs. experimental studies What happened?  Case-control study What’s happening?  Cross-sectional.
Evidence Based Medicine Meta-analysis and systematic reviews Ross Lawrenson.
Introduction to Systematic Reviews Afshin Ostovar Bushehr University of Medical Sciences Bushehr, /9/20151.
Appraising Randomized Clinical Trials and Systematic Reviews October 12, 2012 Mary H. Palmer, PhD, RN, C, FAAN, AGSF University of North Carolina at Chapel.
Landmark Trials: Recommendations for Interpretation and Presentation Julianna Burzynski, PharmD, BCOP, BCPS Heme/Onc Clinical Pharmacy Specialist 11/29/07.
PH 401: Meta-analysis Eunice Pyon, PharmD (718) , HS 506.
EBM Conference (Day 2). Funding Bias “He who pays, Calls the Tune” Some Facts (& Myths) Is industry research more likely to be published No Is industry.
META-ANALYSIS, RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS © LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON.
Objectives  Identify the key elements of a good randomised controlled study  To clarify the process of meta analysis and developing a systematic review.
1 Study Design Issues and Considerations in HUS Trials Yan Wang, Ph.D. Statistical Reviewer Division of Biometrics IV OB/OTS/CDER/FDA April 12, 2007.
CAT 5: How to Read an Article about a Systematic Review Maribeth Chitkara, MD Rachel Boykan, MD.
Sifting through the evidence Sarah Fradsham. Types of Evidence Primary Literature Observational studies Case Report Case Series Case Control Study Cohort.
EBM --- Journal Reading Presenter :呂宥達 Date : 2005/10/27.
Compliance Original Study Design Randomised Surgical care Medical care.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 18 Systematic Review and Meta-Analysis.
Systematic reviews and meta-analyses: when and how to do them Andrew Smith Royal Lancaster Infirmary 18 May 2015.
Introduction to Critical Appraisal January 9, 2006 Carin Gouws.
CONSORT 2010 Balakrishnan S, Pondicherry Institute of Medical Sciences.
NURS 306, Nursing Research Lisa Broughton, MSN, RN, CCRN RESEARCH STATISTICS.
Benefits and Pitfalls of Systematic Reviews and Meta-Analyses
Reporting quality in preclinical studies Emily S Sena, PhD Centre for Clinical Brain Sciences, University of
Supplementary Table 1. PRISMA checklist
Heterogeneity and sources of bias
Gerald Dyer, Jr., MPH October 20, 2016
Presentation transcript:

Summarising the evidence from animal models of neurological disease: Publication bias, poor internal validity, and (perhaps) some efficacy Malcolm Macleod University of Edinburgh

Lots of drugs seem to work in animal models of stroke Some of these drugs have been around for ages, and are probably safe Rather than spend billions developing a novel drug, why not test something we already know works in animals? Where we started … IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

What do we already know works in animals? Lots of individual promising studies Little in the way of research synthesis So … Review the field systematically Pick a target and synthesise the evidence IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

interventions in experimental stroke IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

interventions in experimental stroke Tested in focal ischaemia IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

interventions in experimental stroke Effective in focal ischaemia IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

interventions in experimental stroke Tested in clinical trial IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

interventions in experimental stroke Effective in clinical trial IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Outline Review the internal validity of animal studies Review the external validity of animal studies Explore the potential uses of this approach in mechanisms research IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Methodological approach Written review protocol – hypothesis, inclusion and exclusion criteria, analyses prespecified Standardised search strategy – 3 online databases, conference abstracts, dual screening Data extraction to bespoke database/analysis tool Random effects weighted mean difference stratified meta-analysis or meta-regression Publication bias by Funnel plot, Egger regression, Trim and Fill IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary There is nothing new under the sun … …you will meet with several observations and experiments which, though communicated for true by candid authors or undistrusted eye-witnesses, or perhaps recommended by your own experience, may, upon further trial, disappoint your expectation, either not at all succeeding, or at least varying much from what you expected Robert Boyle (1693) Concerning the Unsuccessfulness of Experiments

Internal validity … the extent to which an experiment accurately describes what happened in that model system may be confounded by Selection Bias(Randomisation) Performance Bias(Allocation Concealment) Detection Bias(Blinded outcome assessment) Attrition bias(Reporting drop-outs/ ITT analysis) False positive report bias(Adequate sample sizes) IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

NXY-059 in animal stroke models 9 publications, 29 experiments, 408 animals 44% (35-53%) improvement in outcome IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

A widespread problem … Dopamine Agonists in 6-OH model of Parkinsons Disease IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Internal validity and Journal Impact Factors Regression model offered: –Effect size –Variance –Randomisation –Allocation concealment –Blinded assessment of outcome –Sample size calculation –Statement of possible conflict of interest Univariate: –Allocation concealment –Blinded assessment of outcome –Statement of possible conflict of interest Multivariate: IF = (Conflict of Interest Statement) + 1.2(Allocation Concealment): adjusted r 2 = 0.06 IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

External validity Has the intervention been tested under a range of circumstances similar to those which might be encountered in clinical practice? Are the data you know about representative of all data? (what is the likelihood of publication bias) IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Hypertension in studies of NXY-059 in experimental stroke Hypertension: – 7% of animal studies – 77% of patients in the (neutral) SAINT II study IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Hypertension in studies of tPA in experimental stroke Comorbidity Normal BP Efficacy -2% 25% Infarct Volume: –113 publications –212 experiments –3301 animals –Improved outcome by 24% (20-28) Hypertension: –9% of animal studies –Specifically exclusion criterion in (positive) NINDS study IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

The importance of Time to Treatment Both tPA and tirilazad appear to work in animals tPA works in humans but tirilazad doesnt Time to treatment: tPA: –Animals– median 90 minutes –Clinical trial– median 90 minutes Time to treatment: tirilazad –Animals– median 10 minutes –Clinical trial- >3 hrs for >75% of patients IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Publication bias in experimental stroke Only 11/525 publications (2.2%) reported no significant treatment effects Trim and Fill suggested ~16% (214/1573) of experiments remain unpublished Best estimate of magnitude of problem –Observed efficacy31.3% ( ) –Adjusted efficacy 23.8% ( ) IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Publication bias RandomisationCo-morbidity bias Reported efficacy 24% 32% 18% 4% IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

There are multiple drivers of bias IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Summarising data from more fundamental research The interval validity of findings –Are they the product of bias? The validity of research summaries –Have the authors of pivotal reviews considered all relevant data? The external validity of findings –Do these findings hold only in limited highly controlled situations, or only one species, or are they generally applicable across biology? IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Some examples from EAE … External validity Do different endpoints measure different things?

Publications identified in review

Outcomes reported

Efficacy – for what its worth

Internal validity

External validity

Demyelination Axon loss Neurobehaviour 31% 32% 14% Relationship between endpoints

IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

The added value of a systematic approach … Identifying, and explaining, heterogeneity Developing a systematic evidence base for understanding biological pathways Providing the components for mathematical models predicting the response to interventions IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

The role of Th17 cells in the development of EAE IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Immunisation EAE Th17 Th1 CD4+ -=+ -6 =3 +61 EAE Th1 -= =1 +22 EAE Th17 IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

The internal and external validity of animal experiments is limited Conclusions from such experiments are confounded by many biases There is no reason to believe that more pathophysiological experiments are any better IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary

Future directions Qualitative systematic research Quantitative systematic research –Study quality issues –Evidence based pathways –Strategic research targeting IntroductionTranslation – Internal Validity Translation – External Validity Mechanisms Research Summary