1 Subgroup Reporting in the General Medical Literature: Do Investigators Misinterpret Their Own Findings? Erik Fernandez y Garcia, MD MPH University of.

Slides:



Advertisements
Similar presentations
Katrina Abuabara, MD, MA1 Esther E Freeman MD, PhD2;
Advertisements

What is a review? An article which looks at a question or subject and seeks to summarise and bring together evidence on a health topic.
Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ)
Statistical Analysis and Data Interpretation What is significant for the athlete, the statistician and team doctor? important Will Hopkins
U.S. Food and Drug Administration Notice: Archived Document The content in this document is provided on the FDA’s website for reference purposes only.
Copyright restrictions may apply JAMA Pediatrics Journal Club Slides: Pharmacologic Treatment of Pediatric Headaches El-Chammas K, Keyes J, Thompson N,
Critically Evaluating the Evidence: Tools for Appraisal Elizabeth A. Crabtree, MPH, PhD (c) Director of Evidence-Based Practice, Quality Management Assistant.
ODAC May 3, Subgroup Analyses in Clinical Trials Stephen L George, PhD Department of Biostatistics and Bioinformatics Duke University Medical Center.
1 Moderators of Treatment Effects in the General Medicine Literature: Looking for Improvement Nicole Bloser, MHA, MPH University of California, Davis June.
New or existing slides are easily formatted using built-in layouts that can be applied via the Home tab EMA DRAFT GUIDELINE ON SUBGROUPS DISCUSSION April.
Estimation and Reporting of Heterogeneity of Treatment Effects in Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare.
Critical Appraisal Dr Samira Alsenany Dr SA 2012 Dr Samira alsenany.
Evidenced Based Practice; Systematic Reviews; Critiquing Research
How does the process work? Submissions in 2007 (n=13,043) Perspectives.
Journal Club Alcohol, Other Drugs, and Health: Current Evidence November-December 2007.
Common Problems in Writing Statistical Plan of Clinical Trial Protocol Liying XU CCTER CUHK.
Critical Appraisal of an Article by Dr. I. Selvaraj B. SC. ,M. B. B. S
Accounting for Clinical Heterogeneity in Comparative Effectiveness Research How Can One Examine a Trial for Heterogeneity of Treatment Effect (HTE)? The.
Making all research results publically available: the cry of systematic reviewers.
Writing a Research Proposal
Are the results valid? Was the validity of the included studies appraised?
Published in Circulation 2005 Percutaneous Coronary Intervention Versus Conservative Therapy in Nonacute Coronary Artery Disease: A Meta-Analysis Demosthenes.
CRITICAL APPRAISAL OF SCIENTIFIC LITERATURE
Background to Adaptive Design Nigel Stallard Professor of Medical Statistics Director of Health Sciences Research Institute Warwick Medical School
CHP400: Community Health Program- lI Research Methodology STUDY DESIGNS Observational / Analytical Studies Case Control Studies Present: Disease Past:
Confidence Intervals Elizabeth Garrett-Mayer
Effects of Pediatric Asthma Education on Hospitalizations and Emergency Department Visits: A Meta-Analysis June 3, 2007 Janet M. Coffman, PhD, Michael.
Publication Bias in Medical Informatics evaluation research: Is it an issue or not? Mag. (FH) Christof Machan, M.Sc. Univ-Prof. Elske Ammenwerth Dr. Thomas.
1 ICEBOH Split-mouth studies and systematic reviews Ian Needleman 1 & Helen Worthington 2 1 Unit of Periodontology UCL Eastman Dental Institute International.
Systematic Reviews.
Effective Presentation of Study Results How are RCTs presented in abstracts & publications? and Some things to consider in your own presentations NCIC.
EBC course 10 April 2003 Critical Appraisal of the Clinical Literature: The Big Picture Cynthia R. Long, PhD Associate Professor Palmer Center for Chiropractic.
Literature searching & critical appraisal Chihaya Koriyama August 15, 2011 (Lecture 2)
1 Section 9-4 Two Means: Matched Pairs In this section we deal with dependent samples. In other words, there is some relationship between the two samples.
Critical Appraisal of the Scientific Literature
Landmark Trials: Recommendations for Interpretation and Presentation Julianna Burzynski, PharmD, BCOP, BCPS Heme/Onc Clinical Pharmacy Specialist 11/29/07.
Randomized Trial of Preoperative Chemoradiation Versus Surgery Alone in Patients with Locoregional Esophageal Carcinoma, Ursa et al. Statistical Methods:
Economics 173 Business Statistics Lecture 4 Fall, 2001 Professor J. Petry
Wipanee Phupakdi, MD September 15, Overview  Define EBM  Learn steps in EBM process  Identify parts of a well-built clinical question  Discuss.
Objectives  Identify the key elements of a good randomised controlled study  To clarify the process of meta analysis and developing a systematic review.
Evaluating the Medical Evidence ​ A TOOLKIT FOR THE INTERPRETING THE EFFECTIVENESS OF INTERVENTIONS Niteesh Choudhy, M.D., Ph.D.
CAT 5: How to Read an Article about a Systematic Review Maribeth Chitkara, MD Rachel Boykan, MD.
Health and Disease in Populations 2002 Sources of variation (1) Paul Burton! Jane Hutton.
Probability & Significance Everything you always wanted to know about p-values* *but were afraid to ask Evidence Based Chiropractic April 10, 2003.
Module 3 Finding the Evidence: Pre-appraised Literature.
EBM --- Journal Reading Presenter :呂宥達 Date : 2005/10/27.
1 Lecture 10: Meta-analysis of intervention studies Introduction to meta-analysis Selection of studies Abstraction of information Quality scores Methods.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 18 Systematic Review and Meta-Analysis.
Division of Oncology Drug Products 1 AREAS OF MAJOR STATISTICAL CONCERNS IN THE M01 STUDY Overall (ITT Population) Finding Liver Metastasis Subgroup Finding.
Research Design Evidence Based Medicine Concepts and Glossary.
CONSORT 2010 Balakrishnan S, Pondicherry Institute of Medical Sciences.
بسم الله الرحمن الرحیم.
1 Lecture 10: Meta-analysis of intervention studies Introduction to meta-analysis Selection of studies Abstraction of information Quality scores Methods.
EBM --- Journal Reading Presenter :林禹君 Date : 2005/10/26.
Copyright © 2012 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 27 Systematic Reviews of Research Evidence: Meta-Analysis, Metasynthesis,
Statistical Criteria for Establishing Safety and Efficacy of Allergenic Products Tammy Massie, PhD Mathematical Statistician Team Leader Bacterial, Parasitic.
Is a meta-analysis right for me? Jaime Peters June 2014.
CRITICAL APPRAISAL OF A JOURNAL
Effectiveness of yoga for hypertension: Systematic review and meta-analysis Marshall Hagins, PT, PhD1, Rebecca States,
Critically Appraising a Medical Journal Article
A Meta Analysis of the Impact of SBI on Healthcare Utilization
Supplementary Table 1. PRISMA checklist
Critical Reading of Clinical Study Results
How to publish from your MEd or PhD research
A Systematic Review and Meta-analysis of Randomized Trials of Manual Thrombectomy in ST elevation myocardial infarction Investigators: Ashraf Alazzoni,
A Meta Analysis of the Impact of SBI on Healthcare Utilization
Publication Bias in Systematic Reviews
What is a review? An article which looks at a question or subject and seeks to summarise and bring together evidence on a health topic. Ask What is a review?
Lack of Confidence Interval Reporting in Dermatology: A Call to Action
Evidence Based Diagnosis
Presentation transcript:

1 Subgroup Reporting in the General Medical Literature: Do Investigators Misinterpret Their Own Findings? Erik Fernandez y Garcia, MD MPH University of California, Davis Co-authors: Hien Nguyen, MD (UCD); Naihua Duan, PhD (Columbia University); Nicole Bloser Gabler, MHA MPH (UCD); Diana Liao MPH(UCLA); Richard L. Kravitz, MD MSPH (UCD) Supported by a grant from Pfizer.

2 RATIONALE Randomized-controlled trials (RCTs) yield an average treatment effect. In a trial, treatments may have different net benefits and harms for different patients (heterogeneity of treatment effects, or HTE). Examination of the HTE is important for optimizing treatment for individual patients. Especially critical in era of increasing population diversity and health disparities.

3 RATIONALE Subgroup analysis (SGA) in RCTs is one way of investigating HTE. The usefulness of SGA is hampered by the problems of –insufficient power (false negative) –multiple testing (false positive)

4 RATIONALE Under-use: SGA not performed in studies with sufficient power and theoretical rationale to anticipate helpful results. Over-use: SGA performed in studies which were underpowered or lacked theoretical rationale to anticipate helpful results. Misuse: SGA was performed in the appropriate setting but with inappropriate methodology and/or interpretation.

5 STUDY QUESTIONS We sought to specifically investigate the potential misuse of SGA by asking: 1)How often are HTE analyses and corresponding covariates prespecified and were the reasons (if any) primarily substantive or statistical? 2)What was the objective evidence for or against the presence of HTE? 3)How did authors interpret their own HTE-related findings, and to what extent did their interpretations match the objective evidence?

6 METHODS Design: Systematic Review Population: JAMA, BMJ, Lancet, NEJM, Annals Probability Sample: –Odd months in 1994, 1999, 2004 –Initial search: N = 4,863 articles –After additional random sampling and exclusions, N = 319 clinical trials –Final sample: 87 of 319 trials (27%) reporting test for HTE

7 Covariates Examined in HTE analyses PrespecifiedNot Prespecified All covariatesSome SubstantiveStatistical Number of Covariates with Rationale None Types of Reasons Coding of Covariates

8 Defining Clinicostatistical Divergence Clinicostatistical Divergence: Clinically meaningful and statistically significant differences between subgroup effects and average effect (coded as “none” “weak” “moderate” “strong”) –Clinical Divergence (CD): Was the ratio measure of effect in any subgroup at least 25% greater or smaller than in the sample as a whole? –Statistical Significance (SS): Was a test for interaction associated with a p value of less than or equal to 0.10?

9 Coding Clinicostatistical Divergence * Denotes absence of data

10 Coding Authors’ Interpretations Evidence for HTE sufficient to support different treatment recommendations in one or more subgroups Evidence for HTE insufficient to support different treatment recommendations but sufficient to warrant further systematic research Evidence for HTE was possibly present but insufficient to warrant further research Definite evidence against HTE No interpretation of HTE results

11 87 RCTs PrespecifiedNot Prespecified All covariatesSome SubstantiveStatistical Trials by Number of Covariates with Rationale None Types of Reasons RESULTS 53 (61%)34 (39%) 17 (32%) 12 (23%)24 (45%) 22 (76%)7 (24%)

29/87 = 33%

17/58 = 29%

25/87 = 29%

31/62 = 50% 10/62 = 16% 31/62 = 50%

Overstated = 27%Understated = 9%

18 LIMITATIONS Limited number of journals, years, trials reviewed Data potentially incomplete –HTE analyses performed but not published –HTE analyses performed and published in secondary journals

19 CONCLUSIONS Analysis and reporting of HTE incomplete –Prespecification inconsistent –Rationale incomplete –Effect measures and p-values (or CIs) incompletely reported When reported, objective evidence for clinicostatistical divergence found in approximately 1/3 of trials Authors frequently misinterpret own findings (in both directions)

20 IMPLICATIONS Researchers: –Ensure that SGA are prespecified with a priori rationales for covariate inclusion, or clearly labeled as exploratory –Include a statistical test for interaction or heterogeneity in the analyses –Report all results from SGA (including p values for HTE tests and effect measures with confidence intervals), even if not significant

21 IMPLICATIONS Journal Editors : –Ensure authors report SGA-associated data when SGA is performed –Ensure that authors’ discussion includes interpretation of SGA performed and limitations of such analyses Readers/Clinicians : –Weigh the authors’ interpretations and recommendations in light of the objective evidence presented prior to changing practice or implementing recommendations

22 Thank you