How do we know whether a marker or model is any good? A discussion of some simple decision analytic methods Carrie Bennette (on behalf of Andrew Vickers)

Slides:



Advertisements
Similar presentations
Grading the Strength of a Body of Evidence on Diagnostic Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for.
Advertisements

Statistical Analysis and Data Interpretation What is significant for the athlete, the statistician and team doctor? important Will Hopkins
Receiver Operating Characteristic (ROC) Curves
Departments of Medicine and Biostatistics
Evaluating Diagnostic Accuracy of Prostate Cancer Using Bayesian Analysis Part of an Undergraduate Research course Chantal D. Larose.
Clinical Trial Design Considerations for Therapeutic Cancer Vaccines Richard Simon, D.Sc. Chief, Biometric Research Branch, NCI
V. Petrenkiene*, D. Petrauskas L. Kupcinskas, Lithuanian University of Health sciences Clinic of Gastroenterology Kaunas Utility of non-invasive markers.
1 Using Biostatistics to Evaluate Vaccines and Medical Tests Holly Janes Fred Hutchinson Cancer Research Center.
Darlene Goldstein 29 January 2003 Receiver Operating Characteristic Methodology.
How do we know whether a marker or model is any good? A discussion of some simple decision analytic methods Carrie Bennette on behalf of Andrew Vickers.
Statistics for Health Care
By Dr. Ahmed Mostafa Assist. Prof. of anesthesia & I.C.U. Evidence-based medicine.
EVIDENCE BASED MEDICINE
Cohort Studies Hanna E. Bloomfield, MD, MPH Professor of Medicine Associate Chief of Staff, Research Minneapolis VA Medical Center.
Inferential Statistics
Study Design / Data: Case-Control, Descriptives Basic Medical Statistics Course: Module C October 2010 Wilma Heemsbergen
BASIC STATISTICS: AN OXYMORON? (With a little EPI thrown in…) URVASHI VAID MD, MS AUG 2012.
Tests of significance & hypothesis testing Dr. Omar Al Jadaan Assistant Professor – Computer Science & Mathematics.
Assessment of Model Development Techniques and Evaluation Methods for Binary Classification in the Credit Industry DSI Conference Jennifer Lewis Priestley.
Multiple Choice Questions for discussion
Criteria for Assessment of Performance of Cancer Risk Prediction Models: Overview Ruth Pfeiffer Cancer Risk Prediction Workshop, May 21, 2004 Division.
Surrogate Endpoints and Correlative Outcomes Hem/Onc Journal Club January 9, 2009.
Evidence Evaluation & Methods Workgroup: Developing a Decision Analysis Model Lisa A. Prosser, PhD, MS September 23, 2011.
Division of Population Health Sciences Royal College of Surgeons in Ireland Coláiste Ríoga na Máinleá in Éirinn Indices of Performances of CPRs Nicola.
Statistics for Health Care Biostatistics. Phases of a Full Clinical Trial Phase I – the trial takes place after the development of a therapy and is designed.
EDRN Approaches to Biomarker Validation DMCC Statisticians Fred Hutchinson Cancer Research Center Margaret Pepe Ziding Feng, Mark Thornquist, Yingye Zheng,
The PCA3 Assay improves the prediction of initial biopsy outcome and may be indicative of prostate cancer aggressiveness de la Taille A, Irani J, Graefen.
Biostatistics Case Studies Peter D. Christenson Biostatistician Session 2: Diagnostic Classification.
Evidence-Based Public Health Nancy Allee, MLS, MPH University of Michigan November 6, 2004.
D:/rg/folien/ms/ms-USA ppt F 1 Assessment of prediction error of risk prediction models Thomas Gerds and Martin Schumacher Institute of Medical.
EBCP. Random vs Systemic error Random error: errors in measurement that lead to measured values being inconsistent when repeated measures are taken. Ie:
This material was developed by Oregon Health & Science University, funded by the Department of Health and Human Services, Office of the National Coordinator.
1 SCREENING. 2 Why screen? Who wants to screen? n Doctors n Labs n Hospitals n Drug companies n Public n Who doesn’t ?
HW215: Models of Health & Wellness Unit 7: Health and Wellness Models Geo-political Influences.
Jennifer Lewis Priestley Presentation of “Assessment of Evaluation Methods for Prediction and Classification of Consumer Risk in the Credit Industry” co-authored.
Chris and Abbie’s Vital Statistics. It’s actually quite dull!!! But......It is quite easy if you think it through. Arghhh Statistics...
Introduction to Healthcare and Public Health in the US The Evolution and Reform of Healthcare in the US Lecture b This material (Comp1_Unit9b) was developed.
Evaluating Risk Adjustment Models Andy Bindman MD Department of Medicine, Epidemiology and Biostatistics.
Unit 15: Screening. Unit 15 Learning Objectives: 1.Understand the role of screening in the secondary prevention of disease. 2.Recognize the characteristics.
Organization of statistical research. The role of Biostatisticians Biostatisticians play essential roles in designing studies, analyzing data and.
EBM --- Journal Reading Presenter :呂宥達 Date : 2005/10/27.
EVALUATING u After retrieving the literature, you have to evaluate or critically appraise the evidence for its validity and applicability to your patient.
BIOSTATISTICS Lecture 2. The role of Biostatisticians Biostatisticians play essential roles in designing studies, analyzing data and creating methods.
Laboratory Medicine: Basic QC Concepts M. Desmond Burke, MD.
Research Design Evidence Based Medicine Concepts and Glossary.
Relationship between performance measures: From statistical evaluations to decision-analysis Ewout Steyerberg Dept of Public Health, Erasmus MC, Rotterdam,
Diagnostic Likelihood Ratio Presented by Juan Wang.
Net Reclassification Risk: a graph to clarify the potential prognostic utility of new markers Ewout Steyerberg Professor of Medical Decision Making Dept.
Biostatistics Board Review Parul Chaudhri, DO Family Medicine Faculty Development Fellow, UPMC St Margaret March 5, 2016.
Prostatectomy operations in England South West Public Health Observatory Trends in the use of radical prostatectomy in England Sean McPhail.
© 2010 Jones and Bartlett Publishers, LLC. Chapter 12 Clinical Epidemiology.
What are the Chances Dr? Nick Pendleton. Can I have a Prostate Check? ?
Assessing the additional value of diagnostic markers: a comparison of traditional and novel measures Ewout W. Steyerberg Professor of Medical Decision.
Screening Tests: A Review. Learning Objectives: 1.Understand the role of screening in the secondary prevention of disease. 2.Recognize the characteristics.
Kelci J. Miclaus, PhD Advanced Analytics R&D Manager JMP Life Sciences
Chapter 10 Hypothesis Testing 1.
New Study Offers Support for Prostate Testing
The Role of Statistics in Clinical Trials
Strategies to incorporate pharmacoeconomics into pharmacotherapy
FAMILY MEDICINE AND LABORATORY TESTS Elham
Chapter 7 The Hierarchy of Evidence
Null Hypothesis Testing
NAPLEX preparation: Biostatistics
Diagnosis II Dr. Brent E. Faught, Ph.D. Assistant Professor
Multivariable Logistic Regression Split Cohort into Development &
EBM Dr Adrian Burger 20 March 2007.
Evidence-Based Public Health
Professor of Clinical Biostatistics and Medical Decision Making Nov-19 Why Most Statistical Predictions Cannot Reliably Support Decision-Making:
Logical Inference on Treatment Efficacy When Subgroups Exist
Presentation transcript:

How do we know whether a marker or model is any good? A discussion of some simple decision analytic methods Carrie Bennette (on behalf of Andrew Vickers) University of Washington

Overview of talk Marker research in cancer: state of the science Traditional statistical methods for evaluating predictions Decision analytic approaches

Overview of talk Marker research in cancer: state of the science Traditional statistical methods for evaluating predictions Decision analytic approaches

A combination of common and minor variations in five regions of DNA can help predict a man’s risk of getting prostate cancer, researchers reported Wednesday. A company formed by researchers at Wake Forest University School of Medicine is expected to make the test available in a few months …. It should cost less than $300. This is, some medical experts say, a first taste of what is expected to be a revolution in medical prognostication

SNP panel Predictive accuracy of SNP panel: 0.57 Predictive accuracy of single PSA in middle age: 0.75 Doesn’t add to standard predictors (Nam et al.)

Systematic review of molecular markers in cancer 129 papers published in 2005 and 2006 eligible for analysis More markers than papers 97% included inference statistics 36% included marker in a multivariable model 11% measured predictive accuracy 0 used decision analytic techniques

Overview of talk Marker research in cancer: state of the science Traditional statistical methods for evaluating predictions Decision analytic approaches

Example: Binary test for cancer on biopsy Patients with high PSA are referred to biopsy But most patients with high PSA don’t have prostate cancer Could a second marker help? Study of biopsy cohort: 26% had cancer –Assess presence of two markers

Traditional biostatistical metrics SensitivitySpecificityPPVNPVLR+LR- AUC (Youden) Brier (mean squared error) Test A91%40%35%92% Test B51%78%45%82%

Which test is best? Sensitivity / specificity insufficient to determine which test should be used: –“Depends on whether sensitivity or specificity is more important”

Conclusion about traditional metrics Traditional biostatistical techniques for evaluating models, markers and tests do not incorporate clinical consequences Accordingly, they cannot inform clinical practice

Overview of talk Marker research in cancer: state of the science Traditional statistical methods for evaluating predictions Decision analytic approaches

A hierarchy of evidence Inference statistics –Marker “not unassociated with outcome” Predictive accuracy –How much information does the marker give you? Decision analytic techniques –Do you make better decisions on the basis of the marker?

Threshold probability Predicted probability of disease is p ̂ Define a threshold probability of disease as p t Patient accepts treatment if p ̂ ≥ p t p t describes how patients values relative harm of false positive and false negative

Decision theory “I would biopsy a man if his risk of prostate cancer was 20% or more, that is, I would conduct no more than 5 biopsies to find one cancer. I consider the harms associated with delaying the diagnosis of prostate cancer to be four times worse than the harms, risks and inconvenience of biopsy.”

Treat:Sens.Spec.Prev.Net benefit Test A91%40%26% 91% × 26% - (1 – 40%) × (1 – 26%) × (0.2 ÷ 0.8) = Test B51%78%26% 51% × 26% - (1 – 78%) × (1 – 26%) × (0.2 ÷ 0.8) = Everyone100%0%26% 100% × 26% - (1 – 0%) × (1 – 26%) × (0.2 ÷ 0.8) = No-one0%100%26% 0% × 26% - (1 – 100%) × (1 – 26%) × (0.2 ÷ 0.8) = 0 Worked example at p t of 20%

Net benefit has simple clinical interpretation Net benefit of at p t of 20% Using the model is the equivalent of a strategy that led to 126 patients per 1000 with cancer being biopsied with no unnecessary biopsies

Net benefit has simple clinical interpretation Difference between model and treat all at p t of 20%. – Divide by weighting 0.051/ 0.25 = –204 fewer false positives per 1000 patients for equal number of true positives –E.g. 204 fewer patients undergoing biopsy without missing any cancers

Decision curve analysis 4. Vary p t over an appropriate range Vickers & Elkin Med Decis Making 2006;26:565– Select a p t 2. Positive test defined as 3. Calculate “Clinical Net Benefit” as:

Decision analysis All markers PSA Free, Total PSA Biopsy all Biopsy none Vickers JCO 2009

Parry-Jones A R et al. Stroke. 2013;44:

Conclusion Huge number of markers proposed Evidence base is very weak for most Traditional biostatistical methods do not assess clinical value of a marker Simple decision analytic methods can distinguish potentially useful from useless models and markers