VALIDATION AND UPDATING OF MODELS WITH BIOMARKERS Ewout W. Steyerberg, PhD Center for Medical Decision Making Dept of Public Health Erasmus MC, Rotterdam, the Netherlands E.Steyerberg@ErasmusMC.nl Utrecht, March 25, 2009

Erasmus MC – University Medical Center Rotterdam

Research

Contents Praise for Prof. Paul Ridker Prediction models for better decision making Validation and updating Extension with biomarker(s)

Search Google Scholar …

Prediction models for better decision making Identify low and high patients of cardiovascular disease better targeting of preventive interventions Predictions are probabilities No certainty Validation commonly includes calibration and discrimination Systematically too high / too low predictions Poorer performance than hoped for Better predictions with stronger predictors Much interest in biomarkers

Case study 1: validity of Framingham risk models Updating of regression coefficients: refitting Updating to average outcome: re-calibration

Validity of Framingham predictions (JAMA 2001)

Tab 1

Tab 3: Refit

Tab 5: Performance Improvement in c e.g. 0.67  0.70; native Americans even larger gains  ‘substantial improvement by using locally updated coefficients’ Recalibration important for better calibration

Case study 2: updating with a biomarker (Circulation 2008) Refitting of Framingham model Extension with CRP Many statistics to quantify improvement

Case study 3: updating with a set of biomarkers Refitting of Framingham model Extension with 4 biomarkers

Another example: NEJM 2008

Results: focus on c statistic

Elderly: mean 71 yrs

All markers add significantly, but NT-Pro-BNP is the winner

Substantial improvement in c statistics

.. and substantial NRI

Conclusions Model validation is followed by model updating and model extension Recalibration is a minimum Often new coefficients required Biomarkers need to be strong and beyond discovery phase Incremental contribution to traditional risk factors Be sceptical about new genetic signatures Quantification of model improvement challenging Patterns in C stat and net reclassification index (NRI) coincide No improvement in c  no improvement in NRI Improvement in decisions should be quantified by decision-analytic measures, weighing costs of wrong decisions, e.g. ‘Net Benefit’

Read more .. PubMed Paper with attendents of Kattan symposium “Accuracy of prediction models” Books

26