1. VALIDATION AND UPDATING OF MODELS WITH BIOMARKERS
Ewout W. Steyerberg, PhD
Center for Medical Decision Making Dept of Public Health Erasmus MC, Rotterdam, the Netherlands
Utrecht, March 25, 2009

2. Erasmus MC – University Medical Center Rotterdam

3. Research

4. Contents Praise for Prof. Paul Ridker
Prediction models for better decision making
Validation and updating
Extension with biomarker(s)

5. Search Google Scholar …

6. Prediction models for better decision making
Identify low and high patients of cardiovascular disease
better targeting of preventive interventions
Predictions are probabilities
No certainty
Validation commonly includes calibration and discrimination
Systematically too high / too low predictions
Poorer performance than hoped for
Better predictions with stronger predictors
Much interest in biomarkers

7. Case study 1: validity of Framingham risk models
Updating of regression coefficients: refitting
Updating to average outcome: re-calibration

8. Validity of Framingham predictions (JAMA 2001)

9. Tab 1

10. Tab 3: Refit

11. Tab 5: Performance
Improvement in c e.g  0.70; native Americans even larger gains
 ‘substantial improvement by using locally updated coefficients’
Recalibration important for better calibration

12. Case study 2: updating with a biomarker (Circulation 2008)
Refitting of Framingham model
Extension with CRP
Many statistics to quantify improvement

17. Case study 3: updating with a set of biomarkers
Refitting of Framingham model
Extension with 4 biomarkers

18. Another example: NEJM 2008

19. Results: focus on c statistic

20. Elderly: mean 71 yrs

21. All markers add significantly, but NT-Pro-BNP is the winner

22. Substantial improvement in c statistics

23. .. and substantial NRI

24. Conclusions
Model validation is followed by model updating and model extension
Recalibration is a minimum
Often new coefficients required
Biomarkers need to be strong and beyond discovery phase
Incremental contribution to traditional risk factors
Be sceptical about new genetic signatures
Quantification of model improvement challenging
Patterns in C stat and net reclassification index (NRI) coincide
No improvement in c  no improvement in NRI
Improvement in decisions should be quantified by decision-analytic measures, weighing costs of wrong decisions, e.g. ‘Net Benefit’

25. Read more ..
PubMed
Paper with attendents of Kattan symposium “Accuracy of prediction models”
Books

26. 26