1. A Risk Prediction Model for Recurrent Events in Chronic Coronary Heart Disease: The Heart and Soul Study Ivy Ku, Eric Vittinghoff, Kirsten Bibbins-Domingo, Michael Shlipak, Mary Whooley January 14, 2011

2. Background and Significance 1 in 3 Americans live with cardiovascular disease With advances in therapies, patients live longer with CHD Prognosis varies widely Risk stratification integral to patient management

3. Risk Prediction is Useful

4. Risk Prediction in Primary Prevention 10-year risk of incident coronary heart disease (CHD) Guides cholesterol and BP treatment in primary prevention

5. Risk Prediction in ACS

6. Predictors of worse outcomes in stable CHD Biomarkers: CRP, BNP, hs-troponin

11. Risk Prediction in Stable CHD Clinically useful, up to date, simple, integrated risk scores lacking HERS, LIPID, Framingham severe limitations Furthermore, long-term risk in CHD has not been well-characterized and quantified CHF not included in CHD risk prediction

12. Project Aims To develop a clinical prediction model and point score for 5-year risk of recurrent CV events in stable CHD To quantify and categorize the range of long-term risk in stable CHD

13. Methods The Heart and Soul Study –Cohort study of 1024 subjects with stable CHD enrolled 2000-02 –Effect of psychosocial factors on prognosis in stable CHD –Thorough phenotyping of baseline condition, biomarkers, echo, stress –Mean 6 years follow-up, > 400 CV events

14. Population SF bay area VA, UCSF, CHN clinics Inclusion: hx MI, revascularization, angiographic CAD, abnormal stress test Exclusion: MI within 6 mo, unable to walk 1 block, moving away within 3 years

15. Methods 2 Cox models –Dichotomized predictors –Continuous predictors Composite outcome: time to MI, CVA, CHF hospitalization, or CV death Use baseline survival function, relative hazards to calculate 5-year risk

16. Coding of Predictors Selected functional form of continuous predictors using AIC –categorical (quantiles, clinical cutpoints) –linear –3, 4, 5 knot restricted cubic splines Steyerberg recommends doing this a priori if possible, to avoid over-fitting Cross-validation can also be used

17. Model selection Need to maximize the signal without over-fitting Three main strategies: 1.Outcome-free data reduction: use the literature, expert opinion, practical considerations to eliminate candidate predictors without looking at the outcome 2.Parsimony: select highly significant predictors 3.Cross-validation (CV): mimics external validation

18. Our implementation Outcome free data reduction: eliminated 18 of 36 candidate predictors on the basis of expert judgment, practical considerations Parsimony: cut 4 more using backward selection Cross-validation: 10-fold CV of C-index for ~1,000 candidate models Final decision between top candidates again considered clinical convenience and face validity

19. How cross-validation works Divide sample into 5-10 subsets For each subset: –set aside, fit model to remaining subsets –calculate predictions for set-aside subset Estimate prediction error using quasi- external predictions for all observations Repeat ~20 times and average results –repetition needed to reduce noise

20. C-index A measure of model discrimination Extension of C-statistic, area under ROC curve to survival models Estimates probability that in a randomly selected pair of observations, the earlier failure has the higher predicted risk Naïve C-index is optimistic; cross- validation reduces the optimism

21. Selecting Point Score Model Cross-validation involves five steps for each candidate point score model: 1.fit model using binary predictors only 2.round coefficients to obtain point scores 3.refit model using calculated point scores as sole (continuous) predictor 4.save predictions from the refitted model 5.use predictions to calculate CV C-index

22. Shrinkage using calibration slope Cross-validation to get calibration slope: –calculate xb for omitted subsets –re-fit model using xb as the sole predictor –coefficient for xb <1.0 signals over-fitting Use slope to improve calibration –shrink coefficients by calibration slope (i.e., the coefficient for xb in the refitted model) –pulls in extreme high and low predictions –does not affect discrimination

23. Model Performance Discrimination: C-index Net reclassification improvement (NRI) –continuous vs point score models –continuous model vs Framingham Calibration: goodness-of-fit test, visual inspection, calibration slope

24. External Model Validation Cross-validation is strictly internal –reduces over-fitting –but does not protect against predictor effects that differ across populations Plan external validation in separate cohort –recommended by Altman and Royston, often demanded by reviewers

25. Results

27. Included vs Excluded Included (n=912)Excluded (n=108)P value Outcome, %27%32%0.23 Follow-up time, yrs5.8 (5.6-5.9)5.6 (5.0-6.2)0.47 Age, yrs67 ± 1168 ± 110.36 History of CHF17%22%0.27 smoker20%18%0.69 LVEF, %62 ± 1061 ± 90.60 UACR, mg/g8.7 (5.1-17.9)7.9 (2.2-11.6)0.18 BNP, pg/mL173 (74-452)222 (89-532)0.20

28. Included vs Excluded Included (n=912)Excluded (n=108)P value between HR HR (CI)P P Age1.04 (1.03- 1.06) < 0.0011.01 (0.98-1.05)0.430.10 Hx CHF2.27 (1.72-3.0)< 0.0012.3 (1.12-4.72)0.020.90 smoker1.17 (0.86- 1.59) 0.321.66 (0.75-3.67)0.210.39 LVEF2.74 (2.03- 3.71) < 0.0014.96 (2.15- 11.47) < 0.0010.18 BNP4.9 (3.81-6.31)< 0.0018.4 (3.45-20.45)< 0.0010.22

29. Functional Form Determined by AIC Age linear LVEF dichotomized at 50% UACR, BNP, BMI, CRP: 3-knot restricted cubic splines

30. Backward Selection Eliminated 4 weakest predictors (p>0.5) –HDL, LDL, hx MI, HTN Top 4 predictors were always the same by all exploratory methods –Age, EF, BNP, UACR Remaining 10 candidates –Gender, BMI, smoker, diabetes, CRP, CKD, troponin, hx CHF, med nonadherence, physical inactivity

31. Screening models using CV Base model age, LVEF, BNP, UACR Screened all 5 to 11-predictor models using 20 repetitions of 10-fold cross- validated C-index Targeting 5 to 7 predictor range, for practicality Done for both point score and continuous models

32. Top Models

33. Final Model Age, LVEF, BNP, UACR, smoker Point score –Naïve C-index 0.742 –CV C-index 0.736 Continuous model –Naïve C-index 0.768 –CV C-index 0.763

34. Final Model with Dichotomized Predictors

35. Point score Age ≥ 65 1 Smoker 1 LVEF < 50% 2 BNP > 500 3 UACR ≥ 30 3

38. Continuous Model

39. Calibration Continuous Model Pseudo-Hosmer-Lemeshow goodness-of-fit test: p = 0.94 Cross-validated calibration slope = 0.94

40. Calibration with shrinkage

41. NRI with FHS model 93 cases moved up 47 cases moved down 46 net cases 46 / 243 =18.9%, p < 0.001 329 non-cases moved down 82 non-cases moved up 247 net non-cases 247 / 661 = 37.4% p < 0.001 Net reclassification = 56.3%, p < 0.001 Cases FHSAdding HS variables 0-10%10-20%20-50%≥ 50%Total 0-10%31105 10-20%6313628 20-50%3256172161 ≥ 50%00133649 Total123288114243 Non-Cases FHSAdding HS variables 0-10%10-20%20-50%≥ 50%Total 0-10%2552032 10-20%10949255188 20-50%3316616445408 ≥ 50%04171233 Total16722420862661

42. NRI comparing point to cont. 54 cases moved up 29 cases moved down 25 net cases 25 / 246 =10.2%, p = 0.006 153 non-cases moved down 94 non-cases moved up 59 net non-cases 59 / 670 = 8.8% p = 0.002 Net reclassification = 19%, p < 0.001 Cases Point score Continuous model 0-10%10-20%20-50%≥ 50%Total 0-10%821011 10-20%123532180 20-50%08521878 ≥ 50%0096877 Total20459487246 Non-Cases Point score Continuous model 0-10%10-20%20-50%≥ 50%Total 0-10%1462410171 10-20%126157560339 20-50%31810013134 ≥ 50%0062026 Total27519916333670

43. Summary of results Our model had good discrimination (CV C-statistic 0.76), and had 56% net reclassification vs framingham secondary events model Many traditional risk factors (HTN, lipids, obesity) were not significant predictors

44. Limitations Population (VA men, CHN, urban) No external validation yet

45. Conclusion Developed a risk model with 5 predictors Can stratify 5-year recurrent CV event risk in stable CHD

46. External Validation PEACE cohort –Clinical trial of trandolapril vs placebo in low-risk stable CAD –3600 subjects with biomarkers –Patients were less sick, excluded EF<40% –1996-2000

47. References Steyerberg E. Clinical Prediction Models: A practical approach to development, validation and updating. Springer, NY 2009. Lloyd-Jones D. Cardiovascular risk prediction: Basic concepts, current status, and future directions. Circ 2010; 121: 1768-77. Morrow D. Cardiovascular risk prediction in patients with stable and unstable coronary heart disease. Circ 2010; 121: 2681-91. D’Agostino R. Primary and subsequent coronary risk appraisal: new results from the Framingham study. AHJ 2000; 139: 272-81. Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med, 2000;19:453-473.