A Risk Prediction Model for Recurrent Events in Chronic Coronary Heart Disease: The Heart and Soul Study Ivy Ku, Eric Vittinghoff, Kirsten Bibbins-Domingo, Michael Shlipak, Mary Whooley January 14, 2011

Background and Significance 1 in 3 Americans live with cardiovascular disease With advances in therapies, patients live longer with CHD Prognosis varies widely Risk stratification integral to patient management

Risk Prediction is Useful

Risk Prediction in Primary Prevention 10-year risk of incident coronary heart disease (CHD) Guides cholesterol and BP treatment in primary prevention

Risk Prediction in ACS

Predictors of worse outcomes in stable CHD Biomarkers: CRP, BNP, hs-troponin

Risk Prediction in Stable CHD Clinically useful, up to date, simple, integrated risk scores lacking HERS, LIPID, Framingham severe limitations Furthermore, long-term risk in CHD has not been well-characterized and quantified CHF not included in CHD risk prediction

Project Aims To develop a clinical prediction model and point score for 5-year risk of recurrent CV events in stable CHD To quantify and categorize the range of long-term risk in stable CHD

Methods The Heart and Soul Study –Cohort study of 1024 subjects with stable CHD enrolled –Effect of psychosocial factors on prognosis in stable CHD –Thorough phenotyping of baseline condition, biomarkers, echo, stress –Mean 6 years follow-up, > 400 CV events

Population SF bay area VA, UCSF, CHN clinics Inclusion: hx MI, revascularization, angiographic CAD, abnormal stress test Exclusion: MI within 6 mo, unable to walk 1 block, moving away within 3 years

Methods 2 Cox models –Dichotomized predictors –Continuous predictors Composite outcome: time to MI, CVA, CHF hospitalization, or CV death Use baseline survival function, relative hazards to calculate 5-year risk

Coding of Predictors Selected functional form of continuous predictors using AIC –categorical (quantiles, clinical cutpoints) –linear –3, 4, 5 knot restricted cubic splines Steyerberg recommends doing this a priori if possible, to avoid over-fitting Cross-validation can also be used

Model selection Need to maximize the signal without over-fitting Three main strategies: 1.Outcome-free data reduction: use the literature, expert opinion, practical considerations to eliminate candidate predictors without looking at the outcome 2.Parsimony: select highly significant predictors 3.Cross-validation (CV): mimics external validation

Our implementation Outcome free data reduction: eliminated 18 of 36 candidate predictors on the basis of expert judgment, practical considerations Parsimony: cut 4 more using backward selection Cross-validation: 10-fold CV of C-index for ~1,000 candidate models Final decision between top candidates again considered clinical convenience and face validity

How cross-validation works Divide sample into 5-10 subsets For each subset: –set aside, fit model to remaining subsets –calculate predictions for set-aside subset Estimate prediction error using quasi- external predictions for all observations Repeat ~20 times and average results –repetition needed to reduce noise

C-index A measure of model discrimination Extension of C-statistic, area under ROC curve to survival models Estimates probability that in a randomly selected pair of observations, the earlier failure has the higher predicted risk Naïve C-index is optimistic; cross- validation reduces the optimism

Selecting Point Score Model Cross-validation involves five steps for each candidate point score model: 1.fit model using binary predictors only 2.round coefficients to obtain point scores 3.refit model using calculated point scores as sole (continuous) predictor 4.save predictions from the refitted model 5.use predictions to calculate CV C-index

Shrinkage using calibration slope Cross-validation to get calibration slope: –calculate xb for omitted subsets –re-fit model using xb as the sole predictor –coefficient for xb <1.0 signals over-fitting Use slope to improve calibration –shrink coefficients by calibration slope (i.e., the coefficient for xb in the refitted model) –pulls in extreme high and low predictions –does not affect discrimination

Model Performance Discrimination: C-index Net reclassification improvement (NRI) –continuous vs point score models –continuous model vs Framingham Calibration: goodness-of-fit test, visual inspection, calibration slope

External Model Validation Cross-validation is strictly internal –reduces over-fitting –but does not protect against predictor effects that differ across populations Plan external validation in separate cohort –recommended by Altman and Royston, often demanded by reviewers

Results

Included vs Excluded Included (n=912)Excluded (n=108)P value Outcome, %27%32%0.23 Follow-up time, yrs5.8 ( )5.6 ( )0.47 Age, yrs67 ± 1168 ± History of CHF17%22%0.27 smoker20%18%0.69 LVEF, %62 ± 1061 ± UACR, mg/g8.7 ( )7.9 ( )0.18 BNP, pg/mL173 (74-452)222 (89-532)0.20

Included vs Excluded Included (n=912)Excluded (n=108)P value between HR HR (CI)P P Age1.04 ( ) < ( ) Hx CHF2.27 ( )< ( ) smoker1.17 ( ) ( ) LVEF2.74 ( ) < ( ) < BNP4.9 ( )< ( )<

Functional Form Determined by AIC Age linear LVEF dichotomized at 50% UACR, BNP, BMI, CRP: 3-knot restricted cubic splines

Backward Selection Eliminated 4 weakest predictors (p>0.5) –HDL, LDL, hx MI, HTN Top 4 predictors were always the same by all exploratory methods –Age, EF, BNP, UACR Remaining 10 candidates –Gender, BMI, smoker, diabetes, CRP, CKD, troponin, hx CHF, med nonadherence, physical inactivity

Screening models using CV Base model age, LVEF, BNP, UACR Screened all 5 to 11-predictor models using 20 repetitions of 10-fold cross- validated C-index Targeting 5 to 7 predictor range, for practicality Done for both point score and continuous models

Top Models

Final Model Age, LVEF, BNP, UACR, smoker Point score –Naïve C-index –CV C-index Continuous model –Naïve C-index –CV C-index 0.763

Final Model with Dichotomized Predictors

Point score Age ≥ 65 1 Smoker 1 LVEF < 50% 2 BNP > UACR ≥ 30 3

Continuous Model

Calibration Continuous Model Pseudo-Hosmer-Lemeshow goodness-of-fit test: p = 0.94 Cross-validated calibration slope = 0.94

Calibration with shrinkage

NRI with FHS model 93 cases moved up 47 cases moved down 46 net cases 46 / 243 =18.9%, p < non-cases moved down 82 non-cases moved up 247 net non-cases 247 / 661 = 37.4% p < Net reclassification = 56.3%, p < Cases FHSAdding HS variables 0-10%10-20%20-50%≥ 50%Total 0-10% % % ≥ 50% Total Non-Cases FHSAdding HS variables 0-10%10-20%20-50%≥ 50%Total 0-10% % % ≥ 50% Total

NRI comparing point to cont. 54 cases moved up 29 cases moved down 25 net cases 25 / 246 =10.2%, p = non-cases moved down 94 non-cases moved up 59 net non-cases 59 / 670 = 8.8% p = Net reclassification = 19%, p < Cases Point score Continuous model 0-10%10-20%20-50%≥ 50%Total 0-10% % % ≥ 50% Total Non-Cases Point score Continuous model 0-10%10-20%20-50%≥ 50%Total 0-10% % % ≥ 50% Total

Summary of results Our model had good discrimination (CV C-statistic 0.76), and had 56% net reclassification vs framingham secondary events model Many traditional risk factors (HTN, lipids, obesity) were not significant predictors

Limitations Population (VA men, CHN, urban) No external validation yet

Conclusion Developed a risk model with 5 predictors Can stratify 5-year recurrent CV event risk in stable CHD

External Validation PEACE cohort –Clinical trial of trandolapril vs placebo in low-risk stable CAD –3600 subjects with biomarkers –Patients were less sick, excluded EF<40% –

References Steyerberg E. Clinical Prediction Models: A practical approach to development, validation and updating. Springer, NY Lloyd-Jones D. Cardiovascular risk prediction: Basic concepts, current status, and future directions. Circ 2010; 121: Morrow D. Cardiovascular risk prediction in patients with stable and unstable coronary heart disease. Circ 2010; 121: D’Agostino R. Primary and subsequent coronary risk appraisal: new results from the Framingham study. AHJ 2000; 139: Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med, 2000;19: