1. Building Risk Adjustment Models Andy Auerbach MD MPH

2. Overview Reminder of what we are talking about Building your own Model discrimination Model calibration Model validation

3. Outcomes measurement is an applied form of risk adjustment Typically explain only 20-25% of variation in health care utilization Explaining this amount of variation can be important if remaining variation is extremely random Example: supports equitable allocation of capitation payments from health plans to providers

4. Where does ‘risk adjustment’ fit in our model Donabedian A. JAMA 1988;260:1743-8 StructureProcessOutcomes Community Characteristics Delivery System Characteristics Provider Characteristics Population Characteristics Health Care Providers - Technical processes - Care processes - Interpersonal processes Public & Patients - Access - Equity - Adherence Health Status Functional Status Satisfaction Mortality Cost

5. Patient severity of illness is a kind of confounder ExposureOutcome Confounding Factor ?

6. What risk factors are…. They are…. – Factors that affect the patient’s risk for outcome independent of the treatment These factors can also be associated with: – Risks for receiving the treatment (allocation bias  propensity scores) – Modification of the effect of treatment (interaction terms)

7. Risk factors are not… While there may be some in common, risk factors for an outcome given a health condition are not the same as the risk factors for the condition. – Hyperlipidemia is a risk factor for MI but not for survival following an MI

8. Because in the end your analyses will look like this…. Measure = Predictor + confounders + error term Measure = Predictor + risk of outcome + other confounders + error term

9. Building Your Own Risk-Adjustment Model What outcomes – Must be clearly defined and frequent enough for modeling What population – Generic or disease specific What time period – Single visit/hospitalization or disease state that includes multiple observations What purpose – Implications for how good the model needs to be

10. Inclusion/Exclusion: Hospital Survival for Pneumonia Include – Primary ICD-9 code 480-487 (viral/bacterial pneumonias) – Secondary ICD-9 code 480-487 and primary of empyema (510), pleurisy (511), pneumothorax (512), lung abscess (513), or respiratory failure (518) Exclude – Age <18 years old – Admission in prior 10 days – Other diagnoses of acute trauma – HIV, cystic fibrosis, tuberculosis, post operative pneumonia

11. Episode of Care Does dataset include multiple observations (visits) over time of the same individual? – Re-hospitalizations – Hospital transfers Can dataset support linking observations (visits) over time? Inclusion and exclusion criteria should describe handling of multiple observations

12. Identifying Risk Factors for Model Previous literature Expert opinion/consensus Data dredging (retrospective)

13. Reliability of Measurement Is the ascertainment and recording of the variable standardized within and across sites? Are there audits of the data quality and attempts to correct errors?

14. Missing Data Amount Why is it missing? Biased ascertainment? Does missing indicate normal or some other value? Can missing data be minimized by inclusion/exclusion criteria? May want to impute missing values

15. Risk Factors: Which Value With Multiple Measurements? First? Worst? Last? Consider whether timing of data collection of risk factor accurately reflects relevant health state, could confound rating of quality or number of missing values May be able to improve estimate of some risk factors using multiple measures

16. Co-Morbidity or Complication Including complications in risk adjustment models gives providers credit for poor quality care True co-morbidities may be dropped from risk adjustment models out of concern that they sometimes represent complications

17. Vulnerability Should models include adjustment for rage, gender, or insurance type?

18. Caveats to risk factors: Gaming Situation in which the coding of risk factors is influenced by coder’s knowledge or assumptions regarding how the data will be used to create a performance report or to calculate payment The potential for gaming to alter the results (eg quality comparisons of providers) is related to the degree that it occurs similarly or differently across providers

19. Caveats: Co-morbidities andComplications In administrative data, preexisting and acute illnesses have been coded without differentiation (e.g. acute vs. preexisting thromboembolism). – Generally not an issue for chronic diseases – Link to earlier records (eg previous admissions) can be helpful Condition present at admission (CPAA) coding now a standard part of California hospital discharge data

20. Risk Factors: Patient Characteristics Not Process of Care Processes of care can be indicative of severity – Use of IABP However treatments also reflect practice style/quality Process measures can be explored as a possible mediators as opposed to risk factors for outcomes

21. Coronary Artery Disease: Mortality Rates by Race Elements in ModelBlack/White Risk Ratio Confidence Interval Race1.411.27-1.56 Race + clinical elements* 1.181.05-1.32 Race/clinical elements/Rx 1.080.97-1.20 Age, coronary anatomy, ejection fraction, chf, angina, AMI, mitral regurgitation, periph vasc disease, coexisting illnesses: Peterson et al, NEJM, 1997

22. Building Multivariate Models Start with conceptual framework from literature and expert opinion Pre-specify statistical significance for retaining variables

23. Building models: More in depth coursework Model selection, checking, and bootstrapping Epi 208 (mostly), and some in 209. Selection and evaluation of prediction models and Multiple imputation in Biostat 210. Regression Methods in Biostatistics (Vittinghoff, et. Al.) has lots of information on model selection and evaluation.

24. Empirical Testing of Risk Factors Univariate analyses to perform range checks, eliminate invalid values and low frequency factors Bivariate analyses to identify insignificant or counterintuitive factors Test variables for linear, exponential, u-shaped, or threshold effects

25. Building Multivariate Models Stepwise addition (or subtraction) monitoring for: – 20% or more change in predictor parameter estimate – Statistical significance of individual predictors Test for connections between risk and outcome/predictors – Add interactions between predictor and risk factors (or between risk factors) – Stratified analyses

26. CAGB Registry in NY State: Significant Risk Factors for Hospital Mortality for Coronary Artery Bypass Graft Surgery 1989-1992

27. Significant Risk Factors for Hospital Mortality for Coronary Artery Bypass Graft Surgery in New York State, 1989-1992

28. Risk Factors in Large Data Sets: Can you have too much power? Large datasets prone to finding statistical significance May want to consider whether statistical significance is clinically significant – May also want to select risk factors based on a clinically relevant prevalence… Conversely, consider forcing in clinically important predictors even if not statistically significant

29. Counterintuitive findings in risk adjustment Outcomes of MI treatment – Hypertension is protective - decreased risk of mortality – Perhaps a surrogate for patients on beta blockers If don’t believe hypertension truly protective then best to drop from model

30. Smaller Models are Preferred Rule of thumb: 10-30 observations per covariate not generally an issue in large datasets Smaller models are more comprehensible Less risk of “overfitting” the data

31. Evaluating Model’s Predictive Power Linear regression (continuous outcomes) Logistic regression (dichotomous outcomes)

32. Evaluating Linear Regression Models R 2 is percentage of variation in outcomes explained by the model - best for continuous dependent variables – Length of stay – Health care costs Ranges from 0-100% Generally more is better

33. More to Modeling than Numbers R 2 biased upward by more predictors Approach to categorizing outliers can affect R 2 as predicting less skewed data gives higher R 2 Model subject to random tendencies of particular dataset

34. Evaluating Logistic Models Discrimination - accuracy of predicting outcomes among all individuals depending on their characteristics Calibration - how well prediction works across the range of risk

35. Discrimination C index - compares all random pairs of individuals in each outcome group (alive vs dead) to see if risk adjustment model predicts a higher likelihood of death for those who died (concordant) Ranges from 0-1 based on proportion of concordant pairs and half of ties

36. Adequacy of Risk Adjustment Models C index of 0.5 no better than random,1.0 indicates perfect prediction Typical risk adjustment models 0.7-0.8 – 0.5 SDs better than chance results in c statistic =0.64 – 1.0 SDs better than chance resutls in c statistic = 0.76 – 1.5 SDs better than chance results in c statistic =0.86 – 2.0 SDs better tha chance results in c statistic =0.92

37. Best Model Doesn’t Always Have Biggest C statistic Adding health conditions that result from complications will raise c statistic of model but not make the model better for predicting quality.

38. Spurious Assessment of Model Performance Missing values can lead to some patients being dropped from models Be certain when comparing models that the same group of patients is being used for all models otherwise comparisons may reflect more than model performance

39. Calibration - Hosmer-Lemeshow Size of C index does not indicate how well model performs across range of risk Stratify individuals into groups (e.g. 10 groups) of equal size according to predicted likelihood of adverse outcome (eg death) Compare actual vs expected outcomes for each stratum Want a non significant p value for each stratum and across strata (Hosmer-Lemeshow statistic)

40. Stratifying by Risk Hosmer Lemeshow provides a summary statistic of how well model is calibrated Also useful to look at how well model performs at extremes (high risk and low risk)

41. Hosmer-Lemeshow For k strata the chi squared has k-2 degrees of freedom Can obtain false negative (non significant p value) by having too few cases in a stratum

42. Goodness-of-fit tests for AMI mortality models

43. Individual’s CABG Mortality Risk 65 y.o obese non white woman with diabetes and serum creatinine of 1 mg/dl presents with an urgent need for CABG surgery. What is her risk of death?

44. Calculating Expected Outcomes Solve the multivariate model incorporating an individual’s specific characteristics For continuous outcomes the predicted values are the expected values For dichotomous outcomes the sum of the derived predictor variables produces a “logit” which can be algebraically converted to a probability (e nat log odds /1 + e nat log odds )

46. Individual’s Predicted CABG Mortality Risk 65 y.o obese non white woman with diabetes presents with an urgent need for CABG surgery. What is her risk of death? Log odds = -9.74 +65(0.06) + 1(.37)+1(.16)+1(.42)+1(.26)+1(1.15) +1(.09) = 3.39 Probability of death = e lnodds /1+e lnodds 0.034/1.034=3.3%

47. Observed CABG Mortality Risk Actual outcome of whether individual lived or died Observed rate for a group is number of deaths per the number of people in that group

48. Actual and Expected CABG Surgery Mortality Rates by Patient Severity of Illness in New York Chi squared p=.16

49. Validating Model Eyeball test – Face validity/Content validity – Does empirically derived model correspond to a pre- determined conceptual model? If not is that because of highly correlated predictors? A dataset limitation? A modeling error? Internal validation in split sample Test in a different data set

50. Internal Validation Take advantage of the large size of administrative datasets Establish development and validation data sets - Randomly split samples -Samples from different time periods/areas -Determine stability of model’s predicting power Re-estimate model using all available data

51. Overfitting Data: Overspecified Model Model performs much better in fitted data set than validation data set May be due to – Infrequent predictors – Unreliable predictors – Including variables that do not meet pre-specified statistical significance

52. Model Performance for Risk Adjustment (R 2 ) 5 level risk adjustment variable 5 level plus plus 10 other predictors 5 level plus 40-65 other predictors Fitting0.1340.2600.293 Validating0.1330.2250.195

53. Validating Model in Other Datasets: Predicting Mortality following CABG STSNYVADukeMN C statistic.759.768.722.789.752 Jones et al, JACC, 1996

54. Recalibrating Risk Adjustment Models Necessary when observed outcome rate different than expected derived from a different population This could reflect quality of care or differences in coding practices Assumption is that relative weights of predictors to one another is correct Recalibration is an adjustment to all predictor coefficients to force average expected outcome rate to equal observed outcome rate

55. Recalibrating Risk Adjustment Models New York AMI mortality rate is 15% California AMI mortality rate is 13% Is care or coding different? If want to use New York derived risk adjustment model to predict expected deaths in California need to adjust predictors (eg multiply by 13/15)

56. Summary: Risk Adjustment Using Secondary Data Requires large datasets Risk factors are patient characteristics that predict outcomes, not process of care and not complications Multivariate model building should be guided by literature/expert opinion The smallest model that performs well is generally best Next time we will evaluate model performance

57. Summary Summary statistics provide a means for evaluating the predictive power of multivariate models Care should be taken to look beyond summary statistics to ensure that the model is not overspecified and that it conforms to a conceptual model Models should be validated with internal and ideally external data Next time we will review how risk adjustment models should deal with hierarchical data structures