1. Development and Evaluation of CMS-HCC Concurrent Risk Adjustment Models Presented by Eric Olmsted, Ph.D. Gregory Pope, M.S. John Kautter, Ph.D. RTI International Presented at Academy Health June 26, 2005
411 Waverley Oaks Road ■ Suite ■ Waltham, MA

2. Concurrent Risk Adjustment Introduction
Overview
Risk Adjustment/HCC Model
Concurrent v. Prospective
Project Goals and Challenges
Model Development
Model Evaluation
Summary and Conclusion
Overview – what is population based risk adjustment, what are some of its potential uses?

3. Overview Risk Adjustment Introduction
Population Risk Adjustment:
The process by which the health status of a population is taken into consideration when setting capitation rates or evaluating patterns or outcomes of practice
Risk adjustment is used to create “apples to apples” comparisons
Risk adjustment removes the effect of health status differences
Reduces or eliminates the problem of selection

4. Overview Risk Adjustment Model
Model calibrated on 5% national sample of Medicare fee-for-service beneficiaries
Expenditures are regressed on HCC (& demographic) risk markers to estimate incremental impact of each diagnosis on expenditures
Annualized Expenditures = Σαi + Σβi + Єi
αi = demographic markers
βi = HCC markers
Risk markers are used to predict health expenditures.
Additive not categorical model.

5. Overview HCC Model Full model contains 184 HCCs
CMS-HCC model contains 70 HCCs
CMS-HCCs:
Cover a broad spectrum of health disorders
Have well-defined diagnostic criteria
Exclude highly discretionary diagnoses
Include conditions with significant expected health expenditures
Demographic Markers
Age, Gender, Medicaid, & Originally Disabled Status
Ensure means for demographic populations correctly estimated
Thus, the CMS-HCC model is a "selected significant diseases" model that focuses on adjusting for risk associated with selected high-cost diagnoses; it does not incorporate all diagnoses.

6. Overview Concurrent vs. Prospective
Prospective risk adjustment uses current year diagnoses to predict next year’s expenditures
Chronic conditions are more important
Concurrent risk adjustment uses current year diagnoses to predict this year’s expenditures
Acute conditions are more important
Concurrent also known as ‘retrospective’.

7. Overview Concurrent vs. Prospective
AMI:
Prospective Coefficient = $1,838
Concurrent Coefficient = $12,211
63% of HCC coefficients with >$1,000 difference
R-squared:
Concurrent
Prospective
70 Total Coefficients
23% with greater than $5,000 difference
63% with >1,000 difference

8. Project Goals Concurrent Risk Adjustment Project Goals:
Develop payment model for Pay-for-Performance demonstration
Develop model for use in profiling physicians
Make model consistent with prospective CMS-HCC model that is being used for MA payment, and its data collection requirements
Improve prediction across the spectrum of patient cost
MA = Medicare Advantage

9. Concurrent Modeling Challenges
Applied standard HCC model
Resulted in negative predictions and coefficients
Concurrent HCC coefficients fit high-cost beneficiaries
This forces age-sex coefficients down and they sometimes become negative
Age-sex coefficients reflect the average beneficiary
Negative age-sex coefficients can lead to negative predictions

10. Model Challenges Standard Regression

11. Model Challenges Split Sample Regression

12. Model Challenges Regression through the Origin

13. Model Challenges Nonlinear Regression

14. Project Goals Model Selection
Criteria for Model Selection
Avoid negative predictions, which lack face validity
Avoid negative coefficients
Maintain correct age-sex means to prevent age and sex selection by providers
Prefer simple models to complex models
Select model with good ‘performance’ among model evaluation measures
CMS and physicians evaluate models based on ease of use and interpretability.

15. Model Development Sample Statistics
1.4 million FFS Medicare beneficiaries with mean expenditures of $5,214
Beneficiaries with at least one CMS-HCC represent 61% of the population, but provide 94% of all Medicare expenditures
Benes with one subset of HCCs represent 24% of population but account for 72% of expenditures.

16. Model Development Standard Models
Full HCC Model
184 HCCs & demographics
CMS-HCC Model
70 HCCs & demographics
Interaction and Topcoding Models
Created disease and demographic interactions to tease out high-expense beneficiaries
Created topcoded models to reduce impact of outliers

17. Model Development Alternative Models
Nonlinear Models
Log model
Square root model
Split Sample Models
Designed separate models for populations with different expected expenditures
Community/Institutional
High Cost/Low Cost HCC
Catastrophic HCC
Multi-stage models including two-part and four-part logit models
Simple two-stage model with demographic multipliers
Segmentation

18. Model Evaluation Standard Model Results
Full HCC model suffers not only from 30% negative predictions, but also contains negative HCC coefficients
CMS-HCC model explains 92% of the variation that the Full HCC model explains
CMS-HCC model eliminates negative HCC coefficients
CMS-HCC model has only 10% negative predictions
Interaction and Topcoding Models
Did not sufficiently reduce negative predictions
Introduce the models.
Note: Full HCC model suffers not only from 30% negative predictions, but also contains negative HCC coefficients.
Note: CMS-HCC model explains 92% of the variation that the Full HCC model explains. HMOs are currently collecting the information for the CMS-HCC model. CMS-HCC model eliminates negative HCC coefficients.
Notice that R-Squared within .04 for all models
CPM showed similar results

19. Model Evaluation Alternative Model Results
Nonlinear Models
Log model and square root model did not produce reasonable predictions
Split Sample Models
Splitting sample by community/institutional did not eliminate negative predictions
Splitting sample by disease burden eliminated negative predictions
Log Model – Top 1% predicted expenditures = 1.6 million
Square Root Model - First three deciles predicted expenditures twice as much as actual.

20. Model Evaluation Measures of Model Performance
R2 within .04 for all models
R2 did not differentiate models
Predictive Ratio = Average of model’s predictions
Average of actual expenditures
Where each of the two averages is taken over the individuals in the subgroup
Predicted expenditure deciles
Number of HCCs for a beneficiary
To evaluate the models across the full spectrum of beneficiaries

21. Model Evaluation Predictive Ratios by Expenditure Percentile
Closer to 1.00 equals a better model fit.
High-low cost model predicts well.
As does four part model.

22. Model Evaluation Predictive Ratios by Expenditure Percentile
1.0 is ideal.
Model 8 tracks well, followed by Model 12.
CMS-HCC model performs poorly for lower deciles.

23. Model Evaluation Predictive Ratios by # of HCCs
Again, closer to 1.00 equals a better model fit.
High-low cost model predicts well.
As does four part model.

24. Model Evaluation Predictive Ratios by # of HCCs
Again, closer to 1.00 equals a better model fit.

25. Concurrent Model Evaluation Model Summary
High Cost & Catastrophic Models performs well
Some face validity problems with splitting HCCs into “high-cost” and “low-cost”
Still has negative predictions
Four Part Model also performs well
Computationally advanced and hard to interpret intuitively
No negative predictions
Sample Segmentation Model performs very well
Also computationally advanced
Two-Stage Multiplier Model performs adequately
No face validity problems

26. Concurrent Model Evaluation Conclusion
Nonlinearities cause difficulties in concurrent risk adjustment model calibration
Negative coefficients and predictions
These difficulties can be addressed with:
Nonlinear models
Split sample models
But nonlinear/split sample models add complexity
Difficult to estimate
Difficult to interpret
Adds instability
Two-Stage Multiplier Model
Good face validity, avoids negative coefficients and predictions
Simpler to estimate and interpret