1. Discussion/Presentation of Park and Basu: “Alternative Evaluation Metrics for Risk Adjustment Models”
Stephen P. Ryan, Olin

2. The Really Big Picture
Insurance: third-party can improve utility of risk-averse agents by equating marginal utilities across probabilistic states of the world
Not actually what health insurance looks like in the US
Adverse selection: agents know health type better than insurer
Screening: set up menu of options to induce agents to reveal type
Competition: more firms -> lower prices, but less pooling
Risk adjustment: intervention to make everyone equally profitable
Requirement: gov’t / firm must be able to compute E[Y|X], conditional distribution of expenditures given menu / observables

3. This Paper
Using MarketScan database, predict risks using several techniques:
Nine parametric regressions
Seven machine learning algorithms
Three distributional estimators
Consider several metrics:
Group-level
Individual-level
Tail distributions (these are where your expenses come from)

4. Findings No one method dominates
Parametric methods better tail- and individual-level prediction
Machine learning, distributional methods better at group-level prediction
Assertion: tradeoff between modeling individual risks and group level
Assertion: optimal method must account for insurer behavior

5. Metrics Fraction of population outside l percentage of predicted:
𝑌 𝑖 is forecast (think of draw from distribution)
This is a bit of a strange object, as it doesn’t try to match the distribution of 𝑦 𝑖
Related object on tails:

6. Data Truven MarketScan Commercial Claims and Encounter database
Validates the risk adjustment model used in ACA
Look at working-age adults continuously enrolled in medium and large employer health plans
Outcome: total expenditures, including patient payments
Goal: using data on prior year expenditures, plus 18 age-sex categories, 114 HCCs, 16 interactions with disease groups, and plan-type and state FE -> predict second year expenditures
Authors split data into estimation / prediction

7. Summary Statistics

8. Predictive Models Linear regression Generalized beta of second kind
Five parameter distribution to match distribution of expenditures
Account for zero expenditures using a second probability function (logit)
Fixed mixture model
Machine learning methods
Regularized linear regression: LASSO, ridge regression, elastic net, LARS
Artificial neural network
Decision tree
Super learner
Distributional methods
Ordered logit
Logit
???

9. Results

10. Tail Performance

11. Individual-level Predictions

12. Conclusions No one method dominates across several different metrics
But:

13. Comments The metrics reported are non-intuitive to me
We care about matching the distribution of expenditures
One should use the integrated mean square error across distribution
Bigger issue: Distributions filtered through plans
Brings up question of what exactly we are trying to predict
Performance between aggregate and individual reflects this issue
No post-selection estimation -> Chernozhukov and co-authors have explored this issue extensively
The issue of errors in some models not treated uniformly (e.g. normality in ML)

14. Comment: Mixture Models
In Fox, Kim, Ryan, and Bajari (2011), we show how to estimate mixture models using linear regression
Instead of fixing number (and position) of types and estimating weights, you can recover both using linear regression
Could expand the range of your FMM to nonparametric joint estimation of types

15. Thank you! This is an important topic and I enjoyed reading the paper
Also, learned about the SUPER method which we inadvertently repeated in 2015 (oops!)