Discussion/Presentation of Park and Basu: “Alternative Evaluation Metrics for Risk Adjustment Models” Stephen P. Ryan, Olin

The Really Big Picture Insurance: third-party can improve utility of risk-averse agents by equating marginal utilities across probabilistic states of the world Not actually what health insurance looks like in the US Adverse selection: agents know health type better than insurer Screening: set up menu of options to induce agents to reveal type Competition: more firms -> lower prices, but less pooling Risk adjustment: intervention to make everyone equally profitable Requirement: gov’t / firm must be able to compute E[Y|X], conditional distribution of expenditures given menu / observables

This Paper Using 2013-4 MarketScan database, predict risks using several techniques: Nine parametric regressions Seven machine learning algorithms Three distributional estimators Consider several metrics: Group-level Individual-level Tail distributions (these are where your expenses come from)

Findings No one method dominates Parametric methods better tail- and individual-level prediction Machine learning, distributional methods better at group-level prediction Assertion: tradeoff between modeling individual risks and group level Assertion: optimal method must account for insurer behavior

Metrics Fraction of population outside l percentage of predicted: 𝑌 𝑖 is forecast (think of draw from distribution) This is a bit of a strange object, as it doesn’t try to match the distribution of 𝑦 𝑖 Related object on tails:

Data Truven MarketScan Commercial Claims and Encounter database 2013-2014 Validates the risk adjustment model used in ACA Look at working-age adults 21-64 continuously enrolled in medium and large employer health plans Outcome: total expenditures, including patient payments Goal: using data on prior year expenditures, plus 18 age-sex categories, 114 HCCs, 16 interactions with disease groups, and plan-type and state FE -> predict second year expenditures Authors split data into estimation / prediction

Summary Statistics

Predictive Models Linear regression Generalized beta of second kind Five parameter distribution to match distribution of expenditures Account for zero expenditures using a second probability function (logit) Fixed mixture model Machine learning methods Regularized linear regression: LASSO, ridge regression, elastic net, LARS Artificial neural network Decision tree Super learner Distributional methods Ordered logit Logit ???

Results

Tail Performance

Individual-level Predictions

Conclusions No one method dominates across several different metrics But:

Comments The metrics reported are non-intuitive to me We care about matching the distribution of expenditures One should use the integrated mean square error across distribution Bigger issue: Distributions filtered through plans Brings up question of what exactly we are trying to predict Performance between aggregate and individual reflects this issue No post-selection estimation -> Chernozhukov and co-authors have explored this issue extensively The issue of errors in some models not treated uniformly (e.g. normality in ML)

Comment: Mixture Models In Fox, Kim, Ryan, and Bajari (2011), we show how to estimate mixture models using linear regression Instead of fixing number (and position) of types and estimating weights, you can recover both using linear regression Could expand the range of your FMM to nonparametric joint estimation of types

Thank you! This is an important topic and I enjoyed reading the paper Also, learned about the SUPER method which we inadvertently repeated in 2015 (oops!)