What, Why and How: Modeling to Address Health Policy Questions Deborah Chollet Senior Fellow, Mathematica Policy Research The Robert Wood Johnson Foundation’s State Coverage Initiatives Program Washington, DC November 10, 2004

What is a Model? A structured way to think about a problem A structured way to think about a problem A way to measure responses and outcomes A way to measure responses and outcomes A way to compare alternative options A way to compare alternative options

Why Model Policy? Understand the impacts of policy change and sensitivity to program options Understand the impacts of policy change and sensitivity to program options –Program enrollment and cost –Uninsured population Understand the sensitivity of estimates to environmental factors Understand the sensitivity of estimates to environmental factors –Private insurance premium growth –Changes in employment –Demographic change

Types of Models  Determinate models e.g., Number enrolled = f (X 1... X n )+u = f (X 1... X n )+u Calculate population response to a specific change, all else held equal  “Cell-based” (spreadsheet) models (spreadsheet) models Calculate change by population subgroup, using average relationships  Microsimulation models Calculate change by individual using determinate- model relationships and other parameters

Determinate Models Estimate aggregate response to a program change Estimate aggregate response to a program change Support sensitivity analysis only by subgroups identified in the model Support sensitivity analysis only by subgroups identified in the model “Broad brush” approach, not suited to “fine-grained” analysis of complex systems and interactions “Broad brush” approach, not suited to “fine-grained” analysis of complex systems and interactions Offer a measure of precision of estimates, or a confidence interval Offer a measure of precision of estimates, or a confidence interval

Spreadsheet Model Mimics the operation of a program or system Mimics the operation of a program or system Incorporates average behavior by population subgroup Incorporates average behavior by population subgroup Relies on assumptions borrowed from populations that may differ in unmeasured ways Relies on assumptions borrowed from populations that may differ in unmeasured ways Outputs subgroup estimates only as defined in the model Outputs subgroup estimates only as defined in the model Relatively inexpensive and fast to assemble Relatively inexpensive and fast to assemble

Example: Simple Spreadsheet Model of the Dirigo Program Number of persons by subgroup Calculate subgroup offer and take up from available evidence Calculate financing specific to subgroups: State funds Federal match Employer contribution Individual premiums Sum across subgroups Sensitivity analyses

Microsimulation Model Mimics the operation of a program or system Mimics the operation of a program or system Operates on a large database (e.g., CPS) and outputs the same database with variables of interest calculated Operates on a large database (e.g., CPS) and outputs the same database with variables of interest calculated Incorporates partial responses to many variables at the level of the individual to calculate final response Incorporates partial responses to many variables at the level of the individual to calculate final response Supports relatively flexible analysis of subgroups and sensitivity to assumptions Supports relatively flexible analysis of subgroups and sensitivity to assumptions Often reveals results of complex logical relationships you might otherwise have overlooked Often reveals results of complex logical relationships you might otherwise have overlooked

Population input data file Create insurance families Apply parameters/relationship estimates to calculate individual take up probability Calculate stochastic estimate of take up Population output data file Subgroup and sensitivity analyses; refine actuarial estimates as needed Example: Simple Microsimulation Model of the Dirigo Program Constrain take up by eligibility rules Develop initial actuarial cost estimates

Choosing the Right Model How precisely can you identify what you need to know? How precisely can you identify what you need to know? How much detail do you need to know? How much detail do you need to know? How much time and budget do you have? How much time and budget do you have? What data are available to inform the model? What data are available to inform the model?

Designing the Model What are the key policy questions? What are the key policy questions? How many people will enroll? What woodwork effects? Change in program cost? What measures address the policy questions? What measures address the policy questions? Insurance family membership and income, actuarial factors (family type, family size, age, gender, location) What subgroups or other responses are of special interest? What program features might be changed? What subgroups or other responses are of special interest? What program features might be changed? Eligibility groups (parents, childless adults by income), impacts on linked programs

Structural Issues What program features and system relationships are important? What program features and system relationships are important? –Eligibility rules, outreach activities, income disregards –Funding caps or links to available funds What environmental variables should the model include? What sensitivity analyses do you want? What environmental variables should the model include? What sensitivity analyses do you want? –Aggregate cost growth –Changes in the industry, firm size, or wage structure of employment

Data Issues Do available data include the population of interest to you? Do available data include the population of interest to you? Are adequate estimates of behavioral response already available from the literature? Are adequate estimates of behavioral response already available from the literature? Can available data be “enhanced” to improve sample size and precision? Can available data be “enhanced” to improve sample size and precision?

Enhancing State Data for Modeling Merge population samples (e.g., 3 CPS years) Merge population samples (e.g., 3 CPS years) –Sample overlap –Data are observations of real residents, but trends may be lost “Balance” a national or regional sample “Balance” a national or regional sample –National/regional data are “raked” (re-weighted) to state totals –Data are synthetic, but reflect the most recent time period on key (control) variables

When Do You Need an Actuary? You do not need an actuary to estimate enrollment You do not need an actuary to estimate enrollment You do need an actuary to estimate per capita cost when: You do need an actuary to estimate per capita cost when: –Enrollee demographics are likely to change from past experience –The benefit design is new –The benefit design must be targeted to a cost cap

What Does an Actuary Need? Estimated eligibles and enrollment, by Estimated eligibles and enrollment, by –Individual age and gender –Family type and size –Geographic location –Benefit design option

Key Lessons for Modeling Policy Be clear and selective about what the model absolutely has to do Be clear and selective about what the model absolutely has to do Be pragmatic about structure Be pragmatic about structure If you choose microsimulation: If you choose microsimulation: –Modules are easier to build, debug, and update –Full integration can be costly and unnecessary –When possible, estimate model performance and benchmark

More Lessons Build into the model what you need out of it Build into the model what you need out of it –Variables for subgroup analysis –Parameters for sensitivity analyses Be aware of tautologies and power Be aware of tautologies and power –Are the results showing you only the input assumptions? –Do just a few observations drive the result?