Evaluating Health Policy through Natural Experiments Andrew B. Bindman, MD Professor Medicine, Health Policy, Epidemiology & Biostatistics UCSF

Insurance Reforms and Other Imminent Policy Changes Many of you identified specific policies that are or are about to be implemented that are relevant to your research interests Many of you identified specific policies that are or are about to be implemented that are relevant to your research interests Opportunity to study this change as a way to inform the policy process Opportunity to study this change as a way to inform the policy process

Learning About Policies Read the newspaper Read the newspaper –NY Times, Washington Post, Politico Health newswire services Health newswire services –California Healthline ( –Kaiser Health News ( Academic faculty Academic faculty Professional Societies/Scientific Organizations Professional Societies/Scientific Organizations Community-based organizations Community-based organizations Directly from policy decision-makers Directly from policy decision-makers

My Research Interest Health consequences of public policies Health consequences of public policies Access to and quality of care for low- income, diverse, and patient populations vulnerable to poor health because of their social circumstances Access to and quality of care for low- income, diverse, and patient populations vulnerable to poor health because of their social circumstances

Medi-Cal: California ’ s Medicaid Program ~8 million beneficiaries ~8 million beneficiaries $41 billion last year $41 billion last year 2 nd largest use of general fund (17%) 2 nd largest use of general fund (17%) Pays for 1 in every 2 births in the state Pays for 1 in every 2 births in the state Approximately half of beneficiaries are Latino Approximately half of beneficiaries are Latino Provides 2/3rds of safety net funding Provides 2/3rds of safety net funding

Medicaid Population: Clinical Question Many Medicaid beneficiaries have interruptions in coverage (churning) Many Medicaid beneficiaries have interruptions in coverage (churning) Many uninsured gain Medicaid coverage when hospitalized Many uninsured gain Medicaid coverage when hospitalized Does Medicaid coverage provided at the time of a hospitalization adequate or do interruptions in Medicaid enrollment have a negative impact on the health of beneficiaries? Does Medicaid coverage provided at the time of a hospitalization adequate or do interruptions in Medicaid enrollment have a negative impact on the health of beneficiaries?

Designing a Research Study Randomized trial Randomized trial –Feasibility?

Designing a Research Study Randomized trial Randomized trial –Unethical and impractical Observational study Observational study –Compare the experiences of beneficiaries who have interruptions in coverage with those who have continuous coverage

Reverse Causality Interruption in coverage might not predict worse health outcome so much as worse health might predict whether or not have interrupted coverage Interruption in coverage might not predict worse health outcome so much as worse health might predict whether or not have interrupted coverage Bias of higher admissions among those with continuous coverage Bias of higher admissions among those with continuous coverage

Designing a Research Study Randomized trial- is it feasible? Randomized trial- is it feasible? Observed variation - is it biased? Observed variation - is it biased? Natural experiment - does a good one exist? Natural experiment - does a good one exist?

Natural Experiments A natural or quasi-experiment is a naturally occurring instance of observable phenomena which approximate or duplicate the properties of a controlled experiment. In contrast to laboratory experiments, these events aren't created by scientists, but yield data which nonetheless can be used to make causal inferences. A natural or quasi-experiment is a naturally occurring instance of observable phenomena which approximate or duplicate the properties of a controlled experiment. In contrast to laboratory experiments, these events aren't created by scientists, but yield data which nonetheless can be used to make causal inferences.experimentlaboratoryexperimentsexperimentlaboratoryexperiments

What Are the Elements of a Good Natural Experiment in Health Policy Policy implementation not biased by patient characteristics such as health status Policy implementation not biased by patient characteristics such as health status Policy can be effectively tied to a “ treatment ” exposed group Policy can be effectively tied to a “ treatment ” exposed group Access to before/after data Access to before/after data

Medicaid Population: Policy Question Federal law requires re-determination of eligibility for beneficiaries at a minimum of every 12 months but states have option to do more frequently Federal law requires re-determination of eligibility for beneficiaries at a minimum of every 12 months but states have option to do more frequently Beneficiaries who do not “ re-sign up ” are dropped from program Beneficiaries who do not “ re-sign up ” are dropped from program Does frequency of a state ’ s re-enrollment process increase the number of beneficiaries with interruptions in coverage and if so is this in turn associated with patients ’ health? Does frequency of a state ’ s re-enrollment process increase the number of beneficiaries with interruptions in coverage and if so is this in turn associated with patients ’ health?

Natural Experiment of Interrupted Medicaid Coverage California extended Medicaid re-enrollment period for all children in California from every 3 to every 12 months on January 1, 2001 California extended Medicaid re-enrollment period for all children in California from every 3 to every 12 months on January 1, 2001 Extension of eligibility re-determination period should be associated with an increase in continuity of Medicaid coverage, but should not except through its influence on continuity of coverage be associated with the health status of children. Extension of eligibility re-determination period should be associated with an increase in continuity of Medicaid coverage, but should not except through its influence on continuity of coverage be associated with the health status of children.

Challenging Issues in Studying Natural Experiments Learning about a policy change as it is about to happen or after the fact makes it harder to collect baseline data Learning about a policy change as it is about to happen or after the fact makes it harder to collect baseline data

Primary Data Collection Can be challenging to organize in time to assess pre-policy condition Can be challenging to organize in time to assess pre-policy condition Lots of work but lots of control over data collection (eg surveys, physiological measures, etc) Lots of work but lots of control over data collection (eg surveys, physiological measures, etc) –Time consuming –Expensive Difficult to maintain over time Difficult to maintain over time

Secondary Databases Pre-existing data that are often collected for an alternative purpose Pre-existing data that are often collected for an alternative purpose Individual level or sometimes aggregate data Individual level or sometimes aggregate data Examples: Examples: –National surveys –Registries –Study cohorts –Administrative data

Secondary Data: Advantages/Challenges Efficient - cheap, fast and often very large Efficient - cheap, fast and often very large Little control on what was collected Little control on what was collected Collection is often longitudinal/repeated cross- sectional Collection is often longitudinal/repeated cross- sectional Potential to analyze temporal changes Potential to analyze temporal changes If source is a payer or a provider may have incomplete capture If source is a payer or a provider may have incomplete capture Can be scooped by others with access to same data Can be scooped by others with access to same data

Finding Secondary Data

Medicaid Data for Studying Interruptions in Coverage Comprehensive and detailed regarding eligibility Comprehensive and detailed regarding eligibility Fee for service claims complete Fee for service claims complete Missing claims information for beneficiaries in managed care Missing claims information for beneficiaries in managed care Won ’ t reflect experience of beneficiaries when they aren ’ t covered Won ’ t reflect experience of beneficiaries when they aren ’ t covered

Statewide Hospital Patient Discharge Abstracts Comprehensive capture of all hospitalizations in state regardless of payer Comprehensive capture of all hospitalizations in state regardless of payer Includes information on hospital admission diagnoses Includes information on hospital admission diagnoses Provides payer source at time of hospitalization Provides payer source at time of hospitalization

Ambulatory Care Sensitive Conditions: AHRQ Prevention Quality Indicators 1. Condition with acute course and window for intervention 2. Condition with chronic course amenable to self-management ACS Conditions Acute Conditions: –Dehydration –Ruptured Appendicitis –Cellulitis –Bacterial Pneumonia –Urinary Tract Infection Chronic Conditions: –Asthma –Hypertension –COPD –Diabetes Mellitus –Heart Failure –Angina

Statewide Hospital Patient Discharge Abstracts Provides payer source at time of hospitalization but not over time Provides payer source at time of hospitalization but not over time Critical question for hospitalizations for ambulatory care sensitive admissions is what the insurance status was prior to the admission since many uninsured gain coverage in association with the hospitalization Critical question for hospitalizations for ambulatory care sensitive admissions is what the insurance status was prior to the admission since many uninsured gain coverage in association with the hospitalization

Linked CA Hospital Discharge and Medicaid Eligibility Files OSHPD: Hospital Discharge Data DHS: Medi-Cal Enrollment Database Demographics Monthly enrollment history Aid Category (e.g. TANF or SSI) FFS, managed care Other insurance Diagnosis (ICD-9 Code) Month/Year of admission Linkage

Pre/Post Study of Re-Enrollment Policy Change for Children Children 1-17 years with a minimum of 1 month of Medicaid coverage in California Children 1-17 years with a minimum of 1 month of Medicaid coverage in California Outcome = time to a hospital admission for an ambulatory care sensitive condition Outcome = time to a hospital admission for an ambulatory care sensitive condition Main predictor = time period Main predictor = time period –Pre policy change = Jan ‘ 99-December ‘ 00 –Post policy change = Jan ‘ 01-December ‘ 02

Children 1-17 Years in California Medicaid Before and After Policy to Change Enrollment N 3,288,171 3,288,1713,230,120 Mean Age (yrs) 99 % Female 5051 Ethnicity (%) Hispanic Hispanic5456 Black Black1312 Asian Asian88 Other Other2524 Aid Group (%) TANF TANF4750 SSI SSI33 Other Other5047 Managed Care (%) 4741

Children with Continuous Medicaid Enrollment by Time Period Years of Enrollment Percentage

Probability of a Hospitalization for an ACS Condition Over Time Months

Children: Adjusted Risk of ACS Hospitalization Relative Hazard P-Value P-Value Post policy <.0001 Age <.0001 Female Ethnicity Hispanic Hispanic <.0001 Black Black <.0001 Asian Asian Other Other <.0001 Aid Group TANF TANF <.0001 SSI SSI <.0001 Managed Care <.0001

Quasi- (natural) Experiments "Estimating the internal validity of a relationship is a deductive process in which the investigator has to systematically think through how each of the internal validity threats may have influenced the data. Then the investigator has to examine the data to test which relevant threats can be ruled out.... When all of the threats can plausibly be eliminated it is possible to make confident conclusions about whether a relationship is probably causal." Cook and Campbell

Limitations Could secular changes other than the policy change explain the observed differences? Could secular changes other than the policy change explain the observed differences?

Strengthening the Design of a Natural Experiment Pre/post changes ideally with a comparison group not exposed to policy Pre/post changes ideally with a comparison group not exposed to policy “ Difference in differences ” “ Difference in differences ” Need to establish conceptual basis for selection of specific comparison group Need to establish conceptual basis for selection of specific comparison group

Potential Comparison Group: Adults in Medi-Cal Medicaid eligibility re-determination period did not change during study period for adults Medicaid eligibility re-determination period did not change during study period for adults Therefore, would not expect a decrease over time in hospitalizations for ambulatory care sensitive conditions among adults Therefore, would not expect a decrease over time in hospitalizations for ambulatory care sensitive conditions among adults

Comparison Group: Adults in Medicaid Adults with Medicaid coverage Adults with Medicaid coverage – = 62% – = 60% Adjusted relative hazard of a hospitalization for an ACS condition for adults in post vs pre period= 1.11 Adjusted relative hazard of a hospitalization for an ACS condition for adults in post vs pre period= 1.11

Second Comparison Group: Children with Continuous Coverage Comparison of children with continuous coverage in each time period revealed no significant difference in hospitalizations for ambulatory care sensitive conditions Comparison of children with continuous coverage in each time period revealed no significant difference in hospitalizations for ambulatory care sensitive conditions Suggests no difference in treatment approach to ambulatory care sensitive conditions over time period of study Suggests no difference in treatment approach to ambulatory care sensitive conditions over time period of study

How Do We Know This is About Access to Ambulatory Care? Hospitalization rates among children for non ambulatory care sensitive conditions (appendicitis and gastrointestinal obstruction) did not change over time Hospitalization rates among children for non ambulatory care sensitive conditions (appendicitis and gastrointestinal obstruction) did not change over time

Most with Interruption in Medicaid Coverage Do Not Have Alternative for Ambulatory Care At the time of hospitalization At the time of hospitalization –59% regain Medi-Cal with admission –7% remain uninsured –33% had another form of insurance

Policy Implications States need to become more aware of the hidden costs in their Medicaid policies States need to become more aware of the hidden costs in their Medicaid policies Continuity of Medicaid coverage can support better health and decrease wasteful spending on hospitalizations that could have been avoided with less costly outpatient care Continuity of Medicaid coverage can support better health and decrease wasteful spending on hospitalizations that could have been avoided with less costly outpatient care

Translating Research into Policy Results of study used Results of study used –in testimony to California legislature to prevent more frequent eligibility re- determination as part of budget cut process –in Congress to support Maintenance of Effort requirements as a part of CHIP reauthorization Published in scientific journals for other states to consider in their policy making Published in scientific journals for other states to consider in their policy making

Are There Opportunities for Randomized Evaluations of Health Policies? Randomized designs are least susceptible to bias Randomized designs are least susceptible to bias Political considerations often make this approach impractical in health policy interventions Political considerations often make this approach impractical in health policy interventions May be opportunities to use a lottery in implementing policies that have more demand than supply (a wait list) May be opportunities to use a lottery in implementing policies that have more demand than supply (a wait list)

Oregon Health Study: Randomized Implementation Opportunity for those yrs <100% FPL otherwise ineligible to obtain Medicaid Opportunity for those yrs <100% FPL otherwise ineligible to obtain Medicaid 85,000 applied but only available for 30,000 85,000 applied but only available for 30,000 Lottery used to randomly select who gained Medicaid coverage Lottery used to randomly select who gained Medicaid coverage

Oregon Health Study: Analytic Plan Comparisons made in utilization and outcomes between those offered Medicaid coverage through lottery and those who were not Comparisons made in utilization and outcomes between those offered Medicaid coverage through lottery and those who were not Intention to treat analysis- some offered did not accept Intention to treat analysis- some offered did not accept –~10,000 of 30,000 selected enrolled Some not offered may have gained coverage through other means Some not offered may have gained coverage through other means

Oregon Health Study: Results Those offered Medicaid were Those offered Medicaid were –70% more likely to have a regular source of care –60% more likely to have a mammogram –20% more likely to have cholesterol screening Also improvements in self reported health status Also improvements in self reported health status

Homework Identify or design a plausible natural experiment to evaluate a policy relevant to your area of research Identify or design a plausible natural experiment to evaluate a policy relevant to your area of research Describe data you could use to study it and possible comparison group(s) Describe data you could use to study it and possible comparison group(s)

Not All Policy Changes Make Good Natural Experiments Voluntary Medicare managed care Voluntary Medicare managed care –voluntary implementation can have health selection bias Expansion of public insurance coverage Expansion of public insurance coverage –uptake by uninsured and “ crowd out ” of privately insured can make it hard to isolate who got “ treatment ” of insurance