1. An Assessment of the Impact of Two Distinct Survey Design Modifications on Health Insurance Coverage Estimates in a National Health Care Survey Steven B. Cohen, Trena Ezzati-Rice, and Marc Zodet International Total Survey Error Workshop 2010 Stowe, VT June 13-16, 2010

2. Background and motivation for research The Medical Expenditure Panel Survey (MEPS) is a national resource to inform health care policy The Medical Expenditure Panel Survey (MEPS) is a national resource to inform health care policy MEPS is a key survey resource to monitor: MEPS is a key survey resource to monitor: – Trends in estimates of the uninsured – Population characteristics of the uninsured – Duration of spells of uninsurance and long term uninsured – Financial consequences of being uninsured – Relationship between uninsurance and health status Thus need for timely, high quality MEPS data Thus need for timely, high quality MEPS data

3. MEPS survey background Annual survey since 1996; nationally representative sample of households Annual survey since 1996; nationally representative sample of households 5 rounds of data collection covering 2 calendar years 5 rounds of data collection covering 2 calendar years Used to estimate medical care utilization, access to care, and health care expenses for the U.S. civilian noninstitutionalized population Used to estimate medical care utilization, access to care, and health care expenses for the U.S. civilian noninstitutionalized population Integrated survey design Integrated survey design – Each annual sample is a subsample of responding households (from prior year) from another large ongoing U.S. health survey, the National Health Interview Survey (NHIS) Overlapping panel design Overlapping panel design – Data from 1st year of new panel combined with data from 2nd year of previous panel

4. Illustration of how panels/rounds comprise MEPS calendar year data

5. MEPS survey design modifications in 2007: Panel 12 1. Re-engineered CAPI survey instrument: Windows-based platform replaced the DOS-based system (questionnaire remained virtually unchanged) 1. Re-engineered CAPI survey instrument: Windows-based platform replaced the DOS-based system (questionnaire remained virtually unchanged) 2. New sample design resulting from the sample redesign of the NHIS in 2006 2. New sample design resulting from the sample redesign of the NHIS in 2006 – MEPS Panel 12 fielded in January 2007 was the 1st Panel to be selected based on the new NHIS sample – New NHIS sample design conceptually very similar to the 1995-2005 design: complex area probability sample – Changes in 2006 NHIS design that affected 2007 MEPS  Sampling PSUs and SSUs independent of those sampled under the previous design  Previously only HHs with Black and Hispanic persons were oversampled - New – oversampling of Asian persons

6. Goals of this research Evaluate if any significant impact on MEPS estimates of insurance coverage as a result of the dual survey changes implemented in Panel 12 which began January 2007 Evaluate if any significant impact on MEPS estimates of insurance coverage as a result of the dual survey changes implemented in Panel 12 which began January 2007 Evaluate the effectiveness of the MEPS nonresponse adjustment strategies in reducing potential bias Evaluate the effectiveness of the MEPS nonresponse adjustment strategies in reducing potential bias

7. Two evaluation approaches 1. Internal survey comparisons 1. Internal survey comparisons – takes advantage of MEPS overlapping panel design  compare the two individual panel specific estimates of health insurance coverage for 2007 and other years – conduct multivariate analysis to determine if panel is a significant predictor of insurance status after controlling for other independent variables 2. External comparisons 2. External comparisons – takes advantage of MEPS linkage to NHIS survey – after MEPS household nonresponse and survey attrition, can insurance estimates for the sampling frame source (NHIS) be replicated using just the MEPS survey respondents and the MEPS nonresponse adjusted weights?

8. Sources of data 2007 MEPS compared to 2004-2006 2007 MEPS compared to 2004-2006 – Panel specific estimates within each calendar year (using panel specific weights) 2003-2006 NHIS data 2003-2006 NHIS data

9. Evaluation Approach 1: Internal MEPS Comparison Step 1: Panel specific estimates within years Step 2: Multivariate analysis

10. Step 1: Evaluation of concordance of panel specific health insurance coverage estimates Focus is on 2007 coverage estimates Focus is on 2007 coverage estimates  Panel 12 – dual survey changes  Panel 11 – original sample design and original DOS-based survey instrument

11. MEPS annual estimation weights Each panel is weighted separately and then combined Each panel is weighted separately and then combined Panel specific weight is a function of: Panel specific weight is a function of: – NHIS base weight – Poststratification to NHIS full sample – Nonresponse adjustment for dwelling unit nonresponse and survey attrition within year 1 and year 2 – Final raking adjustment to CPS population control totals

12. Evaluating the joint effect of the dual survey design changes in 2007 Comparison of panel specific national health insurance coverage estimates for the population under age 65 Measures evaluated: Measures evaluated: – 1) some private coverage during the year – 2) public-only coverage during the year – 3) full year uninsured

13. Estimates of health insurance coverage for the civilian non-institutionalized population <65 years of age by panel within year CalendarCoverage Year 2 in Panel Year 1 in Panel YearMeasure%(SE)%(SE) 2007 Any private 69.2(0.83)68.7(1.04) Public only 15.8(0.61)15.8(0.73) Uninsured15.0(0.53)15.5(0.68) 2006 Any private 69.9(0.81)70.9(0.78) Public only 15.6(0.56)15.0(0.56) Uninsured14.5(0.51)14.1(0.48) 2005 Any private 70.7(0.83)70.8(0.72) Public only 15.3(0.62)15.1(0.53) Uninsured14.0(0.48)14.2(0.47) 2004 Any private 70.5(0.85)71.7(0.80) Public only 15.2(0.60)14.4(0.57) Uninsured14.3(0.49)13.9(0.51) Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07

14. Step 2: Multivariate analysis Logistic regression analysis to test for a panel effect controlling for other explanatory variables related to health insurance coverage Logistic regression analysis to test for a panel effect controlling for other explanatory variables related to health insurance coverage Separate models by year: 2004-2007 Separate models by year: 2004-2007

15. Covariates included in the models Panel classification Panel classification Sex Sex Race/ethnicity Race/ethnicity Self-reported health status Self-reported health status Marital status Marital status Education Education Income Income MSA, Census region MSA, Census region Total healthcare expenses Total healthcare expenses

16. 2007200620052004 Degrees of Freedom Wald F P-value P-value P-value P-value Overall Model 2694.84<0.001120.92<0.001101.55<0.00184.21<0.001 Model minus intercept 2542.66<0.00166.92<0.00150.91<0.00150.28<0.001 Panel classification 11.790.18140.060.81350.340.56270.030.8644 Sex171.57<0.00166.09<0.00175.24<0.00169.68<0.001 Race/ethnicity353.83<0.00142.42<0.00156.03<0.00147.57<0.001 Health status 42.150.07372.720.03021.180.31944.900.0008 Marital status 414.84<0.00113.02<0.00118.21<0.00126.26<0.001 Highest year of education 412.31<0.00132.71<0.00115.37<0.00112.69<0.001 Poverty status 469.96<0.001103.81<0.00171.14<0.00179.53<0.001 Region321.00<0.0019.01<0.00111.96<0.00117.58<0.001 Total health care expenditures 134.88<0.00168.44<0.00132.47<0.00135.68<0.001 -2 x Normalized Log- Likelihood Full Model: 13,422.914,319.714,223.514,052.5 Pseudo Model R-Square: 0.15500.15880.15160.1569 Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07 Note: In logistic regression analysis, Y=1 specifies full year uninsured, Y=0 specifies other Final logistic regression model of the uninsured, ages 18-64 years; testing for panel effect

17. 2007200620052004 Degrees of Freedom Wald F P-value P-value P-value P-value Overall Model 26105.10<0.001140.17<0.001108.79<0.001100.36<0.001 Model minus intercept 2589.83<0.001130.70<0.001105.91<0.00192.48<0.001 Panel classification 12.640.10482.390.12340.310.57630.510.4749 Sex117.37<0.00110.730.001213.420.000328.81<0.001 Race/ethnicity341.34<0.00134.14<0.00149.22<0.00145.73<0.001 Health status 420.36<0.00131.84<0.00132.96<0.00121.91<0.001 Marital status 434.08<0.00128.26<0.00133.95<0.00137.65<0.001 Highest year of education 441.56<0.00176.06<0.00164.07<0.00158.32<0.001 Poverty status 4245.13<0.001308.26<0.001261.46<0.001282.70<0.001 Region33.700.01193.990.00850.490.69062.090.1027 Total health care expenditures 119.94<0.00157.94<0.00130.63<0.00134.51<0.001 -2 x Normalized Log- Likelihood Full Model: 14555.515,608.215,268.015,354.9 Pseudo Model R-Square: 0.28370.28540.29180.2883 Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07 Note: In logistic regression analysis, Y=1 specifies some private insurance in year, Y=0 specifies other Final logistic regression model of the privately insured, ages 18-64 years; testing for panel effect

18. Evaluation Approach 2: External Evaluation Takes advantage of the linkage of the MEPS to the NHIS Takes advantage of the linkage of the MEPS to the NHIS National estimates of health insurance are derived from NHIS (NHIS insurance variables) for two analytical samples: National estimates of health insurance are derived from NHIS (NHIS insurance variables) for two analytical samples: – Total NHIS sample → NHIS weights – Reduced NHIS sample obtained from matching with MEPS full year respondents → MEPS weights as adjusted for MEPS nonresponse Since this analysis examines only NHIS insurance data, the estimates are not impacted by the MEPS CAPI redesign Since this analysis examines only NHIS insurance data, the estimates are not impacted by the MEPS CAPI redesign – Thus, assessment of joint impact of the MEPS sample redesign and effectiveness of MEPS nonresponse adjustments.

19. MEPS-NHIS linked analysis The following 4 NHIS measures of health insurance coverage were examined: any coverage at the time of the interview (covered; not covered; refused/not ascertained/DK) any coverage at the time of the interview (covered; not covered; refused/not ascertained/DK) private coverage at the time of the interview (mentioned; not mentioned; refused/not ascertained/DK) private coverage at the time of the interview (mentioned; not mentioned; refused/not ascertained/DK) Medicaid coverage at the time of the interview (mentioned; not mentioned; refused/not ascertained/DK) Medicaid coverage at the time of the interview (mentioned; not mentioned; refused/not ascertained/DK) No health insurance coverage for more than one year (yes, no; refused/not ascertained/DK) No health insurance coverage for more than one year (yes, no; refused/not ascertained/DK)

20. NHIS Variable Using the MEPS Full Year panel specific weight for calendar year respondents Using the full sample NHIS and NHIS weight (prior year) Calendar Year NOTCOV (Health insurance coverage status) %SE%SE 2007 Not covered 15.3* (0.69)16.8(0.29) Covered 83.7* (0.71)82.0(0.29) Refused/NA/DK1.1(0.18)1.2(0.08) 2006 Not covered 15.9(0.53)16.3(0.24) Covered83.3(0.56)82.8(0.24) Refused/NA/DK0.7(0.12)1.0(0.05) 2005 16.5(0.54)16.2(0.23) Covered82.7(0.56)82.8(0.24) Refused/NA/DK0.8(0.14)1.0(0.06) 2004 15.7(0.60)16.3(0.26) Covered83.4(0.61)82.6(0.27) Refused/NA/DK0.9(0.14)1.1(0.06) Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07; National Health Interview Survey (NHIS), 2004-07 *significant difference in MEPS derived NHIS estimate relative to the NHIS derived estimate at the.05 level NHIS any insurance coverage at time of interview derived from MEPS sample and estimation weights compared to NHIS sample and weights, age <65

21. NHIS private insurance derived from MEPS sample and estimation weights compared to NHIS sample and weights, age <65 NHIS Variable Using the MEPS Full Year panel specific weight for calendar year respondents Using the full sample NHIS and NHIS weight (prior year) NHIS weight (prior year) Calendar Year HIKINDA (Private insurance) %SE%SE 2007Mentioned 68.9* (1.00)66.3(0.44) Not Mentioned 30.1* (1.01)32.6(0.44) Refused/NA/DK1.1(0.18)1.2(0.08) 2006Mentioned69.2(0.81)68.3(0.39) Not Mentioned 30.0(0.80)30.7(0.39) Refused/NA/DK0.7(0.12)1.0(0.05) 2005Mentioned69.4(0.80)68.6(0.39) 29.8(0.80)30.4(0.39) Refused/NA/DK0.8(0.14)1.0(0.06) 2004Mentioned70.4(0.85)69.2(0.39) 28.7(0.84)29.7(0.39) Refused/NA/DK0.9(0.14)1.1(0.06) Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07; National Health Interview Survey (NHIS), 2004-07 *significant difference in MEPS derived NHIS estimate relative to the NHIS derived estimate at the.05 level

22. NHIS Medicaid coverage derived from MEPS sample and estimation weights compared to NHIS sample and weights, age <65 NHIS Variable Using the MEPS Full Year panel specific weight for calendar year respondents Using the full sample NHIS and NHIS weight (prior year) Calendar Year HIKINDD (Medicaid) %SE%SE 2007Mentioned8.5(0.66)9.2(0.25) Not Mentioned 90.4(0.66)89.7(0.26) Refused/NA/DK1.1(0.18)1.2(0.08) 2006Mentioned8.2(0.41)8.8(0.21) 91.0(0.42)90.3(0.21) Refused/NA/DK0.7(0.12)1.0(0.05) 2005Mentioned7.9(0.46)8.5(0.20) 91.3(0.47)90.5(0.20) Refused/NA/DK0.8(0.14)1.0(0.06) 2004Mentioned7.8(0.38)8.6(0.21) 91.4(0.40)90.3(0.22) Refused/NA/DK0.9(0.14)1.1(0.06) Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07; National Health Interview Survey (NHIS), 2004-07 *significant difference in MEPS derived NHIS estimate relative to the NHIS derived estimate at the.05 level

23. NHIS insurance coverage for >1 year derived from MEPS sample and estimation weights compared to NHIS sample and weights, age 1 year derived from MEPS sample and estimation weights compared to NHIS sample and weights, age <65 NHIS Variable Using the MEPS Full Year panel specific weight for calendar year respondents Using the full sample NHIS and NHIS weight (prior year) Calendar Year HILAST (No health insurance for more than one year) %SE%SE 2007Mentioned10.7(0.61)11.5(0.24) Not Mentioned 89.0(0.61)87.9(0.25) Refused/NA/DK0.2(0.05)0.5(0.04) 2006Mentioned10.7(0.43)11.1(0.20) 88.8(0.45)88.3(0.21) Refused/NA/DK0.5(0.12)0.6(0.04) 2005Mentioned11.2(0.43)11.1(0.18) 88.5(0.44)88.4(0.19) Refused/NA/DK0.3(0.06)0.5(0.04) 2004Mentioned10.3(0.45)10.8(0.21) 89.3(0.46)88.6(0.21) Refused/NA/DK0.4(0.09)0.6(0.05) Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07; National Health Interview Survey (NHIS), 2004-07 *significant difference in MEPS derived NHIS estimate relative to the NHIS derived estimate at the.05 level

24. What did we learn? MEPS overlapping panel design facilitates the evaluation of periodic survey design changes. MEPS overlapping panel design facilitates the evaluation of periodic survey design changes. The linkage of MEPS and NHIS also facilitates data quality assessments. The linkage of MEPS and NHIS also facilitates data quality assessments. The internal comparison of MEPS insurance estimates (<65) by panel for 2007 and for years prior to the redesign did not show any significant differences. The internal comparison of MEPS insurance estimates (<65) by panel for 2007 and for years prior to the redesign did not show any significant differences. The logistic regression analysis likewise did not reveal any significant effect for Panel. The logistic regression analysis likewise did not reveal any significant effect for Panel.

25. What did we learn (cont.)? Phase 2 of this study evaluated the joint impact of the new 2007 sample design and the survey’s post-survey adjustments. Phase 2 of this study evaluated the joint impact of the new 2007 sample design and the survey’s post-survey adjustments. The MEPS-NHIS linked analysis only showed modest differences in 2 of 4 NHIS insurance variables examined. The MEPS-NHIS linked analysis only showed modest differences in 2 of 4 NHIS insurance variables examined. No evidence of nonresponse bias attributable to year 1 survey attrition. No evidence of nonresponse bias attributable to year 1 survey attrition. Limitation of this study: We could not tease out the independent effects of the two survey modifications. Limitation of this study: We could not tease out the independent effects of the two survey modifications. Nevertheless, the dual survey changes in 2007 did not appear to have a major impact on insurance coverage estimates. Nevertheless, the dual survey changes in 2007 did not appear to have a major impact on insurance coverage estimates.

26. Future research Assess the 2008 MEPS panel specific estimates Assess the 2008 MEPS panel specific estimates  CAPI platform and sample design will be the same in both individual overlapping panels. Continued evaluations of variables used to adjust for nonresponse. Continued evaluations of variables used to adjust for nonresponse. Evaluate transitions in insurance coverage from NHIS to MEPS, pre-and post-new design. Evaluate transitions in insurance coverage from NHIS to MEPS, pre-and post-new design. Evaluate trends in transitions in coverage in MEPS from year 1 to year 2, pre- and post new design. Evaluate trends in transitions in coverage in MEPS from year 1 to year 2, pre- and post new design. Review CAPI programming and edit specifications. Review CAPI programming and edit specifications.

27. Discussion Comments on the internal and external evaluation strategies used in this study? Comments on the internal and external evaluation strategies used in this study? What other strategies have been (or can be) used to evaluate survey data quality following implementation of design or methodological modifications? What other strategies have been (or can be) used to evaluate survey data quality following implementation of design or methodological modifications?