1. Does Inclusion of Both Partial and Complete Interviews from the Source Sampling Frame Have an Effect on Nonresponse Error and Measurement Error in a National Health Survey? Trena M. Ezzati-Rice, Frederick Rohde, Steven B. Cohen International Total Survey Error Workshop 2010 Stowe, VT June 13-16, 2010

2. Motivation for research Large observed differences in response rates between two types of previously interviewed cases included in the Medical Expenditure Panel Survey (MEPS) sample as a result of its integration with another national health survey Large observed differences in response rates between two types of previously interviewed cases included in the Medical Expenditure Panel Survey (MEPS) sample as a result of its integration with another national health survey

3. MEPS survey background Annual survey since 1996; nationally representative sample of households Annual survey since 1996; nationally representative sample of households 5 rounds of data collection covering 2 calendar years 5 rounds of data collection covering 2 calendar years Used to estimate medical care utilization, access to care, and health care expenses for the U.S. civilian noninstitutionalized population Used to estimate medical care utilization, access to care, and health care expenses for the U.S. civilian noninstitutionalized population Widely used to inform health care policy Widely used to inform health care policy Integrated survey design Integrated survey design – Each annual panel of households is a subsample of responding households (from prior year) from another large ongoing U.S. health survey, the National Health Interview Survey (NHIS)

4. Integration of NHIS and MEPS MEPS inherits the NHIS complex sample design with its oversampling of selected minority populations MEPS inherits the NHIS complex sample design with its oversampling of selected minority populations Reduces household screening costs for MEPS Reduces household screening costs for MEPS Provides valuable sampling frame information to adjust for survey nonresponse Provides valuable sampling frame information to adjust for survey nonresponse Provides enhanced analytical potential (NHIS and MEPS linked analyses) Provides enhanced analytical potential (NHIS and MEPS linked analyses) MEPS response rate is conditioned on the NHIS response rate MEPS response rate is conditioned on the NHIS response rate MEPS inherits the composition of the NHIS interviewed sample MEPS inherits the composition of the NHIS interviewed sample – Completed household interviews – “Partially Completed” household interviews

5. Definition of an NHIS “partially completed” interview All questionnaire modules are not completed All questionnaire modules are not completed NHIS questionnaire composed of 5 major sections NHIS questionnaire composed of 5 major sections 1. Household composition (demographic information) 1. Household composition (demographic information) 2. Family core (basic health and sociodemographic data) 2. Family core (basic health and sociodemographic data) Family relationships, marital status Family relationships, marital status Health status, activity limitations, injury, poisoning Health status, activity limitations, injury, poisoning Access and utilization and citizenship Access and utilization and citizenship Health insurance Health insurance Educational attainment STOP Educational attainment STOP Employment, earnings, or income data for partials NO Data – 3. Sample adult qx. NO Data – 4. Sample child qx. NO Data – 5. Immunization section NO Data

6. Data Matrix: NHIS and MEPS sample integration NHIS complete Interviews MEPS Respondents MEPS Nonrespondents MEPS Respondents MEPS Nonrespondents NHIS partially completed interviews NHIS Sample Frame (NHIS Respondents eligible for MEPS)

7. Background for this research Previously conducted research (Chiu et al, 2001) indicated that “late”/difficult” NHIS interviews were more likely to be partial completes. Previously conducted research (Chiu et al, 2001) indicated that “late”/difficult” NHIS interviews were more likely to be partial completes. Partially completed NHIS cases in the MEPS sample observed to: Partially completed NHIS cases in the MEPS sample observed to:  require more field effort  have lower response rates compared to the NHIS complete cases

8. Research questions Do the NHIS partially completed cases as included in the MEPS Do the NHIS partially completed cases as included in the MEPS – Have lower response propensity at Round 1 and subsequent rounds? – Adversely affect data collection burden? Are there differences in the demographic and socioeconomic characteristics of the MEPS respondents when stratified by their NHIS interview status (partial versus complete)? Are there differences in the demographic and socioeconomic characteristics of the MEPS respondents when stratified by their NHIS interview status (partial versus complete)? Do the NHIS partially completed cases as carried over to MEPS affect survey data quality? Do the NHIS partially completed cases as carried over to MEPS affect survey data quality? – Do they have higher item nonresponse rates on MEPS variables? – Do MEPS survey estimates differ when stratified by partial vs. complete NHIS interview status?

9. Source of Data MEPS MEPS – Historical data for examining response rate trends – Panels 11 & 12 for detailed analyses – Paradata (survey burden assessment) – Survey data (item nonresponse and weighted survey estimates) NHIS NHIS – Paradata (NHIS interview status and other data) – Survey data (response propensity modeling )

10. Percentage of MEPS reporting units with a previous NHIS partially completed interview: MEPS Panels 3-14 Year MEPS Panel Started Panel NHIS “partially completed” interview (%) 1998310 1999421 2000524 2001622 2002717 2003820 2004919 20051016 20061123 20071219 20081325 20091426

11. MEPS Round 1 response rates (reporting unit level) by NHIS interview completion status: 2004-2009

12. Percent refused MEPS interview (at the RU level) by round and NHIS interview status, MEPS Panel 12 Round NHIS Partial (%) NHIS Complete (%) 128.7*15.9 210.3* 5.5 5.5 3 4.7 4.7 3.6 3.6 43.2 2.6 2.6 51.8 1.1 1.1 *p<.001

13. Percent ever refused (reluctant respondent or final refusal) by Round and NHIS interview completion status: RU level MEPS Panel 12

14. Percent of noncontacts and true contacts by NHIS interview status: MEPS Panel 12, Round 1 ParadataNHISpartialNHIScompleteSignif Round 1 noncontacts 4+ 4+34.726.9p<.001 Round 1 “true” contacts 1 2-3 2-3 4-5 4-5 6-10 6-10 11-14 11-14 15+ 15+ 11.142.121.518.53.73.013.351.217.513.82.51.6p<.001

15. Other paradata measures by NHIS interview status: MEPS Panel 12, Round 1 Paradata NHIS partial NHIS complete Signif Reluctant respondent (%) 14.38.4p<.001 Not located (%) 6.13.3p<.001 MEPS interview type In person (%) In person (%) Phone (%) Phone (%)90.2 9.8 9.893.86.2p<.001 Mean interview time (minutes) 110.3107.3p=.232 Any break-offs during the interview (%) 10.79.2p=.006

16. Bivariate analysis of R1 respondent characteristics (reference person) by NHIS interview completion status: MEPS Panels 11 & 12 combined Bivariate analysis of R1 respondent characteristics (reference person) by NHIS interview completion status: MEPS Panels 11 & 12 combined Characteristic Overall (%) Partial (%) Complete(%) Age of Reference person* 65+ 65+18.113.019.2 Race/ethnicity * Hispanic Hispanic Non-Hispanic Black Non-Hispanic Black Non-Hispanic Asian Non-Hispanic Asian Non-Hispanic Other Non-Hispanic Other19.717.44.658.326.517.9 6.7 6.748.918.217.2 4.2 4.260.4 Marital Status* Married Married Widowed/separated/divorced Widowed/separated/divorced Never married Never married49.828.321.856.923.419.648.229.422.3 Year 1 Poverty Status* In/near poverty In/near poverty Low income Low income Middle income Middle income High income High income Missing Missing23.514.225.126.211.022.813.124.324.515.323.714.525.326.610.0 * significant at alpha = 0.05

17. Bivariate analysis (cont.) of R1 respondent characteristics (MEPS reference person) by NHIS interview completion status: MEPS Panels 11 & 12 combined (* = significant at alpha =.05) Characteristic Overall (%) Partial (%) Complete (%) Size of RU* 1 2 3 4 5+ 5+28.928.515.814.312.419.125.419.218.517.831.129.215.113.411.2 Employment status* Employed Employed65.369.464.4 Region* Northeast Northeast Midwest Midwest South South West West16.321.238.024.517.318.135.828.816.021.938.523.6 MSA status* MSA MSA Non MSA Non MSA83.716.387.912.182.717.3

18. Logistic regression analysis of Round 1 (RU) respondent (reference person charac) being a prior NHIS partial complete: MEPS Panels 11 & 12 EffectDFWald X 2 Pr > X 2 EffectDFWald X 2 Pr > X 2 Race/ethnicity352.1<.0001 Yr1 poverty status541.9<.0001 No. of people in RU421.8.0002 Family structure528.5<.0001 Employment status215.0.0006 Region310.2.0167 MSA status115.3<.0001

19. Odds ratios of logistic model predicting a “NHIS Partial Complete” respondent at MEPS Round 1, Panels 11 & 12 Race/ethnicityMSA Asian1.61* MSA 1.31* Asian1.61* MSA 1.31* Black1.31* Non-MSA1.00 Black1.31* Non-MSA1.00 Hispanic1.45* Hispanic1.45* White/Other1.00 White/Other1.00 Reporting Unit SizePoverty Status 10.26* In poverty1.01 10.26* In poverty1.01 20.57* Low income0.95 20.57* Low income0.95 30.77* Middle income1.01 30.77* Middle income1.01 40.90 Missing income1.52* 40.90 Missing income1.52* 5+1.00 Near poverty1.10 5+1.00 Near poverty1.10 High income1.00 High income1.00Region Midwest0.84*Employment Status Midwest0.84*Employment Status Northeast0.95 Employed1.14* Northeast0.95 Employed1.14* South0.85* Missing/<16 yr2.42* South0.85* Missing/<16 yr2.42* West1.00 Unemployed1.00 West1.00 Unemployed1.00

20. Predicting Round 1 dwelling unit level response propensity -- 29 potential (NHIS) covariates as used in MEPS weights production Demographic Household Characteristics Socio- Economic Status Geographic Health- related Age ref. person DU size Poverty status Census region Health status Race/ethnicity Has phone Education MSA size Need help Marital status Working or not IncomeMSA/nonMSA # nts. in hospital Gender Type of PSU Employment status Urban/Rural Healthcare coverage Any Asian Type of home – house, apt., etc. Home ownership Medical expenses category Any Black Time w/ no phone Interview language U.S. citizen U.S. born

21. Additional NHIS para data included in logit regression (predicting round 1 response propensity) -- * available for MEPS panel 12 only NHIS interview completion status (partial vs. complete) NHIS interview completion status (partial vs. complete) – Panels 11 and 12 Household cooperativeness* Household cooperativeness* How likely respond to later linked survey* How likely respond to later linked survey* # of contacts and # of non-contacts* # of contacts and # of non-contacts* Language problem* Language problem* Health problem* Health problem* Time constraints* Time constraints* Content/privacy concerns* Content/privacy concerns* Hostility mentioned* Hostility mentioned*

22. Results of logistic regression predicting round 1 DU level response propensity -- MEPS Panel 11 EffectDF Wald X 2 Pr > X 2 EffectDF Wald X 2 Pr > X 2 Predicted poverty1 12.6.0004 Age4 10.4.0338 Race/ethnicity 3 24.6<.0001 Marital status1 6.3.0121 Income4 48.3<.0001 DU size4 36.5<.0001 MSA status2 21.0<.0001 Region3 18.7.0003 Has phone3 27.0<.0001 Home type1 7.8.0053 Interview language2 6.9.0325 U.S. born 2 11.8.0028 NHIS interview completion status1105.4<.0001

23. Results of logistic regression predicting round 1 response propensity -- MEPS Panel 12 (additional NHIS paradata available) EffectDFWald X 2 Pr > X 2 EffectDFWald X 2 Pr > X 2 Race/ethnicity 333.9 <.0001 Marital status1 4.2.0405 Income415.3.0041 DU size421.3.0003 MSA status2 6.0.0496 Region321.0.0001 Has phone319.6.0002 PSU type1 4.5.0340 Interview language212.3.0021 Health status211.2.0037 NHIS interview completion status1 3.9.0500 Med expend category4 1.4.0343 HH cooperativeness428.3 <.0001 Likely to respond to later survey110.6.0011 # contacts w/ sample unit635.5 <.0001 # noncontacts530.7 <.0001 Privacy concerns120.6 <.0001 Hostility mentioned1 5.5.0194

24. RU level item nonresponse rates, MEPS Panel 12 (if any respondent in the RU had a missing value in year 1) VariableNHIS Partial (%) NHIS Complete (%) Signif Yr1 poverty status 15.810.3p<.001 Education3.92.1p=.002 High blood pressure (>17) 2.40.9p<.001 High cholesterol (>17) 2.31.2p=.025 Diabetes diagnosis 1.50.8p=.059 How often dental check up 5.32.5p<.001 How long last routine check up (>17) 10.76.4p<.001 How long last flu shot (>17) 9.64.9p<.001 How long since mammogram (>29) 8.64.5p<.001 How long since last PSA (>39, M) 17.010.5p<.001 Usual source of care 5.22.2p<.001 Employment status 3.72.3p=.014 Mental health status 2.11.1p=.014 Health status 1.60.9p=.059

25. Comparison of selected Year 1 estimates (percents) according to NHIS interview completion status: MEPS Panels 11 & 12 (* = signif p <.05) Person level health Panel 11 Panel 12 measurePartialCompletePartialComplete Insurance (<65) Any private Any private68.371.4*65.068.9 Public only Public only15.215.317.415.9 Uninsured Uninsured16.513.317.615.2 Any activity limitation* 7.310.46.19.5 High cholesterol dx (>17)* 20.725.524.430.5 Office based provider visit* 68.075.265.772.6 Rx (including refills)* 58.665.255.163.4 Total mean expenditures* $2,760$3,815$3,171$3,943 5+ contacts 5+ contacts$2,484 $3,390* $3,390*$3,021$3,823

26. MEPS Year 1 estimates (in percent) for All MEPS respondents versus excluding NHIS partial cases (person weights for NHIS completes re-calculated treating partials as nonrespondents) (* = signif p<.05) Person level health Panel 11 Panel 12 measureAllCompletes Only (rewt) All Completes Only (rewt) Insurance (<65) Any private Any private70.771.168.168.6 Public only Public only15.315.516.216.0 Uninsured Uninsured14.013.515.715.3 Any activity limitation 9.810.0*8.99.1 High cholesterol dx (>17)* 24.525.129.530.0 Office based provider visit* 73.774.771.472.1 Rx (including refills)* 63.864.461.962.6 Total mean expenditures $3,593$3,723*$3,802$3,836 5+ contacts 5+ contacts$3,162$3,313*$3,647$3,702

27. Summary We examined the impact of the carry-over of two types of prior interview cases in the MEPS as a result of its integration with the NHIS. We examined the impact of the carry-over of two types of prior interview cases in the MEPS as a result of its integration with the NHIS. In particular, we wanted to examine if the partially completed NHIS cases may bias MEPS survey estimates and impact survey burden. In particular, we wanted to examine if the partially completed NHIS cases may bias MEPS survey estimates and impact survey burden. NHIS partial cases had significantly higher ever refused rates, higher contact rates, lower response rates at MEPS Round 1, and higher attrition rates. NHIS partial cases had significantly higher ever refused rates, higher contact rates, lower response rates at MEPS Round 1, and higher attrition rates. Partial versus complete was a significant predictor of MEPS Round 1 response propensity controlling for other variables. Partial versus complete was a significant predictor of MEPS Round 1 response propensity controlling for other variables.

28. Summary (cont.) Item nonresponse rates were higher for the partially completed cases relative to the NHIS completes for a number of MEPS key variables. Item nonresponse rates were higher for the partially completed cases relative to the NHIS completes for a number of MEPS key variables. Estimates for several selected health items were different between the NHIS partials and completes as carried over in the MEPS sample. Estimates for several selected health items were different between the NHIS partials and completes as carried over in the MEPS sample. Excluding the NHIS partial cases from the MEPS survey estimates (and re-weighting) resulted in slightly higher estimates of selected conditions, medical events, and expenditures. Excluding the NHIS partial cases from the MEPS survey estimates (and re-weighting) resulted in slightly higher estimates of selected conditions, medical events, and expenditures. Thanks Thanks

29. Discussion Initial exploratory analysis Initial exploratory analysis – What additional analyses can be carried out to assess potential nonresponse bias and measurement error bias related to the two types of interview cases in the MEPS? – What approaches could be explored to correct for nonresponse or measurement error bias? Should MEPS consider excluding or subsampling the NHIS partials in future panels of the MEPS? Should MEPS consider excluding or subsampling the NHIS partials in future panels of the MEPS? Ideas/suggestions for future research? Ideas/suggestions for future research?

30. Logistic regression analysis of the insured (<65 years) reference person: testing for NHIS interview status effect, MEPS, Panels 11 and 12 (RU level analysis) Panel 11 Panel 12 Panel 11 + 12 DF Wald F p-valueDF P-valueDF p-value Age39.13.027614.75.0029321.81<.0001 Sex110.92.000917.67.0056118.61<.0001 Race/ethnicity399.39<.00011111.99<.00013206.47<.0001 Highest year of education 469.78<.0001452.75<.00014122.32<.0001 Poverty status (Y1) 5237.16<.00015174.76<.00015406.38<.0001 Region339.01<.0001353.02<.0001386.51<.0001 Total health care Expenditures6233.63<.00016194.35<.00016417.21<.0001 NHIS Interview Completion Status 10.36.547714.75.029312.93.0868