Building Wave Response Rates in a Longitudinal Survey: Essential for Nonsampling Error Reduction or Last In - First Out? Steven B. Cohen Fred Rohde and William Yu Agency for Healthcare Research and Quality

Advancing Excellence in Health Care Purpose of Discussion Need for essential longitudinal data on health care coverage, use and expenditures to inform health care policy and practice Need for essential longitudinal data on health care coverage, use and expenditures to inform health care policy and practice Description of the Medical Expenditure Panel Survey (MEPS): purpose, longitudinal design and analytical capacity Description of the Medical Expenditure Panel Survey (MEPS): purpose, longitudinal design and analytical capacity Focus on field efforts to achieve target response rates Focus on field efforts to achieve target response rates

Advancing Excellence in Health Care Purpose of Discussion Evaluations of the quality of the MEPS nonresponse adjustment strategies Evaluations of the quality of the MEPS nonresponse adjustment strategies Determination of characteristics for cases fielded at end of field period (EOF) & conversion of temporary refusals (TNR). Determination of characteristics for cases fielded at end of field period (EOF) & conversion of temporary refusals (TNR). Examine ROI for inclusion of these cases. Examine ROI for inclusion of these cases.

Advancing Excellence in Health Care Purpose of Discussion Examine impact on annual and longitudinal response rates; completion of self administered questionnaires (SAQ) Examine impact on annual and longitudinal response rates; completion of self administered questionnaires (SAQ) Impact on key survey estimates of health insurance coverage and expenditures Impact on key survey estimates of health insurance coverage and expenditures Implications of alternative field strategies Implications of alternative field strategies

Advancing Excellence in Health Care Medical Expenditure Panel Survey (MEPS) Annual Survey of 15,000 households: provides national estimates of health care use, expenditures, insurance coverage, sources of payment, access to care and health care quality Permits studies of: Distribution of expenditures and sources of payment Distribution of expenditures and sources of payment Role of demographics, family structure, insurance Role of demographics, family structure, insurance Expenditures for specific conditions Expenditures for specific conditions Trends over time Trends over time

Advancing Excellence in Health Care Key Features of MEPS-HC Survey of U.S. civilian noninstitutionalized population Survey of U.S. civilian noninstitutionalized population Sub-sample of respondents to the National Health Interview Survey (NHIS) Sub-sample of respondents to the National Health Interview Survey (NHIS) – Linkage to NHIS Oversample of minorities and other target groups Oversample of minorities and other target groups Panel Survey – new panel introduced each year Panel Survey – new panel introduced each year – Continuous data collection over 2 ½ year period – 5 in-person interviews (CAPI) – Data from 1st year of new panel combined with data from 2nd year of previous panel

Advancing Excellence in Health Care MEPS Overlapping Panels (Panels 8 and 9) MEPS Household Component MEPS Panel Round 2Round 3Round 4Round 5 Round 1Round 2Round 3 MEPS Panel /1/20031/1/2004 Round 1 NHIS2002 NHIS2003 Round 4Round 5

Advancing Excellence in Health Care MEPS Household Component Sample Design Oversampling of policy relevant domains 1996 Minorities (Blacks & Hispanics) 1997 Minorities Low income Children with activity limitations Children with activity limitations Adults with functional limitations Adults with functional limitations Predicted high expenditure cases Predicted high expenditure cases Elderly Elderly Minorities 2002+Minorities, Asians, Low Income 15,000 households; ~35,000 persons

Advancing Excellence in Health Care Design Specifications Target Precision Specifications for national and regional estimates; policy relevant subgroups Target Precision Specifications for national and regional estimates; policy relevant subgroups Overall Design effect of 1.6 Overall Design effect of PSU design (Max) 200 PSU design (Max) Overall/round specific survey response rate requirements Overall/round specific survey response rate requirements Linkage to NHIS Linkage to NHIS Multistage design Multistage design Disproportional sampling Disproportional sampling Longitudinal design Longitudinal design Minimize survey cost for fixed precision Minimize survey cost for fixed precision

MEPS, 1996–2006: Number of uninsured, under age 65 Number in millions Source: Center for Financing, Access, and Cost Trends, AHRQ, Household Component of the Medical Expenditure Panel Survey, 1996–2005 Full-Year and 1996–2006 Point-in-Time Files

Advancing Excellence in Health Care Trends in Concentration Percentage of expenditures Source: National Medical Care Expenditure Survey, 1977; National Medical Expenditure Survey, 1987; Medical Expenditure Panel Survey, 1996 and 2005.

Advancing Excellence in Health Care MEPS Field Force Westat is data collection organization Westat is data collection organization 500 interviewers 500 interviewers Sample allocated in ~ 200 PSUs, spread across all 50 states Sample allocated in ~ 200 PSUs, spread across all 50 states extensive training modules extensive training modules Information on socio-demographic characteristics of households available based on linkage with NHIS Information on socio-demographic characteristics of households available based on linkage with NHIS Remuneration for survey participation Remuneration for survey participation

Advancing Excellence in Health Care Tool Chest of Methods to Maximize Survey Response Recruitment of experienced and bilingual interviewer Recruitment of experienced and bilingual interviewer 10+ days training (including procedures for obtaining signed consents) 10+ days training (including procedures for obtaining signed consents) Uses of MEPS data as reference materials for interviewers Uses of MEPS data as reference materials for interviewers Periodic retraining and special trainings (e.g. methods to improve response rates) Periodic retraining and special trainings (e.g. methods to improve response rates) Respondent remuneration Respondent remuneration Advance mailings from co-sponsors of survey Advance mailings from co-sponsors of survey Monthly planning calendar and MEPS DVD Monthly planning calendar and MEPS DVD Daily s to interviewers regarding interviewing progress Daily s to interviewers regarding interviewing progress Multiple contacts for refusal conversions Multiple contacts for refusal conversions

Advancing Excellence in Health Care MEPS Target Response Rates by Round and Overall Response Rate NHIS90%* Round 180% Round 295% Round 396% Year 1 of Panel66% Round 497% Round 598% Pooled Response Rate-Two Panels65% *NHIS response rate among households designated for MEPS. Note: Year 1 and the Overall response rate include the NHIS response rate.

Advancing Excellence in Health Care Person Level (survey attrition) Nonresponse Adjustment Covariates Factors associated with survey attrition (after R1) Factors associated with survey attrition (after R1) – Indicator for initial refusal to R1 interview – Family size – Age – MSA, census region – Marital status (family reference person) – Race/ethnicity – Education of reference person – Employment status – Health insurance status – Total expenditures (in yr 1 for yr 2 adj.) – # doctor visits (in yr 1) – Self reported health status

Advancing Excellence in Health Care Person Level Adjustments: Annual Estimates Each MEPS panel weighted separately Each MEPS panel weighted separately Nonresponse adjustment for complete nonresponse and for survey attrition Nonresponse adjustment for complete nonresponse and for survey attrition Final Poststratification adjustment –CPS 12/31: Final Poststratification adjustment –CPS 12/31: age, race/ethnicity, sex, region, MSA status, poverty status age, race/ethnicity, sex, region, MSA status, poverty status

Advancing Excellence in Health Care Testing for Panel Effect

Advancing Excellence in Health Care Capacity of MEPS to Produce Comparable NHIS Estimates of Health Insurance Coverage

Advancing Excellence in Health Care Characteristics of Respondents Fielded at End of First Round or Temporary Refusal Initial Refusals: Higher likelihood: MSA residence; Northeast region; white Non- Hispanic; elderly; excellent health; some high school; family size 2+; Attrite in future waves of data collection MSA residence; Northeast region; white Non- Hispanic; elderly; excellent health; some high school; family size 2+; Attrite in future waves of data collection End of Field Period Cases Higher likelihood: Race: Asian or Black Race: Asian or Black in excellent health in excellent health Attrite in future waves of data collection Attrite in future waves of data collection

Advancing Excellence in Health Care Testing for Reluctant Response Effect on Coverage Estimates DF Wald F P-Value DF Wald F P-Value OVERALL MODEL JULY INTERVIEW TEMP. REFUSALS SEX < RACE/ETHNICITY < MARITAL STATUS < EDUCATION < POVERTY STATUS < MSA STATUS INDIVIDUAL INCOME < MEDICAL $ < * Normalized Log-Likelihood Full Model: Pseudo R 2: :

Advancing Excellence in Health Care Mean Number of Contacts by Month

Advancing Excellence in Health Care Mean Number of Temporary Refusals by Month

Advancing Excellence in Health Care Conditional Response Rates by Month of Round 1 Response: Panel 9

Advancing Excellence in Health Care Panel/temp. refusal Interviewdate N persons N persons Mean R1 contacts Temp.refusals full year Cond.responseLongitudCond.response Overall P9 Overall 18,250 18, %85.40% P10 18,170 18, %. P9 < Jul 17,537 17, %85.90% P10 17,380 17, %. P9Jul %71.10% P %. 0 P9 < Jul 16,631 16, %86.80% 0 P10 16,377 16, %. 1+ P9 Jul %62.50% 1+ P %. ExclusionsJuly RR reduction June refusal July

Measure Standard Field Operation50% sample33% sample25% sampleExclusion Difficult Cases Mean Expenditures Standard Error Responding sample July cases+June temp refusals RSE Bias MSE RMSE Ratio of MSE to Standard Mean no. hours per complete Add’l sample to meet precision goal 670 1,284 2,132 Impact on MSE of Mean Medical Expenditure Estimates for alternative field strategies

Measure Standard Field Operation50% sample33% sample25% sampleExclusion Difficult Cases Proportion Standard Error Responding sample July cases+June temp refusals RSE Bias MSE RMSE Ratio of MSE to Standard Impact on MSE of Proportion with Medical Expenses in excess of $10,000 for alternative field strategies E

Advancing Excellence in Health Care Summary Need for accurate and reliable national data on health insurance coverage to inform policy and practice Need for accurate and reliable national data on health insurance coverage to inform policy and practice MEPS design features and analytical capacity MEPS design features and analytical capacity Statistical, methodological and operational design features to adjust for nonresponse and attrition Statistical, methodological and operational design features to adjust for nonresponse and attrition Evaluation of estimation strategies to correct for survey attrition Evaluation of estimation strategies to correct for survey attrition Examination of ROI for inclusion of difficult cases Examination of ROI for inclusion of difficult cases Options identified for more efficient and effective field strategies Options identified for more efficient and effective field strategies

Agency for Healthcare Research and Quality Advancing Excellence in Health Care