Interviewer Effects on Paradata Predictors of Nonresponse Rachael Walsh, US Census Bureau James Dahlhamer, NCHS European Survey Research Association, 2015 Disclaimer: The views expressed on statistical, methodological, technical, or operational issues are those of the author and not necessarily those of the U.S. Census Bureau or the National Center for Health Statistics.

Special thanks to National Center for Health Statistics, who sponsors the collection of data used in this presentation! 2

Paradata Predictors of Nonresponse  Contact History Instrument (CHI)  Launched manually or automatically after exiting a survey instrument  Available for major demographic surveys  Collects the type, timing, and outcome of every contact attempt  Correlated with response propensity not key survey estimates 3

4 Effect of Weighting Variables on Bias and Variance Little and Vartivarian (2005)

Interviewer Observations Objective Measures  Graffiti  Condition of sample unit  Access barrier  Well-tended yards  Damaged walls  Bars on windows  Multiple door locks  Children < 6 years of age  Disability  Adult bicycle  Smoking Subjective Measures  Income  Employed adult  Language other than English  Age composition of the household  All Surveys  Housing Surveys  Crime Surveys  Employment Surveys  Health Surveys 5

Key Health Survey Estimates  Health insurance coverage  Usual place to go for medical care  Failure to obtain needed medical care due to cost  Receipt of influenza vaccination  Receipt of pneumococcal vaccination  Obesity  Leisure-time physical activity  Current smoking  Alcohol consumption  HIV testing  General health status  Personal care needs  Serious psychological distress  Diagnosed diabetes  Asthma episodes in the past 12 months  Current asthma 6

Available Data  National Health Interview Survey (NHIS)  Cross-sectional, in-person, household survey  Oversamples Black, Hispanic, and Asian persons  Multi-stage sampling  Interviewer Observations Data  January through June 2014  First set of interviewer observations for each case  Over 56,000 interviewer observation records 7

Assessing Interviewer Effects  Variance: What are the interviewer effects on the relationship between the observations and final case disposition?  Bias: What is the percentage absolute relative nonresponse bias of each observational measure?  Objective measures versus subjective measures 8

Variance Methods  Multilevel Multinomial Logistic Regression  Cases nested within interviewers  Modeled final case disposition:  Noncontact vs. completed interview  Refusal vs. completed interview  Controlling for Urban/rural and Regional Office, and: Case Level CHI Indicators Number/type of contact attempts Interim refusals Doorstep concerns Interviewer Level Monthly caseload Supervisory status Tenure Interviewer observations experience 9

Variance Results  Overall interviewer effect (controlling for CHI)≈20%  Adding all interviewer observation indicators reduced the variance attributable to interviewers:  4% for refusal versus completed interviews  6% for noncontact versus completed interviews  Individual Observations: range [-3, 1] 10

Results for Objective Measures Interviewer ObservationChange in % Variance across Interviewers NoncontactRefusal Graffiti00 Condition of sample unit-21 Access barrier-31 Well-tended yards0 Damaged walls00 Bars on windows0 Multiple door locks00 Children < 6 years of age00 Disability00 Adult bicycle00 Smoking00 Source: National Health Interview Survey Paradata, January 2014 – June

Results for Subjective Measures Interviewer ObservationChange in % Variance across Interviewers NoncontactRefusal Low income0 High income00 Employed adult-21 Language other than English0 Young household (all < 30)00 Older household (all > 65)0 Source: National Health Interview Survey Paradata, January 2014 – June

Bias Methods 13

Bias Results  Range:  Objective: range = [0, 18]  Subjective: range = [3, 16]  Mean:  Objective: 8.5  Subjective:

Results for Objective Measures Interviewer Observation% Relative Bias Graffiti1 Condition of sample unit0 Access barrier9 Well-tended yards4 Damaged walls3 Bars on windows7 Multiple door locks2 Children < 6 years of age18 Disability16 Adult bicycle17 Smoking16 Source: National Health Interview Survey Paradata, January 2014 – June

Results for Subjective Meausres Interviewer Observation% Relative Bias Low income3 High income7 Employed adult3 Language other than English16 Young household (all < 30)8 Older household (all > 65)9 Source: National Health Interview Survey Paradata, January 2014 – June

Low BiasHigh Bias Low Var. Graffiti Well-tended yards Damaged walls Bars on windows Multiple door locks Income Age composition Condition of SU (noncontact) Access Barriers (noncontact) Employed adult (noncontact) Children < 6 years of age Disability Adult bicycle Smoking Language other than English Some Var. Condition of SU (refusal) Access barriers (refusal) Employed adult (refusal) NONE! 17 Summary

Future Research  Measurement error models  Age composition  Children < 6 years of age  Employed adult  Comparison of traditional NR adjustment versus observation-based adjustments  Point estimates  Variance estimates  Model observational measures as dependent variables and explain the variation with environmental attributes 18

Thank you! 19