1 Dependent Interviewing: Evidence from Field Tests The SIPP Methods Panel Nancy Bates and Joanne Pascale U.S. Census Bureau Seminar on Dependent Interviewing Univ. of Essex, September 17, 2004

2 Acknowledgements Pat Doyle, Jeff Moore, Joanne Pascale, Julia Klein Griffiths, Anna Chan Heather Holbert, Elaine Hock, Johanna Rupp, Aniekan Okon, Ceci Villa, Judy Eargle U.S. Census Bureau Field Division

3 Methods Panel Research Project 4-yr research project to study and refine recommendations –Extant data and research, cognitive assessments, large-scale field experiments, interviewer assessments 3 Field Experiments –Treatment group—experimental instrument –Control group—SIPP production instrument –1000 households per treatment –Conducted in 6 Regional Offices –CAPI interview (CASES language)

4 Methods Panel Research Project 3 Field Experiments –Experiment in 2000 — Wave 1 –Experiments in 2001 & 2002 — Waves 1 and 2 –Interviewer assessments, data analysis –Response Rates from 83% - 92% Produced SIPP 2004 instrument

5 Cognitive Assessments Wave 1: 34 households interviewed Wave 2: 4 months later, 23 households re- interviewed Cognitive interview followed by debriefing Reactions to dependent interviewing techniques: –Sharing of data with other members –Reaction to Respondent Identification Policy (RIP)

6 Cognitive Assessments RIP Question: “We re-contact households every 4 months to update information. If we talk to someone else in your household next time, instead of you, is it OK if we use your answers as a starting point?” D.I. did not elicit privacy concerns Respondents expected previous information to be stored Few respondents declined RIP request (misunderstood?)

7 Interviewer Debriefings Pencil and paper debriefing after each wave (Wave 1 n=152 ; Wave 2 n=131) Section in Wave 2 about D.I. Techniques: –Use of “Same as last time” reminders –Dependent questions vs. “from scratch” –Dependent question nonresponse follow-ups –Correcting mis-reported breaks in receipt Rated on 5 point scale (Strongly disagree – Strongly agree)

8 Interviewer Debriefings D.I. preferred over “from scratch” (assets) D.I. perceived as easier for respondent; smoother interview; easier to administer; more accurate (earnings, unearned income, assets, & health insurance) D.I. not perceived more accurate in correcting seam bias (not sig. different from control). D.I. perceived more accurate regarding health insurance coverage

9 Attrition Response Rates (Unit-level attrition) “Learning Curve” effect (year) D.I. effect (treatment)

10

11 Attrition On surface, little evidence D.I. impacted attrition No significant difference in test/control comparisons Closer look at Wave 1 across years and treatment groups Logistic regression –Significant (p<.10) main effect for year (negative) –Borderline (p=.11) year*treatment interaction –Nonresponse decline across years is sig. for test group but not control –Wave 2 exhibits similar trend, but interaction not sig.

12 Dependent Interview Nonresponse Follow-up Techniques If “Don’t know” “Refused” or “Same as last time”…. Earnings: Things may have changed since then, but I have recorded from last time that [NAME] earned about $X,XXX a month from this job with [EMPLOYER NAME]. Does that still sound about right? Unearned Income: It says here that [NAME] received [$XXX] in food stamp benefits last [MONTH]. Does that still sound about right? Asset Income: Things may have changed a lot but I have recorded from last time your income from these rental properties was about $[X,XXX] annually. Does that still sound about right?

13 Assessing Dependent Interview Nonresponse Follow-up Techniques Item nonresponse for income amounts (don’t know, refused) Computed aggregate nonresponse ratio: # D.K. + Refused # questions asked Average of ratios across all adults in universe

14

15

16

17 SIPP interview months and their associated reference period months (example): OCTNOVDECJANFEBMARAPRMAYJUNJULAUG M1M2M3M4M5  Wave 1 Interview M1M2M3M4M5  Wave 2 M1M2[etc.]

18 First InterviewSecond Interview

19 Dependent Interview Seam-Bias Reduction Techniques Unearned Income (and School Enrollment): –If reported in M5 previous: “Last time I recorded that you received [food stamps] in [month]. Is that correct?” –If reported in M4 but not M5: “Last time I recorded that you received [food stamps] in [month 4]. Did you continue to receive any benefits from [food stamps] after [month 5] 1 st ? Health Insurance: –“Last time I recorded [NAME] was covered by Medicaid. Is [NAME] still covered by that type of insurance plan?”

20 Assessing Dependent Interview Seam-Bias Reduction Techniques Month-to-month transitions in recipiency/enrollment/coverage (yes/no) Transitions at the “seams” (between M4-M5) where fewer transitions = improvement Calculated % of month-to-month transitions for: unearned income, school enrollment, and health insurance coverage

21 Expected % of transitions if no seam bias

22 Expected % of transitions if no seam bias

23

24 Expected % of transitions if no seam bias

25

26 Conclusions D.I. In 2004 SIPP represents major departure from past practice Thorough testing program suggest: –Substantially improved interview experience from Interviewer perspective –Respondents had few concerns with D.I. (qualitative tests) –R.I.P didn’t have big impact on D.I. (3-5%)

27 Conclusions D.I. beginning to impact attrition? D.I. Nonresponse follow-up techniques successful Decreased item nonresponse for earning, asset, and unearned income amounts D.I. techniques helped to reduce seam bias in income recipiency & enrollment D.I. helped reduce “on/off” seam transitions? Still much room for improvement SIPP D.I. techniques first fielded in June 2004