1. Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis, James Wagner, William Axinn, University of Michigan, USA William Mosher, US National Center for Health Statistics Research partially supported by contract with the US National Center for Health Statistics, Contract No. 200-2000-07001

2. Definition: Responsive Design 1.Preidentify a set of alternative features potentially affecting costs and errors of statistics 2.Identify a set of indicators of the cost and error properties of those feature 3.Monitor indicators in initial stages of data collection 4.Alter the active features of the survey based on cost/error tradeoff decision rules 5.Combine data from separate phases into a single estimator Survey designs that:

3. The Why’s of Responsive Designs One-off surveys are mounted with large uncertainties (e.g., eligibility of frame elements, effort required to contact, cooperation rate, length of interview) Most survey budgets are relatively fixed at start of project Some survey errors are functions of effort during production Hence, quality is out of control of researcher unless designs are permitted to change based on production experience

4. Effort The NSFG Dashboard hours % production Active Sample nonworked interviews Productivity calls/day % peak calls scrn’r/main calls mean calls 8+ calls locked bldgs resistant hard appt. occupied eligible propensity calls/interview hours/interview % with kids response rate I’rs working calls/hour noncontacts cum. interviews Data Set Balance % sexually active group rates CV group rates X

5. Evaluation of Two Interventions Based on Paradata Survey setting: face-to-face survey screening (3- 5 min.) to locate 60% of households with 15-44 year old; one eligible sampled (60-80 min.) Interventions –Increasing relative effort on screening interviews versus main interviews with selected respondent –Increasing relative effort on a small subset of cases with high selection weights and high propensities to respond

6. Ratio of Screener Calls to Main Interview Calls by Day by Quarter

7. Autoregressive Time Series Coefficients for Model Predicting Daily Number of Screener Calls (p-values for coefficients) Q1Q2Q3Q4 Intercept279.9 (<.0001) 385.0 (<.0001) 345.3 (<.0001) 294.8 (<.0001) Day-0.73 (.48) -3.2 (0.28) -2.1 (.21) -2.0 (.17) Screener Week 28.3 (.29) 97.5 (.009) 24.6 (.56) 39.0 (.29)

8. Autoregressive Time Series Coefficients for Model Predicting Daily Number of Screener Interviews (p-values for coefficients) Q1Q2Q3Q4 Intercept63.6 (<.0001) 94.5 (<.0001) 64.6 (<.0001) 63.4 (<.0001) Day-0.51 (.032) -1.23 (0.001) -0.54 (.16) -0.69 (.09) Screener Week 6.1 (.30) 18.5 (.01) 12.3 (.20) 6.1 (.54)

9. Second Intervention: Increased Emphasis on Subset of Active Cases In the last weeks of Phase 1, a subsample of cases with high propensities and high selection weights are identified These cases are chosen to improve balance of respondent pool Interviewers are asked to give greater emphasis to these cases

10. Mean Expected Probability to be Interviewed on Next Call, Screeners (Red) and Main (Green) by Day of Data Collection by Quarter

11. Analytic Approach Not all interviewers active workloads contain both “intervention” cases and “non-intervention” cases We limit the analysis to those who have both types of cases We examine two indicators of success: –Mean number of calls (imperfect) –Response rate in intervention period

12. Comparison of Mean Screener Calls During Intervention Period for Intervention and Nonintervention Cases Little evidence of increased calling on intervention cases

13. Comparison of Screener Response Rate During Intervention Period for Intervention and Nonintervention Cases Little evidence of increased response rate on intervention cases

14. Comparison of Mean Main Calls During Intervention Period for Intervention and Nonintervention Cases Mixed evidence on higher calls on intervention cases

15. Comparison of Main Response Rate During Intervention Period for Intervention and Nonintervention Cases General tendency to higher response rates for intervention cases

16. Logistic Regression For Likelihood of Main Interview Quarter 2Quarter 3Quarter 4Quarter 5 Intercept-2.13*** -1.97***-0.81*** -0.84** Intervention-0.07 0.72*0.13 0.50* Weight0.00 Propensity5.46*** 2.98**2.02*** 0.65

17. Conclusions Intervention 1: Management direction to focus on screeners vs. main increases calls, sometimes dramatically; screener interviews follow Intervention 2: Effectiveness at focusing on individual cases greater for main interviews than screener interviews

18. Next Steps on Responsive Design with Paradata Responsive design requires effective central management direction of interviewer behavior We’re still learning how to communicate these directives consistently well