1 Responsive Design and Survey Management in the National Survey of Family Growth (NSFG) William D. Mosher, NCHS FCSM Statistical Policy Seminar Washington, DC, December 4, 2012

2 Outline Goals & Features of the NSFG Why the NSFG Uses Responsive Design How the NSFG Uses Responsive Design Some Statistical Policy Issues 2

3 Premises behind responsive design PROBLEM: Response rates are falling, and survey costs rising, in the US and other developed countries. Government surveys are known for outstanding quality, but if we don’t change the way we work our samples, our response rates will drop and the quality of our data will deteriorate. QUESTION: But how do we know what to change and whether it worked? ANSWER: Measure the effects by collecting paradata. 3

4 Main Features of the National Survey of Family Growth National multi-stage area probability sample Topics: Pregnancy, contraception, infertility, marriage, parenting Interview men & women age All interviews in person. No phone, no internet. 14,000 addresses selected & screened each year to yield at least 5,000 interviews per year. Continuous design, using four 12-week “quarters” per year. Contractor: University of Michigan ISR. – Project Director: Mick Couper – Field Director: Nicole Kirgis 4

5 5 Each 12-week quarter is divided into 2 parts Two phases of data collection in each 12-week quarter: – Phase 1: 10 week data collection with a $40 incentive. – Phase 2: Subsample cases remaining after 10 weeks  Select a sample of about 1/3 of the remaining cases;  That reduces each interviewer’s case load by 2/3  Change data collection model:  Double the incentive,  Triple the number of hours the interviewer spends on each case. – Combine data and response rates from two phases using weights

6 Goals of Responsive Design in the NSFG The major expense in an in-person area probability sample is interviewer time, including labor and local travel costs. So the NSFG’s initial goals in using responsive design were to use paradata to ration and allocate the interviewers’ time to manage costs and response rates. The NSFG is trying to balance the following goals: – Control costs & prevent cost increases; – Keep response rates at acceptable levels (70- 80% overall & for sub-groups) – Get at least 5,000 interviews per year. 6

7 NSFG Intervention: Screener vs Main 7 Extra effort directed to screening

8 NSFG Intervention: Subgroup Response Rates Response rates for Hispanic adult males lag before intervention; higher after it. 8

9 NSFG Intervention: Response Rates for sub-groups varied more in Quarter 1 than in Quarter 16 (Male & Female, & 20-44; Hispanic & Black & other race) 9

10 Summary: Responsive Design in the NSFG-1 In the NSFG, responsive design is used to (a) limit costs, (b) get adequate sample sizes, and (c) equalize response rates across subgroups by age-race- sex that are strongly correlated with NSFG outcomes. Techniques included flagging groups with lagging response rates for extra visits, encouraging adequate screening effort, and designing the non-response follow-up. We are also beginning to use responsive design to monitor data quality, but this is in its beginning stages. 10

11 Some policy issues-1 First: The purpose of responsive design: In the NSFG, we want to represent all groups (by age-sex-race) equally well, so that – response rates are equal across groups, & – key statistics are as unbiased as possible. Responsive design allows us to track response rates in 12 sub- groups, and intervene to try to keep them equal. We also want to use our resources as efficiently as possible and get the best possible return for it. This design cut our cost per case by over 30%, so we believe we accomplished that. 11

12 Some policy issues-2 Second: The design depends on the use of incentives. 92% of respondents who complete the interview receive $40. About 8% of respondents (in Phase 2) receive $80. Is this an Ethical issue? Or: to reduce non-response bias, you have to change the protocol to get different respondents. A different incentive can be part of that strategy, but Groves et al. suggest it should not be all of it. – NOTE: NSFG doubles the incentive and triples the hours on each case. – RESULT: This results in more childless women in the sample, a key group for a fertility survey. 12

13 Some issues-3 Third: We have used the same incentives for 10 years, since But inflation has reduced the value of those dollars by 20% since Periodically, these incentives have to be adjusted for inflation. (Once a decade may be a reasonable interval.) Fourth: There’s a risk in offering incentives to people whom you have not been able to contact at all, because they do not understand what you are doing. (e.g., Dept of the Interior.) We limit this risk by doing this for a small group of screener respondents. 13

14 References and further information 1.NSFG web site: 2. ISR/U of Michigan methodology research on NSFG: 3.Groves et al, “Planning & Development of the Continuous National Survey of Family Growth.” NCHS, Vital & Health Statistics, Series 1, No. 48, Sept 2009, pages Lepkowski et al, The National Survey of Family Growth: Sample Design & Analysis of a Continuous Survey. Vital & Health Statistics, Series 2, No. 150, June 2010, pages