1. Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea Noack ISR, York University

2. Outline  general comments on response rates how response rates are calculated a very brief history of response rates what strategies have/are being put in place to deal with declining response rates

3. Outline (continued)  response rates and RRFSS what did it take to get the 62% rate for 2003 RRFSS number of calls refusal conversions results of the advance letter experiment

4. Calculating Response Rates  Completions / estimate of number of eligible households (HH) eligible HHs include completions, refusals, callbacks, and a % of the “never answered”  ISR method same as BRFSS, aka “CASRO 3”  RRFSS 2003 = 62%, exclude callbacks = 71%

5. Response Rates for American Election Study

6. Response Rates for BRFSS

7. Strategies for Improving Response Rates  interviewer training  increase call attempts  “convert” refusals  use advance letters  payments (as a lottery, to completers, to the whole sample)

8. Data Collection at ISR for RRFSS Response Rates  minimum number of 14 calls (more when there is reason to think extra calls might obtain a completion) limitation of one month sample release costs about 3 to 7 points on response rate  at least one attempt to convert almost all refusals

9. 390,106

10. RRFSS: Fun with Numbers 1 (2003 Data)  number of calls: 390,106  percent of interviews completed first call: 21  number of interviews completed on the 10 th or subsequent calls: 3,158  number of interviews completed after a refusal: 2,678

11. RRFSS: Fun with Numbers 2 (2003 Data)  average number of calls per completed interview: 4.65  most calls made for a single completion: 33 (for two (different) interviews)  response rate if 10 plus calls and refusal conversions are dropped: 48.2%  number of complaints about interviewer calling registered at ISR: 13

12. Characteristics of Refusers: 2003 RRFSS Data variablestandardconverted mean age47.9953.86 education: > than high school (%) university (%) 16.7 47.8 23.0 39.2 % employed61.752.4 % saying health fair or poor11.816.5 % doctor told high blood pressure22.327.1 % smoke 100 cigarettes52.555.0 # of cases: standard = 24,700, converted = 2,640 all differences significant

13. Characteristics of Easy and Hard to Reach: 2003 RRFSS Data variableeasy to reach hard to reach mean age50.4343.50 education: > than high school (%) university (%) 18.6 45.6 12.5 52.8 % employed54.873.6 % saying health fair or poor13.88.1 % doctor told high blood pressure25.016.8 % smoke 100 cigarettes52.750.6 # of cases: easy = 17,000, hard = 3,150 all differences significant

14. Letter Experiment: 1  six Health Units participated (Durham, London, Grey Bruce, Halton, Waterloo, Sudbury)  test two versions of letter: ISR and HU  needed to work with our monthly target and wanted to acknowledge random variation in response rates per HU per month  used sample “replicates” to implement experiment

15. Letter Experiment: 2  Month one: replicate 1, ISR letter; replicate 2, HU letter; replicate 3, and 4 (when used), control group changed presentation in months 2 and 3 copy of letter at the end of this set of handouts exactly the same text, different letterhead, signature & envelope Except Halton

16. Letter Experiment: 3  survey introduction exactly the same except one additional sentence “Recently, we sent a letter to your household about an important research project.”  questions about the letter the same except Durham

17. Why the Letter Might Improve Response Rates to RDD Surveys  reduces the possibility that the telephone call catches people by surprise  increases legitimacy of research project in the eye of the potential respondents  demonstrates social value  improves the confidence of the interviewer

18. Why Advance Letters Might Not Improve Response Rates to RDD Surveys  letter does not reach, or is not read by, respondent  ceiling effects  survey topic & subpopulations  they give “timid” participants a chance to prepare to say “no”

19. Response Rates for Months 1 & 2 of the Experiment p value =.035 (for letter (1,200) versus no letter (1,345)) p value =.025 (ISR (600) versus HU (600))

20. Response Rates Months 1 & 2: All Six Health Units See next slide for numbers & p values

21. RR by HU and Treatment treatment, RR (%) P value HU none ISR HU none/ ISR none/ HU ISR/ HU Durham58.963.665.3.422.275.803 London57.864.569.2.254.048.475 Grey70.573.278.7.618.128.367 Halton57.557.068.7.925.053.087 Waterloo64.760.265.8.441.851.416 Sudbury69.863.271.4.255.776.213 Number of cases per HU: ISR = 50, HU = 50, none = 100

22. Mean Calls per Completion mean # of calls P value None ISR HU none/ ISR none/ HU ISR/ HU Durham5.895.695.57.825.721.905 London6.416.064.43.751.049.060 Grey4.364.584.69.763.622.896 Halton5.186.426.53.157.124.935 Waterloo4.826.424.38.049.510.047 Sudbury5.955.145.42.445.597.809 Number of cases per HU: ISR = 50, HU = 50, none = 100

23. Mean Calls per Completion by Letter Status Letter/no letter p =.890, ISR/HU p =.230, not see/saw p =.001

24. % of First Call Attempts Leading to Completions & Refusals letter/no letter p =.204, ISR/HU p =.008

25. At the Start of the Interview

26. Awareness of Letter Variable (based on 602 cases) total R indicated saw letter at intro30 R indicated letter came to house21 total respondents aware of letter51 personally read the letter40 got more info (web site, 1-800)1 letter made a lot of difference to decision to participate 26

27. Data Characteristics VariableISR (n=300) HU (n=300) none (n=620) P year of birth19551954.820 male (%) 585657.800 employed (%) 59 60.627 health excellent22.951 smoked at least 100 cigarettes (%) 5056.197

28. Costs: Month One  cost of materials: $314; staff cost: $1,919  total: $2,233  per case cost: $3.62  buys 72 interviews or 12 per HU  need to estimate savings from making fewer calls, and making fewer refusal conversion calls

29. Conclusions  HU letter (seems to) increase response and warrants consideration as a tool to improve RRFSS response rates  affect on variable distributions minimal, but small sample size limits scope of examination  social-political distance between respondent and sender probably matters  letters may have value other than just increasing response rates

30. Questions