Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea Noack ISR, York University

Outline  general comments on response rates how response rates are calculated a very brief history of response rates what strategies have/are being put in place to deal with declining response rates

Outline (continued)  response rates and RRFSS what did it take to get the 62% rate for 2003 RRFSS number of calls refusal conversions results of the advance letter experiment

Calculating Response Rates  Completions / estimate of number of eligible households (HH) eligible HHs include completions, refusals, callbacks, and a % of the “never answered”  ISR method same as BRFSS, aka “CASRO 3”  RRFSS 2003 = 62%, exclude callbacks = 71%

Response Rates for American Election Study

Response Rates for BRFSS

Strategies for Improving Response Rates  interviewer training  increase call attempts  “convert” refusals  use advance letters  payments (as a lottery, to completers, to the whole sample)

Data Collection at ISR for RRFSS Response Rates  minimum number of 14 calls (more when there is reason to think extra calls might obtain a completion) limitation of one month sample release costs about 3 to 7 points on response rate  at least one attempt to convert almost all refusals

390,106

RRFSS: Fun with Numbers 1 (2003 Data)  number of calls: 390,106  percent of interviews completed first call: 21  number of interviews completed on the 10 th or subsequent calls: 3,158  number of interviews completed after a refusal: 2,678

RRFSS: Fun with Numbers 2 (2003 Data)  average number of calls per completed interview: 4.65  most calls made for a single completion: 33 (for two (different) interviews)  response rate if 10 plus calls and refusal conversions are dropped: 48.2%  number of complaints about interviewer calling registered at ISR: 13

Characteristics of Refusers: 2003 RRFSS Data variablestandardconverted mean age education: > than high school (%) university (%) % employed % saying health fair or poor % doctor told high blood pressure % smoke 100 cigarettes # of cases: standard = 24,700, converted = 2,640 all differences significant

Characteristics of Easy and Hard to Reach: 2003 RRFSS Data variableeasy to reach hard to reach mean age education: > than high school (%) university (%) % employed % saying health fair or poor % doctor told high blood pressure % smoke 100 cigarettes # of cases: easy = 17,000, hard = 3,150 all differences significant

Letter Experiment: 1  six Health Units participated (Durham, London, Grey Bruce, Halton, Waterloo, Sudbury)  test two versions of letter: ISR and HU  needed to work with our monthly target and wanted to acknowledge random variation in response rates per HU per month  used sample “replicates” to implement experiment

Letter Experiment: 2  Month one: replicate 1, ISR letter; replicate 2, HU letter; replicate 3, and 4 (when used), control group changed presentation in months 2 and 3 copy of letter at the end of this set of handouts exactly the same text, different letterhead, signature & envelope Except Halton

Letter Experiment: 3  survey introduction exactly the same except one additional sentence “Recently, we sent a letter to your household about an important research project.”  questions about the letter the same except Durham

Why the Letter Might Improve Response Rates to RDD Surveys  reduces the possibility that the telephone call catches people by surprise  increases legitimacy of research project in the eye of the potential respondents  demonstrates social value  improves the confidence of the interviewer

Why Advance Letters Might Not Improve Response Rates to RDD Surveys  letter does not reach, or is not read by, respondent  ceiling effects  survey topic & subpopulations  they give “timid” participants a chance to prepare to say “no”

Response Rates for Months 1 & 2 of the Experiment p value =.035 (for letter (1,200) versus no letter (1,345)) p value =.025 (ISR (600) versus HU (600))

Response Rates Months 1 & 2: All Six Health Units See next slide for numbers & p values

RR by HU and Treatment treatment, RR (%) P value HU none ISR HU none/ ISR none/ HU ISR/ HU Durham London Grey Halton Waterloo Sudbury Number of cases per HU: ISR = 50, HU = 50, none = 100

Mean Calls per Completion mean # of calls P value None ISR HU none/ ISR none/ HU ISR/ HU Durham London Grey Halton Waterloo Sudbury Number of cases per HU: ISR = 50, HU = 50, none = 100

Mean Calls per Completion by Letter Status Letter/no letter p =.890, ISR/HU p =.230, not see/saw p =.001

% of First Call Attempts Leading to Completions & Refusals letter/no letter p =.204, ISR/HU p =.008

At the Start of the Interview

Awareness of Letter Variable (based on 602 cases) total R indicated saw letter at intro30 R indicated letter came to house21 total respondents aware of letter51 personally read the letter40 got more info (web site, 1-800)1 letter made a lot of difference to decision to participate 26

Data Characteristics VariableISR (n=300) HU (n=300) none (n=620) P year of birth male (%) employed (%) health excellent smoked at least 100 cigarettes (%)

Costs: Month One  cost of materials: $314; staff cost: $1,919  total: $2,233  per case cost: $3.62  buys 72 interviews or 12 per HU  need to estimate savings from making fewer calls, and making fewer refusal conversion calls

Conclusions  HU letter (seems to) increase response and warrants consideration as a tool to improve RRFSS response rates  affect on variable distributions minimal, but small sample size limits scope of examination  social-political distance between respondent and sender probably matters  letters may have value other than just increasing response rates

Questions