1. Survey Response Rates: Trends and Standards Karen Donelan, ScD Senior Scientist in Health Policy Massachusetts General Hospital/Harvard Medical School AcademyHealth - June 27 th, 2006

2. Overview What is a response rate? Why do we care about response rates? Trends Does nonresponse = biased response? How should rates be calculated and reported? How can we improve?

3. What is a Response Rate? The number of complete interviews with reporting units divided by the number of eligible reporting units in the sample

4. Other Rates Cooperation rates - The proportion of all cases interviewed of all eligible units ever contacted. Refusal rates - The proportion of all cases in which a sampled unit or the respondent refuses to be interviewed, or breaks-off an interview, of all potentially eligible cases. Contact rates - The proportion of all cases in which some responsible sampled unit member was reached.

5. Why Do We Care About Response Rates? Sources of Error in Surveys Coverage, Measurement, Nonresponse, Processing, Sampling Practical implications Generalizability Publishability (often cutoff or major test of quality) Credibility Fundability

6. What Do We Know About Response Rate Trends?

7. http://www.census.gov/prod/2000pubs/tp63.pdf Current Population Survey (CPS)

8. Behavioral Risk Factors Survey (BRFSS) : Telephone Survey Source: Groves et al. 2004

9. U Michigan: Survey of Consumers Source: Groves et al. 2004

10. Why Does Nonresponse Happen? Who is in the sample? (demographics, SES, lifestyle) # Contacts Schedule of contacts Mode of contact Respondent selection Respondent cooperation (refusal) Incentives ($, benefit) Respondent burden (time, boredom, frustration) Salience Respondent ability to respond to questions Sponsorship Privacy concerns

11. How Should Rates Be Calculated and Reported? Disclosure Units drawn and attempted Addresses mailed Phone numbers called Households approached Patients invited Sample Disposition: what happened? Eligible and ineligible respondents Bad contact information How many attempts, at what intervals # Never reached, # actually contacted # completed, refused, never got a decision

12. www.aapor.org

13. Does Nonresponse = Biased Response? Low response rates may not be problematic Representativeness doesnt necessarily increase with response rate Sample representativeness is a function of the difference between respondents and nonrespondents on the statistic(s) of interest High response rates can can yield an unrepresentative sample (high nonresponse bias) Low response rates can still yield a representative sample Keeter et al. (2000), Curtin, Presser and Singer (2000), Merkle and Edelman (2002)

14. Measuring Nonresponse Bias With response rates falling, understanding the impact on study findings is essential Assess quantitatively if possible, but at least consider qualitatively Methods for assessing nonresponse bias: Response rate comparisons across groups Follow-up interviews with non-respondents Comparing early vs. late respondents Comparisons to similar estimates from other sources Post survey adjustment for nonresponse

15. Ways Not to Approach Nonresponse Ignore it Obscure it Omit it Fail to measure/comment on potential associated bias Give up on it

16. Improving Response Rates Multiple contacts Multiple modes of contact Interviewer training Advance Notification/Endorsement Incentives Reduce respondent burden Increase relevance to respondent Time and money to get quality

17. Reporting Response Rates Know and apply standards (AAPOR, CASRO) Engage and manage with vendors Structure sample disposition Response and nonresponse analysis Publish enough information to allow others to do the calculations Peer review: build awareness of standards