1. Response rates and disposition coding PHC 6716 June 8, 2011 Chris McCarty

2. RDD Telephone Disposition Codes I = Complete interview (1.1) P = Partial interview (1.2) R = Refusal and break-off (2.10) NC = Non-contact (2.20) O = Other (2.30) UH = Unknown if household/occupied HU (3.10) UO = Unknown, other (3.20) e = Estimated proportion of cases of unknown eligibility that are eligible

4. Types of rates Response Rate (RR) = Proportion of cases interviewed of all eligible cases Cooperation Rate (COOP)= Proportion of cases interviewed of eligible cases contacted Refusal Rate (REF) = Proportion of eligible sample that refused Contact Rate (CON) = Proportion of eligible sample where a household was reached

6. What gets reported? It is rare to see anything other than the response rate reported Occasionally people will report the cooperation rate This is often reported when the response rate is low

7. AAPOR Response Rates

8. I = Complete interview (1.1) P = Partial interview (1.2) R = Refusal and break-off (2.10) NC = Non-contact (2.20) O = Other (2.30) UH = Unknown if household/occupied HU (3.10) UO = Unknown, other (3.20) e = Estimated proportion of cases of unknown eligibility that are eligible

9. Calculating e e is an estimate of the proportion of non-contacts that are eligible Its calculation depends on survey design and execution There are a number of ways it can be calculated Some people just make assumptions about e http://www.aapor.org/pdfs/erate.pdf

10. Example of calculating e for May 2011 CCI

11. Comparison of response rates for May 2011 CCI RR111.1 RR211.1 RR312.9 RR412.9 RR516.2 RR616.2

12. Exceptions to AAPOR Disposition Codes In some cases AAPOR codes may not be detailed or descriptive enough to allow for exceptional circumstances with sample If possible AAPOR codes should be used If new codes must be used they should aggregate to existing AAPOR codes

13. Last versus Final Dispositions Last Disposition is actually the “most recent disposition” Final disposition requires a rule for evaluating call history There are no firm standards for evaluating call history Research suggests that the difference is trivial

14. Last versus Final Dispositions -- Research RDD 1RDD 2RDD 3Listed 1Listed 2Listed 3 Sample released350029174445030084162681100 Call limit1512 10 Average attempts per case5.65.14.15.14.12.2 Completed interviews5018010100276303823402 Response rate using most recent disposition.28.41.32.43.49.40 Response rate using ever eligible.21.33.28.31.39 Response rate using all five final disposition rules.26.40.32.38.44.40 Response rate using only Rule 1.23.34.29.32.39 Response rate using only Rule 2.22.36.28.35.38.39 Response rate using only Rule 3.22.34.28.32.33.39 Response rate using only Rule 4.22.33.28.31.39 Response rate using only Rule 5.22.33.28.32.39 Response rate using only Rule 1 and Rule 2.24.37.30.36.38.39 For most recent call: Rule 1: Business always coded as business Rule 2: No eligible respondent always coded as no eligible respondent Rule 3: Disconnected number previously coded as no answer or temporary phone problem always coded as disconnected number Rule 4: Fax/data line previously coded as no answer or temporary phone problem always coded as Fax/data line Rule 5: Non-working number previously coded as a temporary phone problem always coded as a non-working number

15. Council of American Survey Research Organizations (CASRO) Disposition Codes and Response Rates

16. Example response rates BRFSS Response rates by state (page 32): ftp://ftp.cdc.gov/pub/Data/Brfss/2010_Summ ary_Data_Quality_Report.pdf ftp://ftp.cdc.gov/pub/Data/Brfss/2010_Summ ary_Data_Quality_Report.pdf Quinnipiac polls: – http://www.quinnipiac.edu/x271.xml http://www.quinnipiac.edu/x271.xml

17. What affects Response Rates? Maximum call attempts Maximum number of refusal callbacks Fielding time Rotation of calls across day of the week and time of day The population surveyed Survey length Sample quality Listed versus random-digit dial Salience of survey topic to respondent

18. Variables in model used to predict response rate across 205 surveys Overall (R 2 =0.52) Listed Sample (R 2 =0.42)RDD Sample (R 2 =0.67) VariableCoeff.Prob>tCoeff.Prob>tCoeff.Prob>t Constant0.3270.0000.4250.0000.0960.024 Salience level 2−0.7830.030−0.1770.0000.0190.549 Salience level 30.0660.007−0.0020.9460.0820.018 Number of calls per case0.0010.7760.00080.8700.0030.643 Minutes per piece of sample0.0220.0010.0170.0410.0340.007 Refusal calls0.0050.5920.00020.9890.0130.447 Survey length−0.0070.000−0.0060.000-0.0050.059 Fielding time7.0780.0005.1980.00717.2580.005 Fielding time squared−23.9650.003−16.2270.067−147.5110.042 Listed versus RDD−0.1180.019na

22. What would it take to get a 70% response rate?

23. Ways people overstate response rates Pre-screening samples incorrectly (e.g. removing persistently unavailable numbers) Unreasonable calculations of e Purchasing RDD sample from higher density banks

24. Response rates with lists When sampling from a list most of these rules must be modified The fundamental question is “Who is eligible?” If being on the list makes them eligible then they are

25. Analyze the effects of non-response If you have information about all potential respondents, compare respondents to non- respondents Compare converted refusals to non-refusals Compare early responders to late responders