1. How Cell Phone Interviewing Costs are Changing—and Why
Guterbock, et al.: Who Needs RDD?Who Needs RDD?
Who Needs RDD?
DRAFT SLIDES 4/22/08
5/16/08--AAPOR New OrleansDRAFT SLIDES 4/22/08
How Cell Phone Interviewing Costs are Changing—and Why
Presented at
AAPOR 2016
Austin, TX
May 13, 2016
[Updated 5/23/2016]
UVa Center for Survey ResearchUVa Center for Survey Research
UVa Center for Survey Research 1

2. Thomas M. Guterbock David Dutwin Grant Benson Jenny Kelly
Center for Survey Research
University of Virginia
Grant Benson
Survey Research Center
University of Michigan
Paul Lavrakas
Independent Consultant
David Dutwin
SSRS
Jenny Kelly
NORC
. . . A unit of the Weldon Cooper Center for Public Service

3. Thanks to:
Marketing Systems Group and ASDE Survey Sampler, for assisting us with recruitment of respondents
Special thanks to Trent Buskirk (M-S-G) and Michel Durocher (ASDE)
University of Michigan SRC, for hosting the survey
Grant Benson, for managing survey fielding, data analysis assistance

4. Overview Study purposes: Have cost ratios declined?
What factors affect production ratios in dual-frame RDD surveys?
Trends in cell phone survey costs
2015 Cell Phone Task Force survey
Comparison to similar surveys from 2010, 2013
Comparison of cost ratios across years
Factors that affect production rates
Components of the production rate
Call duration, effort & yield
Summary of results, estimated cost examples
Whither cell phone survey costs?

5. Study purposes
Build upon, extend, and compare to 2010 Cell Phone Task Force survey and 2013 Cost Survey
Sample SROs more broadly
Obtain greater detail on production
Examine distinct components of production
Gain better understanding of factors that drive production ratios & their components
Our main question: Are cost ratios of cell vs. landline calling trending downward further?

6. Cost per interview [CPI]: three components
Guterbock, et al.: Who Needs RDD?Who Needs RDD?
5/16/08--AAPOR New OrleansDRAFT SLIDES 4/22/08
Cost per interview [CPI]: three components
CPI is sum of three types of cost:
Interviewer hours
Including interview time and non-interview time
Cost of incentives (plus any mailing costs)
Cost of purchased sample phone numbers and pre-screening
Our focus: ratio of interviewer hours per complete
Ratio of cell phone hours to landline hours
Interviewer hours captures most of the data collection labor cost
Incentive amount differences are a design decision rather than determined by the type of sample
Cost of samples is relatively low and similar
Andy: Same slide, just some changes
UVa Center for Survey ResearchUVa Center for Survey Research 6

7. A closer look at interviewer hours
Interviewer Hours per Completion = HPC = hours/completes
HPC = 1/CPH
Screening and Recruiting Hours per Completion = SRHPC
SRHPC = HPC ─ interview length

8. Four factors affect production rates
Working number rate
Contact rate
Eligibility rate
Cooperation rate
Of interest: differentials in these rates (CP/LL) on the same study
In 2010, first three were all lower for cell phones!
Result: lower yields, more hours of calling for cell phones
As noted: the production rate ratio drives the cost ratio
These rates are relevant in theory, but difficult to measure in practice

9. Why might the cost ratio be decreasing?
Working number rate
Shift in working-number densities as households migrate to cell and CPO
Dutwin shows strong trends in opposite directions for LL(↓) and CP (↑)
Working number rate now higher for CP
Contact rate
More likely to answer if CPO, and CPO share is increasing
But Dutwin shows increase in answering machine/no answer rates
Eligibility rate
(Not clear how this might be changing)
Cooperation rate
Dutwin’s data show little change in rates of refusal on either LL or CP
In addition, SROs may have learned and trained on ways to achieve greater cell phone success

10. A look back at 2010
Cell phone cost survey was part of 2010 AAPOR Cell Phone Task Force Report
Cost section by Guterbock, Lavrakas, Tompson and zuWallack
Data collection: Winter
Convenience sample of 8 SRO’s
4 academic, 4 private sector
26 usable surveys reported

11. Productivity Statistics and Cost Ratios for Dual Frame RDD Surveys
A look back at 2010
Productivity Statistics and Cost Ratios for Dual Frame RDD Surveys
Screening and Recruiting Hours per Completion
Hours per Completion
Overall Cost per Interview
Ratio (cell/landline)
SRHPC Ratio
HPC Ratio
Cost Ratio
Mean
2.53
2.00
2.05
Minimum
1.21
1.17
1.35
Maximum
5.37
3.47
3.97 N 26 20
Std. deviation
1.02
.63
.77
Source: 2010 survey for AAPOR Task Force

12. Guterbock, et al.: Who Needs RDD?Who Needs RDD?
5/16/08--AAPOR New OrleansDRAFT SLIDES 4/22/08
The 2013 Cost Survey
A dual-frame survey, too!
Member organizations of AASRO
All AAPOR cell phone session presenters,
112 individuals in frame
98 organizations
Web survey in Qualtrics, March-April 2013
allowed reporting on up to 3 recent surveys
Responses from 27 organizations (RR3 = 42%)
Usable data on 37 dual-frame surveys
UVa Center for Survey ResearchUVa Center for Survey Research

13. Guterbock, et al.: Who Needs RDD?Who Needs RDD?
5/16/08--AAPOR New OrleansDRAFT SLIDES 4/22/08
The 2015 TF Cost Survey
Broader sampling plan
Member organizations of AASRO
All AAPOR cell phone session presenters,
Outreach to cell phone clients by sample vendors
M-S-G and ADPE Survey Sampler
Web survey in Qualtrics, Sept – Oct 2015
allowed reporting on up to 3 recent surveys
72 individuals responded, some had no data
22 firms gave us data on 53 studies
3 surveys: data on 114 studies in all,
Need to compute correct n of organizations solicited and finalize dispo report using AAPOR adjustment of rate. Check dates of AAPOR presenters used by Debbie.
UVa Center for Survey ResearchUVa Center for Survey Research

14. Thanks to 2015 participating SROs:
Abt SRBI, Inc.
Castleton College Polling Institute
Elon University Poll
IPSOS Public Affairs
Marist College Poll
NORC at the University of Chicago
Penn State Harrisburg Center for Survey Research
Princeton Survey Research Associates
Public Opinion Strategies
RTI
Rutgers University Eagleton Center for Public Interest Polling
SSI
SSRS
South Dakota Department of Health
University of Nebraska-Lincoln Bureau of Sociological Research
University of Northern Iowa Center for Social & Behavioral Research
University of Oklahoma Public Opinion Learning Laboratory
University of Virginia Center for Survey Research
University of Waterloo Survey Research Centre
University of Wisconsin Survey Center
U.S.Food and Drug Administration
plus Four anonymous contributors
Commercial firms in blue

15. We asked about . . . Main survey topic Sample geography
Interview length (for CP and LL)
Screened or “overlap” design?
Eligibility rate among the screened landline sample cases
Use of advance mailings and gift card incentives
How LL numbers were dialed
Use of “enhanced” cell phone sample
Production details . . .

16. Two formats for reporting production data:
Detailed format (38 surveys):
Average interview length (in minutes)
Method of telephone dialing (manual, one-at-a-time autodial, or predictive dialer?)
Number of phone numbers attempted
Total dialings for the numbers attempted
Number of completes
Total interviewing hours
Completes per hour (CPH)
[divide n of completes by interviewing hours]
Landline RDD
Cell Phone
RDD*
XXX 
Alternative format (17 surveys):
If you prefer, instead of reporting the numbers in the shaded boxes, you can just report the CPH ratio, as calculated below:
CPH Ratio = (Cellphone CPH) / (Landline CPH) = ______________

17. Cell Phone Sample Share is Increasing

18. Cash incentives for cell phone: now rare

19. Guterbock, et al.: Who Needs RDD?Who Needs RDD?
5/16/08--AAPOR New OrleansDRAFT SLIDES 4/22/08
Hours per Completion, 2015
Mean HPC
ratio (CP/LL)
= 1.39
Landline
median
(reported)
Cell
(projected)
Ratio reported for 53 surveys; hours reported for 35 surveys providing detail
UVa Center for Survey ResearchUVa Center for Survey Research

20. Screening and Recruitment HPC, 2015
Guterbock, et al.: Who Needs RDD?Who Needs RDD?
5/16/08--AAPOR New OrleansDRAFT SLIDES 4/22/08
Screening and Recruitment HPC, 2015
Mean SRHPC
ratio (CP/LL)
= 1.53
Landline
median
(reported)
Cell
(projected)
Ratio reported for 53 surveys; hours reported for 35 surveys providing detail
UVa Center for Survey ResearchUVa Center for Survey Research

21. Productivity Statistics for Dual Frame RDD Surveys
2015 compared to prior years
Productivity Statistics for Dual Frame RDD Surveys
Screening and Recruiting Hours per Completion
Hours per Completion
Ratio (cell/landline)
SRHPC Ratio
HPC Ratio
2010
2013
2015
Mean
2.5
1.7**
1.5
2.0
1.5**
1.4
Minimum
1.2
.77
.43
.85
.45
Maximum
5.4
3.7
3.8
3.5
2.9
3.3 N 27 37 50 38 53
Std. deviation
1.0
.71
.80
.63
.51
.64
Outliers (hpcratio > 4.0) excluded
**2013 & 2015 significantly different from 2010.
2015 not significantly different from 2013.

22. HPC Ratios Compared

23. SRHPC Ratios Compared

24. Factors that might affect the ratio
Predictive dialing of landlines (25 surveys*)
Increase SRHPC ratio by making landline calling faster
Geography (26 national surveys)
Expect lower ratio with national studies
BRFSS studies (10 surveys)
Used modified cell-only design; required 15 attempts for landline (but not for cell phone); used out-of-state cases
Enhanced cell samples showing activity (31 surveys)
Fewer bad cell phone numbers, lower SHRPC ratio
Monetary or gift card incentives for cell phone (17)
Lower SHRPC ratio by increasing cell phone cooperation
[*Based on 91 surveys reported in 2014 & 2015]

25. Log of SRHPC is well distributed
SRHPC ratio
Log of SRHPC ratio
[*Based on 114 usable surveys reported in 2010, 2014 & 2015]

26. Factors affecting SRHPC
Log: SRHPC
Factor:
Cell Phone
Land Line
Ratio (CP/LL)
LL Predictive Dialer
-.259
-.546**
.407**
National sample
.275
.309*
-.089
BRFSS study
.172
.209*
-.076
Enhanced cell sample
-.172
.217+
-.463**
CP Gift card
-.069
-.118
.074 R2
.070
.434
.595
SRHPC = Screening and recruitment
hours per completion
OLS Standardized Coefficients
N = 60

27. Predictive dialing speeds LL calling: higher ratios

28. Enhanced cell samples more efficient: lower ratios

29. 3 components of production
Average Call Duration = Screening and Recruitment Hours [SRH] / number of dialings
Average time spent on a non-interview call attempt
Effort = Dialings / number of phone numbers attempted
Yield = Completes / number of phone numbers attempted
These components are directly measurable!
We can also calculate a CP/LL ratio for each component.

30. Duration of non-interview calling, 2015
Mean Duration ratio (CP/LL)
= 1.57
Landline
median
(reported)
Cell
(projected)
n = 29 surveys

31. Effort (dialings per number), 2015
Mean Effort
ratio (CP/LL)
= 1.03
Landline
median
(reported)
Cell
(projected)
n = 29 surveys

32. Yield (completes per number), 2015
Median Yield
ratio (CP/LL)
= 1.09
Completing
about 1 in 14
Landline
median
(reported)
Cell
(projected)
About 1 in 20
About 1 in 15
n = 32 surveys

33. Factors affecting effort
Log: Effort
Factor:
Cell Phone
Land Line
Ratio (CP/LL)
LL Predictive Dialer
-.590**
-.038
-.462**
National sample
.141
-.451**
.632**
BRFSS study
.030
.205
-.206+
Enhanced cell sample
.318*
.261*
-.024
CP Gift card
.314
.060
.201 R2
.374
.466
.354
Effort = Dialings/attempted number
OLS Standardized Coefficients
N = 54

34. Factors affecting yield
Log: Yield
Factor:
Cell Phone
Land Line
Ratio (CP/LL)
LL Predictive Dialer
-.423**
-.551**
.318*
National sample
-.156
-.292*
.247
BRFSS study
.174
.208*
-.103
Enhanced cell sample
.430**
-.004
.497**
CP Gift card
.367**
.083
.299 R2
.596
.672
.441
Yield = Completes/number attempted
OLS Standardized Coefficients
N = 54

35. 3 components  SRHPC
Product of Duration times Effort divided by Yield equals SRHPC:
(Duration)
(Effort)
(1/Yield)
SRH
Dialings
Numbers
––––––––––
––––––––––
––––––––––
Dialings
Numbers
Completes

36. 3 components  SRHPC
Product of Dialtime times Effort divided by Yield equals SRHPC:
(Duration)
(Effort)
(1/Yield)
(SRHPC)
SRH
Dialings
Numbers
SRH
––––––––––
––––––––––
––––––––––
––––––––––
Dialings
Numbers
Completes
Completes

37. 3 components  SRHPC
Product of Dialtime times Effort divided by Yield equals SRHPC:
(Duration)
(Effort)
(1/Yield)
(SRHPC)
SRH
Dialings
Numbers
SRH
––––––––––
––––––––––
––––––––––
––––––––––
Dialings
Numbers
Completes
Completes
Therefore:
Log Duration +
Log Effort ─
Log Yield =
Log SRHPC

38. 3 logged components account for all the variance in logged SRHPC ratio
Duration
Ratio
1.400**
Log
SRHPC
ratio
Log
HPC
ratio
Log
Effort
Ratio
1.081**
0.994
Log
Yield
Ratio
-1.314**
R2 = 0.998
R2 = 0.989
Updated for 2015

39. Main effects of Predictive Dialing
Predictive Dialing of Landlines
Log
Duration
Ratio
1.027**
1.4**
-0.462**
Log
SRHPC
ratio
Log
HPC
ratio
Log
Effort
Ratio
0.318*
1.1**
0.99
R2 = 0.998
R2 = 0.989
-1.3**
Log
Yield
Ratio
Updated for 2015

40. Main effect of National Geography
Log
Duration
Ratio
1.4**
National Geography
.632**
Log
SRHPC
ratio
Log
HPC
ratio
Log
Effort
Ratio
1.1**
0.99
R2 = 0.998
R2 = 0.989
-1.3**
Log
Yield
Ratio
Updated for 2015

41. Main Effect of BRFSS Study Design
Log
Duration
Ratio
1.4**
Log
SRHPC
ratio
Log
HPC
ratio
Log
Effort
Ratio
BRFSS Study
1.1**
0.99
R2 = 0.998
R2 = 0.989
-1.3**
Log
Yield
Ratio
Updated for 2015

42. Main Effect of Enhanced Cell Sample
Log
Duration
Ratio
1.4**
Log
SRHPC
ratio
Log
HPC
ratio
Log
Effort
Ratio
1.1**
0.99
Enhanced Cell Phone Sample
R2 = 0.998
R2 = 0.989
0.497**
-1.3**
Log
Yield
Ratio

43. Main Effect of Gift Cards
Log
Duration
Ratio
1.4**
Log
SRHPC
ratio
Log
HPC
ratio
Log
Effort
Ratio
1.1**
0.99
R2 = 0.998
R2 = 0.989
-1.3**
Log
Yield
Ratio
Gift Card for Cell
0.299*
Updated for 2015

44. Putting it all together . . .
Predictive Dialing of Landlines
Log
Duration
Ratio
1.027**
National Geography
1.400**
-0.462*
R2 = 0.62
0.632**
Log
SRHPC
ratio
Log
HPC
ratio
Log
Effort
Ratio
0.318+
BRFSS
Study
1.081**
0.994
R2 = 0.35
Enhanced Cell Phone Sample
R2 = 0.998
R2 = 0.988
-1.314**
0.497**
Log
Yield
Ratio
0.299*
Gift Card for Cell
R2 = 0.44
Updated for 2015

45. 5 factors predict 60% of variance:
Predictive Dialing of Landlines
0.407**
National Geography
-0.089
Log SRHPC ratio
-0.076
BRFSS
study
-0.463**
R2 = 0.60
Enhanced Phone Sample
0.074
(Really just two factors!)
Gift Card
for Cell Phone

46. Guterbock, et al.: Who Needs RDD?Who Needs RDD?
5/16/08--AAPOR New OrleansDRAFT SLIDES 4/22/08
So, what’s the cost?
n of surveys
Numbers attempted
Cost per number
Sample cost per complete
HPC
HPC Cost ($30/hr)
Total Cost per Interview
All CP 56 18
0.09
$1.62
1.51
$45.30
$46.92
All LL 14
0.07
$0.98
1.16
$34.80
$35.78
Conventional CP sample 38 24
$1.68
1.54
$46.20
$47.88
Enhanced CP sample 13
0.11
$1.54
1.44
$43.20
$44.74
Predictive dial LL 23 31
$2.17
0.94
$28.20
$30.37
Regular dial LL 45
$0.91
1.46
$43.80
$44.71
Assumed: no cash incentives
UVa Center for Survey ResearchUVa Center for Survey Research

47. Summary of findings
Mean HPC and SRHPC ratios are down significantly from the 2010 results
Ratio of Hours Per Completion has decreased from 2.0 to 1.5 to 1.4
Therefore, cost ratios are also down
Variance in production ratios has lessened significantly
Survey shops have learned how to routinize dual-frame phone work

48. Summary of findings (cont.)
Variation in cost ratios is driven primarily by:
Predictive dialing of landlines
increases efficiency on landline calling
raises the cell/landline ratio for duration, SRHPC
under threat from TCPA regulatory climate
Enhanced cell phone sample
increases efficiency of cell phone calling
raises the cell/landline ratio for yield, SRHPC
Not uncommon in 2015 for studies using enhanced sample to show SRHPC < 1.0
That is: Cell frame more productive than landline frame

49. Whither Cell Phone interviewing costs?
Enhanced cell samples are changing the cost picture significantly
Production rates for enhanced cell are often equal to or better than landline RDD production rates
These cell-frame refining methods are being widely adopted
As cost ratio decreases, allocation of sample to the cell phone frame can be further increased decreasing design effects in dual-frame studies
Trend to larger cell allocations will continue
A cell-only (single) frame design is increasingly possible, as cell phone production rates improve
See Peytchev et al (2013)

50. Innovation and research continuing . . .
Future experiments on cell phone calling productivity should give attention to the three separate components of productivity
Call duration, effort, and yield
New variations on screening designs are sure to emerge
Note that BRFSS screens for CPO or cell-mostly (90% or more cell calls)
Dual-frame surveys are becoming routine; we are getting more efficient at doing them; CP and LL costs are equalizing
But: the pace of change in telephony and innovation in survey methods is not slowing down!

51. References
AAPOR New Considerations for Survey Researchers When Planning and Conducting RDD Telephone Surveys in the U.S. with Respondents Reached via Cell Phone Numbers. AAPOR Cell Phone Task Force Report. Paul J. Lavrakas (task force chair), and 20 others. American Association for Public Opinion Research, June. Available at: Guterbock, Thomas M., Paul J. Lavrakas, Trevor N. Tompson, and Randal ZuWallack, “The Variable Costs of Cell Phone Interviewing: Understanding Cost and Productivity Ratios in Dual-Frame Telephone Surveys.” AAPOR paper, Chicago. Guterbock, Thomas M., Paul J. Lavrakas, Trevor N. Tompson, and Randal ZuWallac, “Cost and Productivity Ratios in Dual-Frame RDD Telephone Surveys.” Survey Practice, April: Guterbock, Thomas M., Robin A. Bebel, John Lee P. Holmes, and Peter A. Furia, “Why We No Longer Need Cell Phone Incentives.” AAPOR paper, Orlando, May. Guterbock, Thomas M., Andy Peytchev, and Deborah L. Rexrode “Cell Phone Costs Revisited: Understanding Cost and Productivity Ratios in Dual-Frame Telephone Surveys.” AAPOR paper, Boston, May. Peytchev, Andy and Neely, Benjamin, “RDD Telephone Surveys: Toward a Single Frame Cell Phone Design.” Public Opinion Quarterly 77 (Spring): 51

52. How Cell Phone Interviewing Costs are Changing—and Why
Guterbock, et al.: Who Needs RDD?Who Needs RDD?
Who Needs RDD?
5/16/08--AAPOR New OrleansDRAFT SLIDES 4/22/08
DRAFT SLIDES 4/22/08
How Cell Phone Interviewing Costs are Changing—and Why
Thomas M. Guterbock
Grant Benson
Paul Lavrakas
David Dutwin
Jenny Kelly
Presented at
AAPOR 2016
Austin, TX
May 13, 2016
[Updated 5/23/2016]
UVa Center for Survey ResearchUVa Center for Survey Research
UVa Center for Survey Research 52