1. Center for Survey Research University of Virginia Center for Survey Research University of Virginia From Dual-Frame to Triple Frame: An Assessment of Coverage Bias in a Telephone Survey Design Combining RDD, Directory-Listed And Cell Phone Samples An Assessment of Coverage Bias in a Telephone Survey Design Combining RDD, Directory-Listed And Cell Phone Samples Presented at AAPOR 2011 Phoenix, AZ May 15, 2011

2. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 2 Authors Thomas M.Guterbock University of Virginia TomG@virginia.edu Abdoulaye Diop Qatar University ADiop@qu.edu.qa James M. Ellis University of Virginia jme2ce@virginia.edu John Lee Holmes University of Virginia jlh2r@virginia.edu Trung Kien Le Qatar University kienle@qu.edu.qa

3. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 3 Overview Why triple-frame? The 2010 Behavioral Study of the NCR Coverage and distribution of phone service (5 segments) Contrast of RDD, EWP and Cell Phone frames Calling efficiencies Comparing substantive results: –triple frame vs. RDD+Cell Cost comparisons Conclusion

4. Center for Survey Research University of Virginia Center for Survey Research University of Virginia Why triple-frame? From one frame, to two, to three

5. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 5 From RDD to Dual-frame “Traditional” list-assisted landline Random Digit Dialing is beset with problems –Increasing under-coverage due to Cell Phone Only (CPO) households –Lower working-number rates –Declining response rates Dual-frame telephone surveys are now standard for many survey organizations –Combining landline RDD and cell phone RDD frames –Most often using an ‘overlap’ or ‘all cell’ design –See AAPOR Task Force 2010 for full discussion Adding cell phones covers the CPO’s and favorably alters reachability of dual-phone households

6. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 6 Consider: Electronic White Pages (directory listed) sample Previous studies have compared EWP to landline RDD sample in statewide, regional, local studies. –Guterbock et al. 2003, Oldendick et al. 2004. Guterbock, Diop & Holian 2007 explored race and other predictors of listedness in a survey of the National Capital Region Caution: These studies pre-date the surge in CPO households

7. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 7 EWP vs. RDD Substantive results are similar, in general Advantages of EWP: –More efficient –Lower cost –Greater geographic specificity Disadvantages: –Undercoverage of minorities, lower income, renters –African Americans less likely to have listed numbers –Blacks therefore under-represented in EWP samples

8. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 8 Who needs RDD? There are notable similarities between characteristics of unlisted landline households and CPO households –Minorities, the young, renters, lower incomes Could the gaps in the EWP frame (undercoverage) be filled in by inclusion of the cell phone frame? We proposed in 2008: EWP+Cell as a dual-frame design alternative to RDD+Cell –Based on analysis of NHIS data through 2006 –Just published in Social Science Research We presented a favorable comparison of the two designs in three county-based surveys in Virginia (2009)

9. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 9 From Dual Frame to Triple Frame CSR has completed ten telephone surveys that use a triple- frame design –4 metro-area based; 6 county-based –All cell phones included—no screening for CPO –All in Virginia or DC metro area Two reasons for triple-frame design: –To allow comparison of RDD+Cell vs. EWP vs. Cell –Most of these studies needed comparison to earlier years that used landline RDD exclusively We are at a transitional stage in telephone sampling –Triple-frame designs are a compromise between ‘standard’ dual- frame design and our proposed alternative of EWP+Cell

10. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 10 errorcost It ’ s all about...

11. Center for Survey Research University of Virginia Center for Survey Research University of Virginia The 2009 Behavioral Survey of the National Capital Region

12. 12 2009 Survey of Behavioral Aspects of Sheltering and Evacuation in the National Capital Region Sponsor: VDEM Funding: U.S. DHS

13. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 13 Survey Goal Collect information from residents of the National Capital Region that would predict behavior in the area in the event of an emergency. Included factorial experiment that varied features of a “dirty bomb” attack by terrorists. –Ask how residents would respond to specific “shelter-in-place” scenarios –What variables have the most effect on behavior? –What patterns of evacuation and shadow evacuation should be expected? –Where would the evacuees try to go? The resulting data are being used to inform the decisions made by administrators in the region and beyond. –Details in CSR’s report, 2010.

14. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 14 Features of the Survey In-depth survey: average interview length 28 minutes –Fully supported Spanish language interviews as needed 2,609 interviews conducted by CSR, Sept-Dec 2009. Triple-frame sample design: –1269 Landline RDD completes –898 EWP (directory listed) completes –442 cell phone completes (no screening for CPO’s) RDD sample was backmatched to addresses –Advance postcard sent to EWP and backmatched RDD cases Weighting by ownership, race, gender, geography, and type of telephone service Margin of error: +/- 2.3 percentage points –After weighting

15. Center for Survey Research University of Virginia Center for Survey Research University of Virginia Coverage and distribution of phone service

16. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 16 What percentage of landline phones are unlisted? Each respondent was asked whether their landline is listed in the directory. –Dual users reached by cell phone were asked to report on whether their landline is listed Percent unlisted can be taken directly from the RDD frame. 19.9% of landline RDD completes are unlisted. –As in our other studies, a small portion of those in the EWP frame report their phones to be unlisted.

17. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 17 What is the CPO percentage? NHIS has been used as the ‘gold standard’ for weighting by phone service. NHIS did not provide estimates for this geography. We used the locally based method of estimation described by Guterbock 2009. 31.4% of cell phone respondents were CPO’s. Final estimate: 15.4% of telephone HH are CPO. –After excluding landlines with unknown listed status

18. 2009 NCR Telephone Universe 2 CELL + ULL 18.1% 4 CELL + LLL 61.0% 5 LLL ONLY 4.2% 3 ULL ONLY 1.3% 1 CELL ONLY 15.4%

19. What EWP+Cell would cover 2 CELL + ULL 18.1% 4 CELL + LLL 61.0% 5 LLL ONLY 4.2% 3 ULL ONLY 1.3% 1 CELL ONLY 15.4%

20. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 20 Little undercoverage Households with no cell phone and an unlisted landline phone (ULL only), are but 1.3% of the region’s telephone households. These are covered by the landline RDD frame, but not covered by the EWP frame. EWP frame underestimates unlisted percentage. We weight the unlisted percentage among all landlines to 19.9% (the unlisted percent in RDD).

21. Center for Survey Research University of Virginia Center for Survey Research University of Virginia Contrasting the three frames And the matched vs. unmatched portions of landline RDD frame

22. Comparing calling efficiency Landline RDD EWPCell RDD MatchedUn- matched combined Completes per hour (CPH).69.39.58.84.54 Hours per complete (HPC) 1.452.561.711.191.85 n of completes9683011269898442 Calling hours 140377221751069819

23. Comparing key demographics Landline RDD EWPCell RDD ACS 2008 MatchedUn- matched combined % renter 18.725.320.314.938.434.3 % African- American 16.527.419.113.823.927.5 % never married 19.322.720.116.538.9-- % age 18-25 3.35.43.81.622.011.0 n of cases9683011269898442

24. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 24 Comparing substantive results How are survey results affected when a triple- frame sample is used? We compare our triple-frame result with the results we would have obtained with an RDD+Cell design. Both designs are post-weighted to the same control weights: –Ownership and race (joint distribution), gender, 8 counties, type of telephone service (CPO, LLO, dual user reached by landline, dual user reached by cell phone) and listed status

25. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 25 Comparing substantive results Triple frame design RDD+CellDifference Would leave immediately in maximum hazard 19.4%19.8%-0.4% Has an emergency kit prepared 32.6% 0.0% Strongly agree it is very important to live in this particular area 32.5%31.8%0.7%

26. Center for Survey Research University of Virginia Center for Survey Research University of Virginia Let’s compare 20 variables Would leave scene in a dirty bomb attack –Minimum hazard level –Moderate –Maximum Can trust most people Trust local government Has an emergency plan Has an emergency kit Has a meeting place Worry about attack –3 levels of hazard High perceived risk –Property damage; injury –3 levels of hazard Agree/Strongly agree: –Feel at home where I live –I have a lot in common with neighbors –Important for me to live in this area

27. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 27 Differences for 20 variables Margin of error +/- 2.3%

28. Center for Survey Research University of Virginia Center for Survey Research University of Virginia Cost comparison

29. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 29 Cost factors and assumptions Assume that a survey of like size had been carried out with a ‘traditional’ dual frame design –Same number of cell phone completions as in our triple-frame design –442 cell phone completes –2167 landline RDD completes RDD uses more sampled numbers RDD cost per sampled number is higher –Due to extra charge for backmatching Assume postcard sent to backmatched RDD cases Cell phone completes get $10 incentive

30. Triple frame costs Land-line RDD EWPCell phone RDD Total N of completes 1,2698984422,609 CPH.58.84.54.62 Calling hours 2,1751,0698194,062 Cost of calling hours $69,590$34,210$26,193$129,992 Sampled numbers 14,0836,1279,58529,795 Cost of sampled numbers $2,110$674$1,150$3,934 Postcards 7,6316,127--13,758 Cost of postcards $3,052$2,451--$5,503 Incentive Cost -- $4,420

31. RDD+Cell costs (projected) (Note: Assumes 54% Landline RDD back-matched to directory listing and mailed a postcard) Land-line RDD Cell phone RDD Total N of completes 2,1674422,609 CPH.58.54.58 Calling hours 3,7148194,532 Cost of calling hours $118,835$26,193$145,028 Sampled numbers 24,0499,58533,634 Cost of sampled numbers $3,603$1,150$4,753 Postcards 13,031--13,031 Cost of postcards $5,212--$5,212 Incentive Cost --$4,420

32. Triple frame vs.RDD+Cell costs Each design: 2,609 completes Triple Frame RDD+ Cell Diff CPH.62.58 Calling hours 4,0624,532 Cost of calling hours $129,992$145,028$15,036 Sampled numbers 29,79533,634 Cost of sampled numbers $3,934$4,753$819 Postcards 13,75813,031 Cost of postcards $5,503$5,212- $291 Incentive Cost $4,420 0 Total cost$143,849$159,413 $15,564

33. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 33 Cost summary RDD+Cell would have cost: $159,413 ($61.10 per completion) Triple frame design cost: $143,849 ($55.14 per completion) Substitution of EWP for some of the landline RDD frame saved $15,564 ($5.97 per completion) or 10.8% of the Triple Frame total. –Greater savings could have been realized if EWP percentage were larger relative to landline RDD

34. Center for Survey Research University of Virginia Center for Survey Research University of Virginia Conclusion

35. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 35 errorcost It ’ s all about...

36. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 36 Conclusion For representative general population results, we need to include cell phones in our telephone surveys When combined with the cell phone sample frame, EWP sample frames offer –greater efficiency than landline RDD –lower cost There are good reasons to retain some landline RDD sample in the mix at this transitional stage –Allows direct measurement of unlisted percentage There is little or no loss of accuracy when EWP is substituted for some of the landline RDD frame –and cell phones are included in the design Cost savings are considerable (over 10% in this study)

37. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 37 Conclusion Three frames are better than two! RDD EWPCell

38. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 38 References AAPOR Cell Phone Task Force. 2010. New Considerations for Survey Researchers When Planning and Conducting RDD Telephone Surveys in the U.S. With Respondents Reached via Cell Phone Numbers. Available online at www.aapor.org.www.aapor.org Guterbock TM, 2009. “Estimating Local Phone Service and Usage Percentages.”: How to Weight the Data from a Local, Dual Frame Sample Survey Of Cell Phone and Landline Telephone Users in the United States.” AAPOR paper. Guterbock TM, Diop A, Ellis JM, Le TK, & Holmes JLP, 2009. “Who Needs RDD–Part II: An Assessment of Coverage Bias in Dual-Frame Designs That Combine Directory-Listed And Cell Phone Samples.” AAPOR poster, Hollywood FL. Guterbock TM, Diop A, Ellis JM, Le TK, & Holmes JLP, 2011. "Who Needs RDD? Combining Directory Listings with Cell Phone Exchanges for an Alternative Telephone Sampling Frame". Social Science Research 40:3 (May): 860-872. Also presented as 2008 AAPOR paper, New Orleans. Guterbock TM, Diop A, & Holian L, 2007. “White pages, white people: Reasons for the low listed- phone rates of African-Americans.” AAPOR paper, Anaheim. Guterbock TM, Hartman DE & Hubbard RA, 2003. “RDD vs listed: An experimental study of coverage error, costs and non-response in a statewide telephone survey. AAPOR paper, Nashville. Guterbock TM, Lambert JH, Bebel RA, Ellis JM, & Kermer DA, 2010. Population Behaviors in Dirty Bomb Attack Scenarios: A Survey of the National Capital Region. Prepared for the Virginia Department of Emergency Management. University of Virginia Center for Survey Research, April. Oldendick, Robert W., et al. 2004. “Differences in an RDD and list sample: An experimental comparison.” AAPOR paper, Phoenix.

39. Center for Survey Research University of Virginia Center for Survey Research University of Virginia From Dual-Frame to Triple Frame: An Assessment of Coverage Bias in a Telephone Survey Design Combining RDD, Directory-Listed And Cell Phone Samples An Assessment of Coverage Bias in a Telephone Survey Design Combining RDD, Directory-Listed And Cell Phone Samples Presented at AAPOR 2011 Phoenix, AZ May 15, 2011 Contact: TomG@virginia.edu