1. Estimating Phone Service and Usage Percentages: How to Weight the Data from a Local, Dual-Frame Sample Survey of Cellphone and Landline Telephone Users in the United States Presented at AAPOR 2009 Hollywood, FL May 14, 2009 Thomas M. Guterbock TomG@virignia.edu

2. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 2 The Problem Dual-frame telephone surveys are becoming more prevalent in U.S. survey research –The rising percentages and distinctive demographics of cellphone-only [CPO] households make it imperative that sample designs cover them. –Landline RDD + Cellphone RDD sample frames Result: sample data for 3 phone-service segments – CPO; overlap (dual-phone); landline-only [LLO] Problem: what is the correct population distribution across 3 phone service segments?

3. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 3 National data? No problem National Health Interview Survey [NHIS] data are the ‘gold standard’ –Uses a very large N, continuous sampling, in-person mode to establish household phone service. –NHIS provides fairly current data on cellphone coverage, percent CPO, phone segment distributions NHIS data are available for the U.S. & for four census regions –State estimates released in 2009 using CPS + NHIS SOLUTION: Weight phone-service segments in the national sample to NHIS percents for U.S.

4. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 4 What about local studies? We cannot assume that the local phone-service segment distribution is the same as national or regional averages. Cellphone penetration and CPO lifestyle adoption vary considerably across areas. Cell penetration is higher in high density areas, metro areas, high-income areas, flat terrain, near interstates CPO percentage varies with age, ethnicity, urbanicity, landline phone costs NHIS: strong phone service variation across regions, states –Variation within states is probably similar in magnitude

5. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 5 Why not use percents from the local sample data? In a local dual-frame sample, we will directly observe % CPO in the cell sample, % LLO in the landline sample. But estimation from these observed percents is problematic for several reasons: 1)If we just combine the two samples, we overlook the fact that overlap households are double-sampled. 2)It’s not intuitively obvious how to calculate the percentages for the combined sample from the split sample results.

6. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 6 Why not use percents from the local sample data? 3)Cellphone-only cases are substantially overcounted in a cellphone sample. CPOs have different telephone behaviors. More likely than dual-phone users... To have phone with them To have phone turned on To accept calls from unknown numbers 4)Cellphone samples are usually kept small because of higher per-completion cost So we can’t just add up the segment counts from the two samples.

7. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 7 Can we use the local sample data? Collected data from the two realized, local samples surely contain useful information about local phone-service segments Overcounts of CPO and LLO distort these data We have to do the math correctly IDEA: Estimate the amount of CPO and LLO overcount in national dual-frame studies, and then apply an adjustment to the local sample data to arrive at local estimates for %CPO and %LLO

8. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 8 Overview: A proposed solution Develop algebraic solution for combining the two sample results from a dual-frame design into an overall phone service segment distribution, assuming equal response rates. Develop algebraic solution for combining the two samples when response rates are NOT equal –higher response rates (overcounts) are assumed for CPO and LLO (compared to overlap) Compare 2007 CHIS to 2007 NHIS (West region) to estimate ‘response rate ratios’ that correspond to the observed overcount Apply these ratios to newly collected dual-frame survey data from three counties in Virginia –Result: plausible, locality-specific estimates of phone segments

9. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 9 Key assumptions Local phone-service segment distributions vary –Forcing NHIS segment distributions onto local data would distort results Response rate ratios (rates of overcount) are constant across surveys –If fielding and screening procedures are similar Sampling variability is ignorable –In comparison of NHIS to CHIS –In projection from the local samples to local population

10. Center for Survey Research University of Virginia Center for Survey Research University of Virginia How to combine dual-frame sample results (equal response rates)

11. The universe of telephone households 100%

12. Cell phone samples include some that are also in the RDD frame Cell phones (Frame 1) Landline- only households are excluded 81.1%

13. RDD samples cover all landline households RDD (Frame 2) Cell-phone- only households are excluded 86.8%

14. RDD and Cell samples overlap, yield complete coverage Cell phones RDD CPO CELL ONLY 13.2% P aT =.132 OVERLAP CELL + LANDLINE 67.9% P abT =.679 LLO LANDLINE ONLY 18.9% P bT =.189 All percentages are from 2007 NHIS data (West region). a ab b These proportions define the population distribution of segments:

15. With equal response rates, cell sample would show: Cell phones RDD CPO P aT =.132 OVERLAP P abT =.679 LLO LANDLINE ONLY P bT =.189 All percentages are from 2007 NHIS data (West region). a 81.1% CPO as percent of Frame 1 P a ′ =.132/.811 =.163 OVERLAP as percent of Frame 1 P ab ′ =.679/.811 =.837

16. With equal response rates, RDD sample would show: Cell phones RDD CPO P aT =.132 OVERLAP P abT =.679 LLO P bT =.189 All percentages are from 2007 NHIS data (West region). a ab b OVERLAP as percent of Frame 2 P ab″ =.679/.868 =.783 86.8% LLO as percent Of Frame 2 P b″ =.189/.868 =.218

17. So, if response rates were equal, we would have... True values NHIS West 2007 Observed thru Cell sample Observed thru RDD sample CPO P aT 13.2%Pa′Pa′ 16.3% Overlap P abT 67.9%P ab ′ 83.7%P ab″ 78.3% LLO P bT 18.9%Pb″Pb″ 21.7% Total100.0%

18. How do we get from observed percentages to population percents? True values NHIS West 2007 Observed thru Cell sample Observed thru RDD sample CPO P aT ??Pa′Pa′ 16.3% Overlap P abT ??P ab ′ 83.7%P ab″ 78.3% LLO P bT ??Pb″Pb″ 21.7% Total100.0%

19. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 19 Formulas for calculating underlying population distribution With P abT + P aT evaluated, we have:.

20. Center for Survey Research University of Virginia Center for Survey Research University of Virginia Combining dual-frame sample results when response rates are not equal

21. Three segments, four response rates Cell phones RDD Cell sample response rate for CPOs: r a a ab b Cell sample response rate for overlap: r ab ′ RDD sample response rate for LLOs: r b RDD sample response rate for overlap: r ab″

22. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 22 4 response rates, 2 response rate ratios Reduction in base response for dual-phone in the cell sample is: –This is the ‘response rate ratio’ that applies to the cellphone sample. Reduction in base response for dual-phone in the RDD sample is: –This is the response rate ratio for the RDD sample.

23. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 23 It follows that... And our expressions for calculating true population phone service segments are modified by incorporating the response rate ratios:

24. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 24 How to calculate response rate ratios Now assume that we have observed results from a dual-frame phone survey. We also know the true population distribution. We can calculate the response rate ratios:

25. Center for Survey Research University of Virginia Center for Survey Research University of Virginia Deriving response rate ratios by comparing CHIS 2007 to NHIS

26. CHIS 2007 California Health Interview Survey True values NHIS West 2007 Observed thru Cell sample Observed thru RDD sample CPO P aT 13.2%Pa′Pa′ 34.6% Overlap P abT 67.9%P ab ′ 65.4%P ab″ 68.3% LLO P bT 18.9%Pb″Pb″ 32.7% Total100.0% ≠16.3% ≠21.7%

27. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 27 From these data we can evaluate r 1 and r 2 In the cellphone sample, overlap response rate is only 37% of CPO rate. In the RDD sample, overlap response rate is about 60% of LLO rate. Overcount of CPOs is greater than overcount of LLOs. This shows: many dual-phone users still use cellphone as a secondary device.

28. Center for Survey Research University of Virginia Center for Survey Research University of Virginia Calculating local area estimates of population phone-service segment distributions

29. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 29 2008 Prince William County Survey Citizen satisfaction survey in large, suburban county in Northern Virginia N = 1,666 Triple frame design: cellphone, landline RDD, and directory-listed sample –Here we combine the landline samples and treat as a dual-frame design Screening questions patterned after those on CHIS

30. 2008 Results for Prince William County, VA Observed thru Cell sample Observed thru RDD sample CPO P aT Pa′Pa′ 40.6%0.7% Overlap P abT P ab ′ 59.4%P ab″ 88.5% LLO P bT Pb″Pb″ 10.5% Total100.0%

31. 2008 Results for Prince William County, VA True values for PWC Observed thru Cell sample Observed thru RDD sample CPO P aT ??Pa′Pa′ 40.6%0.7% Overlap P abT ??P ab ′ 59.4%P ab″ 88.5% LLO P bT ??Pb″Pb″ 10.5% Total100.0%

32. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 32 Apply formulas given above: Calculations based on: r 1 =.368 r 2 =.598

33. 2008 Results for Prince William County, VA True values for PWC Observed thru Cell sample Observed thru RDD sample CPO P aT 19.0%Pa′Pa′ 40.6%0.7% Overlap P abT 75.3%P ab ′ 59.4%P ab″ 88.5% LLO P bT 5.7%Pb″Pb″ 10.5% Total100.0%

34. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 34 2008 Albemarle County Survey Citizen satisfaction survey Suburban and rural county surrounding City of Charlottesville, VA Similar triple-frame design as in PWC survey Smaller sample size: n = 700

35. 2008 Results for Albemarle County, VA Observed thru Cell sample Observed thru RDD sample CPO P aT Pa′Pa′ 21.9%0.2% Overlap P abT P ab ′ 78.1%P ab″ 82.7% LLO P bT Pb″Pb″ 17.2% Total100.0%

36. 2008 Results for Albemarle County, VA True values for Albemarle Observed thru Cell sample Observed thru RDD sample CPO P aT 8.4%Pa′Pa′ 21.9%0.2% Overlap P abT 81.4%P ab ′ 78.1%P ab″ 82.7% LLO P bT 10.2%Pb″Pb″ 17.2% Total100.0%

37. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 37 2008 Chesterfield County Survey Citizen satisfaction survey Suburban county adjacent to Richmond, VA Similar triple-frame design as in PWC survey –Treated as dual frame here n = 1600

38. 2008 Results for Chesterfield County, VA Observed thru Cell sample Observed thru RDD sample CPO P aT Pa′Pa′ 20.4%0.1% Overlap P abT P ab ′ 79.6%P ab″ 87.6% LLO P bT Pb″Pb″ 12.4% Total100.0%

39. 2008 Results for Chesterfield County, VA True values for Chesterfield Observed thru Cell sample Observed thru RDD sample CPO P aT 8.0%Pa′Pa′ 20.4%0.1% Overlap P abT 84.8%P ab ′ 79.6%P ab″ 87.6% LLO P bT 7.2%Pb″Pb″ 12.4% Total100.0%

40. Contrasting results NHIS CHIS [= NHIS] Prince William Albe- marle Chester- field CPO P aT 13.2% 19.0%8.4%8.0% Overlap P abT 67.9% 75.3%81.4%84.8% LLO P bT 18.9% 5.7%10.2%7.2% Total100.0%

41. Center for Survey Research University of Virginia Center for Survey Research University of Virginia Using the estimated segment distribution to weight the sample data

42. Example: PWC 2008 Observed thru cell sample Observed thru RDD sample Combined sample unweighted CPO7640.6%110.7%875.3% Overlap11159.4%130388.5%141485.4% LLO15410.5%1549.3% Total187100.0%1468100.0%1655100.0%

43. 3-segment weights: PWC 2008 Combined sample unweighted True values for PWC WeightWeighted N CPO875.3%19.0%3.6131419.0% Overlap141485.4%75.3%.88124775.3% LLO1549.3%5.7%.61945.7% Total1655100.0% 1655100.0%

44. But wait... We have 4 segments Observed thru cell sample Observed thru RDD sample Combined sample unweighted CPO7640.6%110.7%875.3% Overlap via cell 11159.4%1116.7% Overlap via RDD 130388.5%130378.7 LLO15410.5%1549.3% Total187100.0%1468100.0%1655100.0%

45. If 2 frames split the overlap equally: Combined sample unweighted True values for PWC WeightWeighted N CPO875.3%19.0%3.6131419.0% Overlap via cell 1116.7%37.7%5.6262337.7% Overlap via RDD 130378.737.7%.4862337.7% LLO1549.3%5.7%.61945.7% Total1655100.0% 1655100.0%

46. If overlap-cell segment gets weight = 2 Combined sample unweighted True values for PWC WeightWeighted N CPO875.3%19.0%3.6131419.0% Overlap via cell 1116.7% 75.3% 2.0022213.4% Overlap via RDD 130378.7.79102561.9% LLO1549.3%5.7%.61945.7% Total1655100.0% 1655100.0%

47. Center for Survey Research University of Virginia Center for Survey Research University of Virginia In Summary...

48. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 48 Problem and solution We don’t have ‘gold standard’ data by which to weight the results of a dual-frame telephone survey in a local area Weighting to national or state averages might not be accurate We developed needed formulas that relate observed percentages to underlying population phone segment distributions We calculated ‘response rate ratios’ by comparing CHIS 2007 to regional NHIS 2007 results. We applied these ratios to calculate underlying distributions in three local telephone surveys

49. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 49 Results The estimates for three suburban counties in Virginia are quite different from national phone- segment distributions—and from each other –Cellphone penetration is higher in Northern Virginia than in downstate suburbs, or in national estimates –CPO lifestyle has been adopted by fewer people in the downstate suburbs The estimates can guide weighting of sample data –But we must use caution in weighting our cellphone samples up too much –Larger cellphone samples needed in the future

50. Center for Survey Research University of Virginia Center for Survey Research University of Virginia 50 Future research This is a time of rapid change in the telephone system –We are just learning how to deal with the weighting issues in cellphone surveys We need to look at optimization of our dual-frame designs (cf. Hartley 1962) Estimates of response rate ratios can be updated using more current national phone surveys compared to NHIS Results would be strengthened if external local data were available to validate the estimates

51. Estimating Phone Service and Usage Percentages: How to Weight the Data from a Local, Dual-Frame Sample Survey of Cellphone and Landline Telephone Users in the United States Presented at AAPOR 2009 Hollywood, FL May 14, 2009 Thomas M. Guterbock TomG@virignia.edu