Examining Best Practices for Sampling and Weighting Dual Frame Surveys Liz Kantor Advised by Dr. David Redlawsk Rutgers, The State University of New Jersey PANJAAPOR 2015 Spring Event

We know we need to include cell phones in telephone RDD samples and weight accordingly… …but we don’t agree on how

PANJAAPOR 2015 Spring Event Considerations Sampling  What % of the sample should be cell phones? Landlines? Weighting  Base weight?  Frame overlap adjustment?  Calibration adjustments?  Trimming?  Low variance!

PANJAAPOR 2015 Spring Event Data and Methods Data  Rutgers-Eagleton Poll Methods  Demographics: Chi-square  Sampling analysis: weight C & LL samples to make up varying proportions of overall sample  Weighting: 17 weighting & trimming combinations

PANJAAPOR 2015 Spring Event Weight Calculations Base weight: sample selection probability multiplied by “household” selection probability Frame adjustment: ½ Compositing, Effective Sample Size From Kennedy (2012)

PANJAAPOR 2015 Spring Event Weight Calculations Calibration  “Basic Sample Balancing” – one manual iteration of each demographic variable  Raking – multiple iterations of each demographic variable; SPSSINC RAKE Extension Trimming  Hard cap:.2 and 5  Percentile cap: 2 nd and 98 th percentiles

PANJAAPOR 2015 Spring Event Weighting Combinations Taking a cue from Kennedy (2012): * 0 = Rutgers-Eagleton Poll practices at the time of research

PANJAAPOR 2015 Spring Event Hypotheses I. Cell-only respondents will follow patterns in the literature of demographic distinctiveness and geographic mobility II. Dual users from the cell frame will differ demographically from dual users from the landline frame III. A higher % of cell phones than used in the Rutgers-Eagleton Poll will be required in the sample to match NHIS estimates

PANJAAPOR 2015 Spring Event Hypotheses IV. Half compositing will be less efficient (higher DEFF) than effective sample size V. No difference in efficiency between hard cap and percentile cap; can’t know a priori which will increase variance more

PANJAAPOR 2015 Spring Event Demographics Statistically significant differences at α =.001 * α =.01 C Only vs. Dual vs. LL only Dual from cell vs. Dual from LL Gender ✔✔ Age ✔✔ Race ✔✔ Party ID ✔ Education ✔✔*✔* Income ✔

PANJAAPOR 2015 Spring Event Demographics Differences in gender composition… Rutgers-Eagleton Poll

PANJAAPOR 2015 Spring Event Geographic Mobility Match between FIPS code (county) from sample provider and from respondent Rutgers-Eagleton Poll

PANJAAPOR 2015 Spring Event Simulated Sample Composition CELL- ONLY CELL- MOSTLY DUAL USE LANDLINE -MOSTLY LANDLINE -ONLY NHIS (NJ) 19.4%25.7%31.1%15.2%6.9% 30% C 70% LL 10.4%33.7%26.6%23.3%6.0% 40% C 60% LL 13.9%34.1%25.8%21.1%5.1% 50% C 50% LL 17.4%34.4%24.9%19.0%4.3% 60% C 40% LL 20.8%34.8%24.1%16.9%3.4% 70% C 30% LL 24.3%35.2%23.2%14.7%2.6% Rutgers-Eagleton Poll

PANJAAPOR 2015 Spring Event Design Effects Rutgers-Eagleton Poll

PANJAAPOR 2015 Spring Event Margins of Error *Calculated using Langer Research Associate’s MoE Machine Rutgers-Eagleton Poll

PANJAAPOR 2015 Spring Event Average deviation from NHIS estimates CELL- ONLY CELL- MOSTLYDUAL USE LANDLINE- MOSTLY LANDLINE- ONLY NHIS (NJ) 19.4%25.7%31.1%15.2%6.9% * indicates that the weighting combination came within 1% of the cell-only estimate Rutgers-Eagleton Poll

PANJAAPOR 2015 Spring Event Discussion Support for Hypotheses I, II, and III:  Significant demographic differences among cell only, dual users, and landline only Estimated geographic mobility much greater for Rs from cell sample vs. Rs from landline sample  Significant demographic differences between dual users from cell and landline samples  More cell phone interviews (~50%) are needed to match NHIS estimates of cell only

PANJAAPOR 2015 Spring Event Discussion No support for Hypothesis IV  ½ Compositing was not less efficient than Effective Sample Size Little support for Hypothesis V  Percentage cap was more efficient than the hard cap

PANJAAPOR 2015 Spring Event Weighting Recommendations Ultimately, no obvious best practices Criteria for choosing weighting and trimming combinations:  Includes a frame adjustment (avoid biased estimators)  Inclusion of base weight Brings percentage of cell-only respondents within 1% of NHIS estimates  Percentile cap – more efficient

PANJAAPOR 2015 Spring Event Weighting Recommendations Under these criteria, good weighting and trimming combinations are:  B: Base Weight, ½ Compositing, Basic Sample Balancing, Percentile Cap  D: Base Weight, ½ Compositing, Raking, Percentile Cap  F: Base Weight, Effective Sample Size, Basic Sample Balancing, Percentile Cap

PANJAAPOR 2015 Spring Event Future Research Replicate using other data Strengthen the theoretical basis of the criteria used to recommend weighting combinations Experiment with more aggressive percentile caps (e.g. 5% and 95%)

PANJAAPOR 2015 Spring Event Thank You