1 Unobserved common causes of measurement and nonresponse error on the 2008 ANES Panel Survey International Total Survey Error Workshop Stowe, VT - June 13-16, 2010 Caroline Roberts – University of Lausanne, CH Patrick Sturgis – University of Southampton, UK Nick Allum – University of Essex, UK

Overview Background and motivation Objectives of this study Data and sample Methods Level of effort & response propensity analysis Structural Equation Modeling Provisional conclusions and discussion points

Background Several theoretical models specify ways measurement error and nonresponse bias might relate (see Olson 2007) ‘Common cause’ model: variables influencing response propensity also influence response accuracy Possibility to test the model restricted by data availability Focus on individual items does not address error from suboptimal response strategies

Motivation To what extent do common causes influence both types of error? – The role of motivation and ability Our approach uses: – Panel data – to investigate a range of candidate common causes – Structural Equation Models to quantify the unobserved component Theoretical and practical interest

Objectives Two elements: 1.Comparison of data quality based on respondent ‘cooperativeness’ in the panel: do the least cooperative differ from the most? 2.Analysis of common causes of response propensity and measurement error using SEM: what is the extent and magnitude of the unobserved component?

Data ANES Internet Panel Survey Recruited by RDD telephone survey Non-internet households got MSN Web-TV 21 monthly Internet surveys: $10 each Fieldwork by Knowledge Networks Advance release data file (June 2009) include recruitment data (including CATI paradata) core profile survey plus 6 ANES waves (Jan, Feb, Jun, Sep, Oct, & Nov 2008) DeBell, Krosnick, Lupia & Roberts, 2009

Sample and Fieldwork Probability sample of US citizens aged 18+ Data from 1 of 2 recruitment cohorts 12,809 landline numbers; 2,371 completed recruitment (18.5%) 4 month fieldwork – up to 50 call attempts 2 protocol changes – Refusal conversion by NORC Internet-only recruitment for 50+ calls AAPOR1 = 26% AAPOR3 = 42% 1,738 completed recruitment + at least 1 ANES wave

Panel retention NumberPercent of total Completed interviews2, Standard telephone2, Refusal conversion853.6 Internet642.7 Completed 1+ ANES waves Profile (88.9) Wave 1 (Jan 08) (90.7) Wave 2 (Feb 08) (82.7) Wave 6 (Jun 08) (80.9) Wave 9 (Sep 08) (84.3) Wave 10 (Oct 08) (85.6) Wave 11 (Nov 08) (85.3)

Cooperativeness 3 indicators of recruitment effort: 1.Number of calls to a complete interview (1-5 vs. 6 or more) 2.Whether respondent or household member refused to participate during call attempts (refused once or more vs. never refused) 3.Respondent recruited after protocol change (by internet or refusal conversion vs. by standard telephone) Actual response propensity Differences in sample composition, responsiveness, key survey estimates, data quality

Data quality Indicators of survey satisficing (Krosnick, 1991) Item non-response (wave 1 only) Non-differentiation of items with same response scale Preference for midpoints in branched questions Item sets repeated across several ANES waves: Condition of the country (5 pts); candidate liking, attitudes to groups, policy attitudes, candidate policy positions (branched 7-pt scales) Validity checks Consistency and accuracy of reports – e.g. voting (but see Berent et al. 2010)

Results Few significant differences in refusal and protocol change comparisons But respondents recruited after 6+ calls are: – Younger and more likely to be Black, non-Hispanic – less likely to have Internet access – Less likely to be Republican and conservative – Slightly more likely to satisfice And reluctance at recruitment leads to lower cooperation in panel

Responsiveness 1-5 calls (n=1222) 6+ calls (n=516) No refusal (n=1401) Refusal (n=337) Standard recruit (n=1641) Protocol change (n=97) No. of waves started (mean) 6.34 (.04) 5.85 *** (.07) 6.25 (.03) 5.95*** (.08) 6.26 (.03) 5.11 *** (.14) No. of complete waves (mean) 6.14 (.04) 5.61*** (.07) 6.05 (.04) 5.73** (.09) 6.05 (.04) 4.82*** (.15) Completed all waves % 62.5 (1.38) 44.2*** (2.19) 59.5 (1.31) 47.2*** (2.72) 60.3 (1.20) 3.1*** (1.77) Missed 1 or more waves % 23.4 (1.21) 39.5*** (2.15) 26.1 (1.17) 36.8*** (2.63) 25.2 (1.07) 79.4*** (4.13) Dropped out %14.1 (1.00) 16.3 (1.63) 14.4 (0.94) 16.0 (2.00) 14.6 (0.87) 17.5 (3.88)

Sociodemographics Less than 6 surveys completed 4-5 surveys Completed 6-7 surveys completed Male % 46.6 * % 33.8***34.6*** % 39.9**39.7* % 26.4*25.7*31.4 White, nH % 76.5***77.4***87.6 Black, nH % 13.5***13.4**7.0 Hispanic % 7.4**7.2*4.1 Other, nH % 6.5**5.13.6

Sociodemographics Less than 6 surveys completed 4-5 surveys Completed 6-7 surveys completed < High Sch % 7.4**6.5*3.8 High Sch % 19.2**19.5*13.8 College % Bachelor % Graduate % 14.4**12.3**20.4 Northeast Midwest 23.7*28.4 South West 19.0*17.8*24.1

Recruitment Variables Less than 6 surveys completed 4-5 surveys Completed 6-7 surveys completed Internet access % 83.2**83.9*88.0 Interest in politics % ★ 37.9 † Interest in computers ★ Number of opinions ★★ Important to obey authority ★

Wave 1 variables Less than 6 surveys completed 4-5 surveys Completed 6-7 surveys completed Democrat 28.3* Independent 22.9***24.0**31.9 Republican 17.4***19.9***33.6 Liberal 22.7***24.0**32.7 Moderate Conservative 27.2***29.8***45.0 Interest in pol 33.6***38.4***55.7 Voted in ***77.0***88.4 Voted Obama 28.1***34.9**43.5

W1 attitudes

Data quality Less than 6 surveys completed 4-5 surveys Completed 6-7 surveys completed W1 MP (mean).47*.48*.45 W1 ND (mean).54*.53 W2 MP (mean).60*.61*.57 W2 ND (mean).64*.61 W6 MP (mean).34**.33*.30 W6 ND (mean).46*.46 †.44 W9 MP (mean).32*.32**.28 W9 ND (mean).47* W10 MP (mean) W10 ND (mean) W11 MP (mean).26 † W11 ND (mean).49*.49.47

Summary Level of effort analysis: – Small differences between respondents as a function of ‘effort’ required to recruit them – Significant differences in their cooperativeness at later panel waves – Significant differences in demographics, on key survey estimates, and on satisficing between more and less cooperative panel recruits – A few differences on substantive items used in satisficing indicators, but not many

Common causes Ability: – Education – Computer/Internet literacy R required MSN-TV device Motivation: – Recruitment difficulty – Interest in computers – Interest in politics Demographic characteristics – sex, age, race & ethnicity

Satisficing Recruitment difficulty Ability Motivation Demographics N of refusals N of calls to complete Education Interest in politics Interest in computers Sex Age Race Web access Response N of panels started Non-differentiation Use of midpoints Common cause? Correlated residual

SEM Estimates No covariates+Recruitment+Ability+Motivation+Demographics RespSatisRespSatisRespSatisRespSatisRespSatis Refusals-.11* *.17 Calls-.04*** ** ** **.023 Postgrad Degree.44**-3.0**.37**-1.90**.26*-1.66 College Degree.42**-2.4**.35**-1.60**.22*-.47 Some college.34**-1.8**.30** **-1.40** Interest politics.37**-7.78** ** Interest computers ** * Female-.13*-1.05** Age.01**-.01 White.40**-1.82** Web TV-.29**.50 Residual correlation-.09**-.08** RMSEA.95

Summary 2 SEM: – Very weak correlation between satisficing and propensity to respond – a ‘reassuring’ result? – Recruitment difficulty predicts response propensity but not satisficing – Motivation variables better predict satisficing; ability better predicts response. Both together can jointly account for the weak correlation between propensity to respond and satisficing.

Discussion points Limitations: – absence of external records – advance release data – specification of SEM Can we improve measures of responsiveness and satisficing (including choice of item sets)? How can we best utilize the strengths and compensate for the limitations of the panel design

Thank you

Sample composition CharacteristicPercent in sample analyzed (n = 1738) Percent in population (CPS, March 2008) Male Northeast Midwest South West or older

Sample composition CharacteristicPercent in sample analyzed (n = 1738) Percent in population (CPS, March 2008) Race/ ethnicity White, non-Hispanic Black, non-Hispanic Hispanic Other, non-Hispanic Educational attainment < High school diploma High school diploma Some college Bachelor’s degree Graduate degree

Sociodemographics 1-5 calls (n=1222) 6+ calls (n=516) No refusal (n=1401) Refusal (n=337) Standard recruit (n=1641) Protocol change (n=97) Male %  % *** ** % % *** * White, nH % ** Black, nH % ** Hispanic % Other, nH %

Sociodemographics 1-5 calls (n=1222) 6+ calls (n=516) No refusal (n=1401) Refusal (n=337) Standard recruit (n=1641) Protocol change (n=97) < High Sch %  High Sch % College % Bachelor %  Graduate %  Northeast Midwest  * South ** West

Recruitment variables 1-5 calls (n=1222) 6+ calls (n=516) No refusal (n=1401) Refusal (n=337) Standard recruit (n=1641) Protocol change (n=97) Internet access % ** Interest in politics % ★ Interest in computers ★  Number of opinions ★★ Important to obey authority ★ ★ Very or extremely ★★ About many things or just about everything

Wave 1 variables Wave 1 variables (n=1577) 1-5 calls (n=1222) 6+ calls (n=516) No refusal (n=1401) Refusal (n=337) Standard recruit (n=1641) Protocol change (n=97) Democrat Independent Republican   Liberal Moderate Conservative *  Interest in pol  Planned to vote * Voted in Voted Obama *

Data quality 1-5 calls (n=1222) 6+ calls (n=516) No refusal (n=1401) Refusal (n=337) Standard recruit (n=1641) Protocol change (n=97) W1 INR (mean) W1 MP (mean).44.47* W1 ND (mean)  W2 MP (mean) W2 ND (mean) W6 MP (mean) W6 ND (mean).44.47** W9 MP (mean)  W9 ND (mean) W10 MP (mean) W10 ND (mean)