© 2005 Knowledge Networks, Inc. Presented at the 2005 Annual Meeting of the American Association of Public Opinion Research Tom Smith, NORC Mike Dennis, Knowledge Networks Rick Li, Knowledge Networks Comparing Knowledge Networks’ Web Enabled Panel with In-Person 2002 General Social Survey

1 Background Examine differences and similarities between in-person and web-based surveys with emphasis on the optimal treatment of the ‘don’t know’ response option. Knowledge Networks Experiment 2000 and General Social Survey (GSS)  Differences in Presentation of “Don’t Know” options.  Differences in results attributed to presentation and format of “Don’t Know” options.  Questions can be re-designed to minimize differences in results. Knowledge Networks Experiments 2002 re-designed presentation of “Don’t Know” options. Basic issue is that there is not a way to capture ‘volunteered don’t knows’ in the web mode.

2 GSS 2002 and KN 2002 Experiments GSS 2002KN 2002 Data collection organization NORCKnowledge Networks Mode of Data Collection In-personWeb-based Field periodFeb-June 2002April-June 2002 Question Items Entire GSS‘National Priority’ items Sample Size (across four experiments)

3 “Don’t Know” Treatments Experiment ConditionsSample size GSS“Don’t know” option is not read by interviewers but volunteered by respondents 2795 KN ARespondent instruction placed on first screen to skip questions to indicate “Don’t know/No opinion”; “Don’t know option is not presented on the screen 655 KN B“Don’t know” option presented on screen333 KN CRespondents who answered “Don’t know” first are prompted to choose answer on the screen that follows. 334 KN DInstruction on all screens to skip questions to indicate “Don’t know/No opinion” 333

4 Question Wording for the Tested GSS Items We are faced with many problems in this country, none of which can be solved easily or inexpensively. I am going to name some of these problems, and for each one I’d like you to tell me whether you think we are spending too much money on it, too little money, or about the right amount. First, are we spending too much, too little, or about the right amount on ______? Too much Too little About the right amount Don’t know / no opinion

5 Spending Priority Items I Standard WordingVariant Wording The space exploration programSpace exploration Improving and protecting the environmentThe environment Improving and protecting the nation's healthHealth Solving the problems of the big citiesAssistance to big cities Halting the rising crime rateLaw enforcement Dealing with drug addictionDrug rehabilitation Improving the nation's education systemEducation Improving the conditions of BlacksAssistance to Blacks The military, armaments and defenseNational defense Foreign aidAssistance to other countries WelfareAssistance for the poor

6 Spending Priority Items II Standard WordingVariant Wording Highways and bridgesN/A Social SecurityN/A Mass transportationN/A Parks and recreationN/A Assistance for childcareN/A Supporting scientific researchN/A

7 Instruction for KN Treatment A

8 Summary of “Don’t Know” Rates % of “Don’t Know” Respondents

9 Scatterplot of “Don’t Know” Rates for KN A vs GSS

10 Substantive Differences Average Percentage Points Difference Between KN and GSS (Across All 28 Spending Priority Items)

11 Scatterplot of “Too Little” Rates: KN A vs GSS

12 Scatterplot of “About Right Amount” Rates: KN A vs GSS

13 Scatterplot of “Too Much” Rates: KN A vs GSS

14 Conclusions KN Experiment A produced most similar results to GSS. Across all 28 spending priority items:  0.6 percentage points difference on “Don’t Know”.  4.8 percentage points difference substantively. Some spending items show more significant differences substantively. For instance:  Dealing with drug addiction / Drug rehabilitation  Improving conditions of blacks / Assistance to blacks  Welfare / Assistance to the poor