Tianshu Zhao & Timothy P. Johnson University of Illinois at Chicago The Effect of Question Characteristics on Support for the Affordable Care Act (2010-2016) Tianshu Zhao & Timothy P. Johnson University of Illinois at Chicago

Background Health care reform has been an important issue of debate in national polls since 1930s. We focus on national polls concerning the ACA (2010-2016).

Research Question What survey-level and question-level characteristics are associated with variability in public opinion regarding the ACA?

Data Source 1 iPOLL Search: Obamacare/health care/health reform/ACA… 2 Select Decade: 2010s 3 Narrow Results: approve/favor/favorable/support

Data Source 1 iPOLL Search: Obamacare/health care/health reform/ACA… 2 Select Decade: 2010s 3 Narrow Results: approve/favor/favorable/support

Data Source 1 iPOLL Search: Obamacare/health care/health reform/ACA… 2 Select Decade: 2010s 3 Narrow Results: approve/favor/favorable/support

Sample Questions (N=312) Do you approve or disapprove of the health care bill which was passed into law last month (March 2010)? (CNN, April 9-April 11, 2010 ) Based on what you know, do you favor or oppose the new health care law? (Fox News, April 6- April 7, 2010 )

Sample Questions (N=312) As you may know, a new health reform bill was signed into law earlier this year (March 2010). Given what you know about the new health reform law, do you have a generally favorable or generally unfavorable opinion of it? (Kaiser, May 11-May 16, 2010) Do you support or oppose the health care law passed by Barack Obama and Congress in 2010? (Quinnipiac, November 6- November 11, 2013)

Percent Support/Oppose by Timeline No opinion

Dependent Variables Only/mostly/strongly/generally/somewhat Approve/favor/favorable/support Percentage support Disapprove/unfavorable/oppose Percentage oppose Don’t know/no answer/no opinion/ unsure/can’t say/undecided/refused/never heard of Percentage no opinion

Reality Check “The order of words in a question and the alteration of small, simple words in the query and response categories can alter the perceived meaning and response distribution of a question (Schuman and Presser, 1981; Payne, 1951). ” --Smith, 1987, p.75.

Independent Variables Question-level measures 1. Question difficulty 2. Response option 3. Key words Survey-level measures 1. Survey sources 2. Key events 3. Sample frame 4. Sample size 5. Timeline

Question-level Measures 1. Question difficulty measures a. Number of sentences b. Total word count c. Fleish-Kincaid grade level 2. Response option measures a. Option of “neither…nor…” b. Branching response options c. Number of options d. Type of response options 3. Key words in question a. “Obama”, “Congress”, “ACA”, “reform”, “overhaul”

Question-level Measures 1. Question difficulty measures a. Number of sentences b. Total word count c. Fleish-Kincaid grade level 2. Response option measures a. Option of “neither…nor…” b. Branching response options c. Number of options d. Type of response options 3. Key words in question a. “Obama”, “Congress”, “ACA”, “reform”, “overhaul” Approve/disapprove Favor/oppose Favorable/unfavorable Support/oppose

Survey-level Measures 1. Survey sources Kaiser 69 New Models 13 YG 1 CBS 39 Democracy Corps 8 TIPP 1 Pew 33 Ipsos-Public Affairs 4 PIPA 1 Associated Press 27 GWU 3 Washington Post 1 CNN 26 Reason-Rupe 3 Harvard U 1 Gallup 18 Resurgent Republic 2 United Technologies 1 ABC 18 NPR 2 Public Religion 1 Quinnipiac 17 Princeton U 2 Allstate 1 Fox 16 Bloomberg 2 U of Connecticut ,etc. 1

Survey-level Measures 2. Key events March 23, 2010: President Obama signs the ACA into law January 2011: A Florida judge rules that elements of the ACA are unconstitutional November 14, 2011: The U.S. Supreme Court agrees to hear arguments in the Obamacare case June 28, 2012: The U.S. Supreme Court upholds the major provisions of the ACA November 6, 2012: Obama re-elected, effectively ensuring the ACA will survive October 1, 2013: Obamacare website problems emerge January 1, 2014: The bulk of ACA changes go into effect with the new year 3. Sample frame: National adults, registered voters, likely voters, etc. 4. Sample size: 522 to 10,013 respondents 5. Timeline: January 2010 to March 2016

Independent Variables: Question Difficulty (Dependent variable: Percentage support) B (SE) Number of sentences 0.01* (0.00) Total words -0.00*** Flesch-Kincaid reading level 0.00* * p < 0.05, *** p < 0.001. (R2=0.16)

Independent Variables: Response Options (Dependent variable: Percentage support) B (SE) “Neither…nor…” as a response option -0.12*** (0.02) Number of response options -0.03 (0.03) Branching response options Response Option Format1 0.01 Approve vs. disapprove (1=yes) -0.02** (0.01) Favor vs. oppose (1=yes) -0.01 Favorable vs. unfavorable (1=yes) -0.03*** ** p < 0.01, *** p < 0.001. (R2=0.31) 1 Contrast group = support/oppose response option format

Independent Variables: Key Words (Dependent variable: Percentage support) B (SE) “Obama” mentioned in question 0.05*** (0.01) “Congress” mentioned in question -0.04*** “ACA” mentioned in question -0.03** “Reform” mentioned in question -0.01 “Overhaul” mentioned in question 0.01 (0.02) ** p < 0.01, *** p < 0.001. (R2=0.23)

Independent Variables: Survey Sources (Dependent variable: Percentage support) B (SE) Kaiser Health Tracking Poll -0.02** (0.01) CBS News Poll -0.03*** Pew Research Center Poll -0.01 Associated Press Poll -0.11*** CNN Poll Gallup Poll ABC News Poll 0.01 Quinnipiac University Poll -0.03* Fox News Poll -0.04** * p < 0.05, ** p < 0.01, *** p < 0.001. (R2=0.34)

Independent Variables: Key Events (Dependent variable: Percentage support) B (SE) Event 1 President Obama signs the ACA into law (Mar, 2010) 0.00 (0.01) Event 2 A Florida judge rules the ACA are unconstitutional (Jan, 2011) -0.00 Event 3 The U.S. Supreme Court agrees to hear arguments (Nov, 2011) -0.01 Event 4 The Supreme Court upholds the major provisions of the ACA (Jun, 2012) 0.01 Event 5 Obama re-elected, effectively ensuring the ACA will survive (Nov, 2012) Event 6 Obamacare website problems emerge (Oct, 2013) -0.02 Event 7 The bulk of ACA changes go into effect with the new year (Jan, 2014) (R2=0.03)

Independent Variables: Other measures (Dependent variable: Percentage support) B (SE) Sample size 0.00 (0.00) Timeline -0.00* Type of Sample Frame1 National adults -0.04* (0.02) National adult voters -0.00 National registered voters N=312. R2=0.05. * p < 0.05. 1 Contrast group included all other sample frames

Final Model: Percentage Support Question- and Survey-level Independent Variables B ( SE) Total words - 0.00*** (0.00) “Neither…nor…” as a response option - 0.07*** (0.01) Approve vs. disapprove response format -0.02** Favor vs. oppose response format -0.02* “Obama” mentioned in question 0.04*** “Congress” mentioned in question - 0.02** CNN News Poll 0.03** The U.S. Supreme Court agrees to hear arguments (Nov, 2011) - 0.02* Sample=National registered voters - 0.05*** Note: * p < 0.05, ** p < 0.01, *** p < 0.001. Non-significant variables not shown. (R2=0.48)

Final Model: Percentage Support Question- and Survey-level Independent Variables B ( SE) Total words - 0.00*** (0.00) “Neither…nor…” as a response option - 0.07*** (0.01) Approve vs. disapprove response format -0.02** Favor vs. oppose response format -0.02* “Obama” mentioned in question 0.04*** “Congress” mentioned in question - 0.02** CNN News Poll 0.03** The U.S. Supreme Court agrees to hear arguments (Nov, 2011) - 0.02* Sample=National registered voters - 0.05*** Note: * p < 0.05, ** p < 0.01, *** p < 0.001. Non-significant variables not shown. (R2=0.48)

Percent Support by Total Question Words

Final Model: Percentage Support Question- and Survey-level Independent Variables B ( SE) Total words - 0.00*** (0.00) “Neither…nor…” as a response option - 0.07*** (0.01) Approve vs. disapprove response format -0.02** Favor vs. oppose response format -0.02* “Obama” mentioned in question 0.04*** “Congress” mentioned in question - 0.02** CNN News Poll 0.03** The U.S. Supreme Court agrees to hear arguments (Nov, 2011) - 0.02* Sample=National registered voters - 0.05*** Note: * p < 0.05, ** p < 0.01, *** p < 0.001. Non-significant variables not shown. (R2=0.48)

Percent Support by “Neither…Nor…”in Questions

Final Model: Percentage Support Question- and Survey-level Independent Variables B ( SE) Total words - 0.00*** (0.00) “Neither…nor…” as a response option - 0.07*** (0.01) Approve vs. disapprove response format -0.02** Favor vs. oppose response format -0.02* “Obama” mentioned in question 0.04*** “Congress” mentioned in question - 0.02** CNN News Poll 0.03** The U.S. Supreme Court agrees to hear arguments (Nov, 2011) - 0.02* Sample=National registered voters - 0.05*** Note: * p < 0.05, ** p < 0.01, *** p < 0.001. Non-significant variables not shown. (R2=0.48)

Final Model: Percentage Support Question- and Survey-level Independent Variables B ( SE) Total words - 0.00*** (0.00) “Neither…nor…” as a response option - 0.07*** (0.01) Approve vs. disapprove response format -0.02** Favor vs. oppose response format -0.02* “Obama” mentioned in question 0.04*** “Congress” mentioned in question - 0.02** CNN News Poll 0.03** The U.S. Supreme Court agrees to hear arguments (Nov, 2011) - 0.02* Sample=National registered voters - 0.05*** Note: * p < 0.05, ** p < 0.01, *** p < 0.001. Non-significant variables not shown. (R2=0.48)

Final Model: Percentage Support Question- and Survey-level Independent Variables B ( SE) Total words - 0.00*** (0.00) “Neither…nor…” as a response option - 0.07*** (0.01) Approve vs. disapprove response format -0.02** Favor vs. oppose response format -0.02* “Obama” mentioned in question 0.04*** “Congress” mentioned in question - 0.02** CNN News Poll 0.03** The U.S. Supreme Court agrees to hear arguments (Nov, 2011) - 0.02* Sample=National registered voters - 0.05*** Note: * p < 0.05, ** p < 0.01, *** p < 0.001. Non-significant variables not shown. (R2=0.48)

Final Model: Percentage Support Question- and Survey-level Independent Variables B ( SE) Total words - 0.00*** (0.00) “Neither…nor…” as a response option - 0.07*** (0.01) Approve vs. disapprove response format -0.02** Favor vs. oppose response format -0.02* “Obama” mentioned in question 0.04*** “Congress” mentioned in question - 0.02** CNN News Poll 0.03** The U.S. Supreme Court agrees to hear arguments (Nov, 2011) - 0.02* Sample=National registered voters - 0.05*** Note: * p < 0.05, ** p < 0.01, *** p < 0.001. Non-significant variables not shown. (R2=0.48)

Final Model: Percentage Support Question- and Survey-level Independent Variables B ( SE) Total words - 0.00*** (0.00) “Neither…nor…” as a response option - 0.07*** (0.01) Approve vs. disapprove response format -0.02** Favor vs. oppose response format -0.02* “Obama” mentioned in question 0.04*** “Congress” mentioned in question - 0.02** CNN News Poll 0.03** The U.S. Supreme Court agrees to hear arguments (Nov, 2011) - 0.02* Sample=National registered voters - 0.05*** Note: * p < 0.05, ** p < 0.01, *** p < 0.001. Non-significant variables not shown. (R2=0.48)

Conclusion Findings reinforce our awareness that variability in responses to survey questions are associated with variability in how, when and to who questions are asked. These findings enable us to better understand how variability influences responses and enable us to better interpret polls concerned with this topic.

Limitations 1. Uncontrolled context effects 2. Multicollinearity 3. Moderate sample 4. Differential response rates

Future Research Multi-level models Respondent level Question level Survey level

Thanks for your questions tzhao20@uic.edu timj@uic.edu