1. Tianshu Zhao & Timothy P. Johnson University of Illinois at Chicago
The Effect of Question Characteristics on Support for the Affordable Care Act ( )
Tianshu Zhao & Timothy P. Johnson
University of Illinois at Chicago

2. Background
Health care reform has been an important issue of debate in national polls since 1930s.
We focus on national polls concerning the ACA ( ).

3. Research Question
What survey-level and question-level characteristics are associated with variability in public opinion regarding the ACA?

4. Data Source 1
iPOLL Search: Obamacare/health care/health reform/ACA… 2
Select Decade: 2010s 3
Narrow Results: approve/favor/favorable/support

5. Data Source 1
iPOLL Search: Obamacare/health care/health reform/ACA… 2
Select Decade: 2010s 3
Narrow Results: approve/favor/favorable/support

6. Data Source 1
iPOLL Search: Obamacare/health care/health reform/ACA… 2
Select Decade: 2010s 3
Narrow Results: approve/favor/favorable/support

7. Sample Questions (N=312)
Do you approve or disapprove of the health care bill which was passed into law last month (March 2010)?
(CNN, April 9-April 11, )
Based on what you know, do you favor or oppose the new health care law?
(Fox News, April 6- April 7, 2010 )

8. Sample Questions (N=312)
As you may know, a new health reform bill was signed into law earlier this year (March 2010). Given what you know about the new health reform law, do you have a generally favorable or generally unfavorable opinion of it?
(Kaiser, May 11-May 16, 2010)
Do you support or oppose the health care law passed by Barack Obama and Congress in 2010?
(Quinnipiac, November 6- November 11, 2013)

9. Percent Support/Oppose by Timeline
No opinion

10. Dependent Variables Only/mostly/strongly/generally/somewhat
Approve/favor/favorable/support
Percentage support
Disapprove/unfavorable/oppose
Percentage oppose
Don’t know/no answer/no opinion/ unsure/can’t say/undecided/refused/never heard of
Percentage no opinion

11. Reality Check
“The order of words in a question and the alteration of small, simple words in the query and response categories can alter the perceived meaning and response distribution of a question (Schuman and Presser, 1981; Payne, 1951). ”
--Smith, 1987, p.75.

12. Independent Variables
Question-level measures
1. Question difficulty
2. Response option
3. Key words
Survey-level measures
1. Survey sources
2. Key events
3. Sample frame
4. Sample size
5. Timeline

13. Question-level Measures
1. Question difficulty measures
a. Number of sentences
b. Total word count
c. Fleish-Kincaid grade level
2. Response option measures
a. Option of “neither…nor…”
b. Branching response options
c. Number of options
d. Type of response options
3. Key words in question
a. “Obama”, “Congress”, “ACA”, “reform”, “overhaul”

14. Question-level Measures
1. Question difficulty measures
a. Number of sentences
b. Total word count
c. Fleish-Kincaid grade level
2. Response option measures
a. Option of “neither…nor…”
b. Branching response options
c. Number of options
d. Type of response options
3. Key words in question
a. “Obama”, “Congress”, “ACA”, “reform”, “overhaul”
Approve/disapprove
Favor/oppose
Favorable/unfavorable
Support/oppose

15. Survey-level Measures
1. Survey sources
Kaiser New Models YG
CBS Democracy Corps TIPP
Pew Ipsos-Public Affairs PIPA
Associated Press GWU Washington Post
CNN Reason-Rupe Harvard U
Gallup Resurgent Republic United Technologies
ABC NPR Public Religion
Quinnipiac Princeton U Allstate
Fox Bloomberg U of Connecticut ,etc. 1

16. Survey-level Measures
2. Key events
March 23, 2010: President Obama signs the ACA into law
January 2011: A Florida judge rules that elements of the ACA are unconstitutional
November 14, 2011: The U.S. Supreme Court agrees to hear arguments in the Obamacare case
June 28, 2012: The U.S. Supreme Court upholds the major provisions of the ACA
November 6, 2012: Obama re-elected, effectively ensuring the ACA will survive
October 1, 2013: Obamacare website problems emerge
January 1, 2014: The bulk of ACA changes go into effect with the new year
3. Sample frame: National adults, registered voters, likely voters, etc.
4. Sample size: 522 to 10,013 respondents
5. Timeline: January 2010 to March 2016

17. Independent Variables: Question Difficulty
(Dependent variable: Percentage support) B
(SE)
Number of sentences
0.01*
(0.00)
Total words
-0.00***
Flesch-Kincaid reading level
0.00*
* p < 0.05, *** p < (R2=0.16)

18. Independent Variables: Response Options
(Dependent variable: Percentage support) B
(SE)
“Neither…nor…” as a response option
-0.12***
(0.02)
Number of response options
-0.03
(0.03)
Branching response options
Response Option Format1
0.01
Approve vs. disapprove (1=yes)
-0.02**
(0.01)
Favor vs. oppose (1=yes)
-0.01
Favorable vs. unfavorable (1=yes)
-0.03***
** p < 0.01, *** p < (R2=0.31) Contrast group = support/oppose response option format

19. Independent Variables: Key Words
(Dependent variable: Percentage support) B
(SE)
“Obama” mentioned in question
0.05***
(0.01)
“Congress” mentioned in question
-0.04***
“ACA” mentioned in question
-0.03**
“Reform” mentioned in question
-0.01
“Overhaul” mentioned in question
0.01
(0.02)
** p < 0.01, *** p < (R2=0.23)

20. Independent Variables: Survey Sources
(Dependent variable: Percentage support) B
(SE)
Kaiser Health Tracking Poll
-0.02**
(0.01)
CBS News Poll
-0.03***
Pew Research Center Poll
-0.01
Associated Press Poll
-0.11***
CNN Poll
Gallup Poll
ABC News Poll
0.01
Quinnipiac University Poll
-0.03*
Fox News Poll
-0.04**
* p < 0.05, ** p < 0.01, *** p < (R2=0.34)

21. Independent Variables: Key Events
(Dependent variable: Percentage support) B
(SE)
Event 1 President Obama signs the ACA into law (Mar, 2010)
0.00
(0.01)
Event 2 A Florida judge rules the ACA are unconstitutional (Jan, 2011)
-0.00
Event 3 The U.S. Supreme Court agrees to hear arguments (Nov, 2011)
-0.01
Event 4 The Supreme Court upholds the major provisions of the ACA (Jun, 2012)
0.01
Event 5 Obama re-elected, effectively ensuring the ACA will survive (Nov, 2012)
Event 6 Obamacare website problems emerge (Oct, 2013)
-0.02
Event 7 The bulk of ACA changes go into effect with the new year (Jan, 2014)
(R2=0.03)

22. Independent Variables: Other measures
(Dependent variable: Percentage support) B
(SE)
Sample size
0.00
(0.00)
Timeline
-0.00*
Type of Sample Frame1
National adults
-0.04*
(0.02)
National adult voters
-0.00
National registered voters
N=312. R2=0.05. * p < Contrast group included all other sample frames

23. Final Model: Percentage Support
Question- and Survey-level Independent Variables B
( SE)
Total words
- 0.00***
(0.00)
“Neither…nor…” as a response option
- 0.07***
(0.01)
Approve vs. disapprove response format
-0.02**
Favor vs. oppose response format
-0.02*
“Obama” mentioned in question
0.04***
“Congress” mentioned in question
- 0.02**
CNN News Poll
0.03**
The U.S. Supreme Court agrees to hear arguments (Nov, 2011)
- 0.02*
Sample=National registered voters
- 0.05***
Note: * p < 0.05, ** p < 0.01, *** p < Non-significant variables not shown. (R2=0.48)

24. Final Model: Percentage Support
Question- and Survey-level Independent Variables B
( SE)
Total words
- 0.00***
(0.00)
“Neither…nor…” as a response option
- 0.07***
(0.01)
Approve vs. disapprove response format
-0.02**
Favor vs. oppose response format
-0.02*
“Obama” mentioned in question
0.04***
“Congress” mentioned in question
- 0.02**
CNN News Poll
0.03**
The U.S. Supreme Court agrees to hear arguments (Nov, 2011)
- 0.02*
Sample=National registered voters
- 0.05***
Note: * p < 0.05, ** p < 0.01, *** p < Non-significant variables not shown. (R2=0.48)

25. Percent Support by Total Question Words

26. Final Model: Percentage Support
Question- and Survey-level Independent Variables B
( SE)
Total words
- 0.00***
(0.00)
“Neither…nor…” as a response option
- 0.07***
(0.01)
Approve vs. disapprove response format
-0.02**
Favor vs. oppose response format
-0.02*
“Obama” mentioned in question
0.04***
“Congress” mentioned in question
- 0.02**
CNN News Poll
0.03**
The U.S. Supreme Court agrees to hear arguments (Nov, 2011)
- 0.02*
Sample=National registered voters
- 0.05***
Note: * p < 0.05, ** p < 0.01, *** p < Non-significant variables not shown. (R2=0.48)

27. Percent Support by “Neither…Nor…”in Questions

28. Final Model: Percentage Support
Question- and Survey-level Independent Variables B
( SE)
Total words
- 0.00***
(0.00)
“Neither…nor…” as a response option
- 0.07***
(0.01)
Approve vs. disapprove response format
-0.02**
Favor vs. oppose response format
-0.02*
“Obama” mentioned in question
0.04***
“Congress” mentioned in question
- 0.02**
CNN News Poll
0.03**
The U.S. Supreme Court agrees to hear arguments (Nov, 2011)
- 0.02*
Sample=National registered voters
- 0.05***
Note: * p < 0.05, ** p < 0.01, *** p < Non-significant variables not shown. (R2=0.48)

29. Final Model: Percentage Support
Question- and Survey-level Independent Variables B
( SE)
Total words
- 0.00***
(0.00)
“Neither…nor…” as a response option
- 0.07***
(0.01)
Approve vs. disapprove response format
-0.02**
Favor vs. oppose response format
-0.02*
“Obama” mentioned in question
0.04***
“Congress” mentioned in question
- 0.02**
CNN News Poll
0.03**
The U.S. Supreme Court agrees to hear arguments (Nov, 2011)
- 0.02*
Sample=National registered voters
- 0.05***
Note: * p < 0.05, ** p < 0.01, *** p < Non-significant variables not shown. (R2=0.48)

30. Final Model: Percentage Support
Question- and Survey-level Independent Variables B
( SE)
Total words
- 0.00***
(0.00)
“Neither…nor…” as a response option
- 0.07***
(0.01)
Approve vs. disapprove response format
-0.02**
Favor vs. oppose response format
-0.02*
“Obama” mentioned in question
0.04***
“Congress” mentioned in question
- 0.02**
CNN News Poll
0.03**
The U.S. Supreme Court agrees to hear arguments (Nov, 2011)
- 0.02*
Sample=National registered voters
- 0.05***
Note: * p < 0.05, ** p < 0.01, *** p < Non-significant variables not shown. (R2=0.48)

31. Final Model: Percentage Support
Question- and Survey-level Independent Variables B
( SE)
Total words
- 0.00***
(0.00)
“Neither…nor…” as a response option
- 0.07***
(0.01)
Approve vs. disapprove response format
-0.02**
Favor vs. oppose response format
-0.02*
“Obama” mentioned in question
0.04***
“Congress” mentioned in question
- 0.02**
CNN News Poll
0.03**
The U.S. Supreme Court agrees to hear arguments (Nov, 2011)
- 0.02*
Sample=National registered voters
- 0.05***
Note: * p < 0.05, ** p < 0.01, *** p < Non-significant variables not shown. (R2=0.48)

32. Final Model: Percentage Support
Question- and Survey-level Independent Variables B
( SE)
Total words
- 0.00***
(0.00)
“Neither…nor…” as a response option
- 0.07***
(0.01)
Approve vs. disapprove response format
-0.02**
Favor vs. oppose response format
-0.02*
“Obama” mentioned in question
0.04***
“Congress” mentioned in question
- 0.02**
CNN News Poll
0.03**
The U.S. Supreme Court agrees to hear arguments (Nov, 2011)
- 0.02*
Sample=National registered voters
- 0.05***
Note: * p < 0.05, ** p < 0.01, *** p < Non-significant variables not shown. (R2=0.48)

33. Conclusion
Findings reinforce our awareness that variability in responses to survey questions are associated with variability in how, when and to who questions are asked.
These findings enable us to better understand how variability influences responses and enable us to better interpret polls concerned with this topic.

34. Limitations 1. Uncontrolled context effects 2. Multicollinearity
3. Moderate sample
4. Differential response rates

35. Future Research Multi-level models Respondent level Question level
Survey level

36. Thanks for your questions