JACK FOWLER CENTER FOR SURVEY RESEARCH UMASS BOSTON

Seatbelt Use  Version 1. When riding in the back seat of a car, do you wear a seat belt all of the time, most of the time….  Version 2.  A. In the past year, have ever ridden in the back seat of a car?  B. When you are riding in the back seat of a car, do you wear a seat belt ……..

SEAT BELT USE Wear Seat Belt in Back Seat One Question % Two questions % All the time3042 Most of the time1716 Some of the time or once in a while 2118 Never244 Don’t ride in back0820

CONCLUSION  Respondents did not know what to do with the second part of the question, the assumption that they rode in the back seat, and they handled the problem in different ways—all to the detriment of the resulting data.

2-question example  1. During the past 6 months, …were you or anyone else …injured or poisoned seriously enough that you or they got medical advice or treatment?  Vs.  1A. During the past 6 months, …were you or anyone else …injured seriously enough…  1B. During the past 6 months, …were you or anyone else poisoned seriously enough…

POISONINGS OR INJURIES REQUIRING MEDICAL CARE Question TypeRate / 100 Households Combined question13/100 Separated questions p<.05 21/100

Conclusion  Respondents were distracted by trying to deal with 2 questions at once and they underreported injuries.  2 questions are better than 1.

Another example,Version 1  To lower your risk of heart problems or stroke, has a doctor or other health professional advised you to:  Cut down on salt or sodium in your diet  Eat fewer high fat..foods  Get more exercise..  Control your weight or lose weight

Version 2.  A. Has a doctor or other health professional ever advised you to cut down on salt…?  B. (IF YES) Did the doctor..recommend this for your general health or specifically to lower your risk of heart problems or stroke?

REPORTS OF DOCTOR ADVISING HEALTHY BEHAVIOR TO REDUCE HEART/STROKE RISKS HEALTHY BEHAVIOR SINGLE QUESTION SEPARATED QUESTIONS Reduce salt22%10% Fewer high-fat foods 45%17% Get more exercise 50%9% Control or lose weight 39%5%

REASON FOR HEALTHY BEHAVIOR ADVICE (2 QUESTION DESIGN) HEALTHY BEHAVIOR GENERAL HEALTH REDUCE HEART/STROKE RISKS Reduce salt41%59% Fewer high-fat foods 51%49% Get more exercise71%29% Control or lose weight 70%30%

HEALTHY BEHAVIOR ADVICE BY WHETHER HEART/STROKE RISK WAS IN QUESTION HEALTHY BEHAVIOR HEART/STROKE RISK REDUCTION IN QUESTION NO REASON MEN- TIONED IN QUESTION Reduce salt22%29% Fewer high-fat foods45%44% Get more exercise50%43% Control or lose weight39%30%

Why are rates when heart/stroke mentioned same or higher?  Hypothesis 1. Mention of risk stimulated recall off advice, offsetting fact that some people should have excluded some advice experiences  Hypothesis 2. People just ignore the heart/stroke qualifier. The question for all is just did you ever get this kind of advice.

CONCLUSIONS  1. Asking two questions at once is never a good idea.  a. Respondents routinely ignore one part of the question  b. The impacts on the data are comparatively huge  c. Whatever time is saved by asking a single, complex question is more than offset by the increased measurement error.

CONCLUSIONS CONT’D  This generalization has been around for decades, yet presumably well-trained researchers continue to ignore it.  Our goals should be to build an increasingly large set of generalizations about question characteristics that adversely affect measurement.

VALUE OF SPLIT BALLOT TESTS  Congratulations to NCHS for sponsoring these studies  The demonstration of the consequences of various wording decisions on the data is surely the most powerful way to argue for better question design  Comparatively small investments can produce important empirical results.