How Much is Too Much? How adding information to a question may narrow, instead of expand, understanding Carol Cosenza Center for Survey Research University of Massachusetts Boston Presented at:QUEST 2007 Ottawa

How We Try to Help R’s Understand Give definitions Add common examples Add less common examples

Definition The next question is about strenuous tasks done around your home. By strenuous, we mean any chores or projects that made you feel very tired by the time you finished them. In the past 30 days, on how many days did you do strenuous tasks in or around your home? A pneumonia shot, also called the pneumoccoccal vaccine, is usually given only once or twice in a person’s lifetime and is different from the flu shot. Have you ever had a pneumonia shot?

How We Try to Help R’s Understand Give definitions Add common examples Add less common examples

(Common) Examples The next question is about vegetables you have eaten, including corn, green beans, spinach, salad vegetables, or any others. The next question is about vegetables you have eaten, including vegetables eaten by themselves, mixed in with other dishes, salad vegetables, or any others.

How We Try to Help R’s Understand Give definitions Add common examples Add less common examples

(Less Common) Examples A health provider is anyone you would see for health care, including eye doctors, mental health workers, and physical therapists.

The Experiments

Wave 1 Split Ballot design (Spring 2005) 454 respondents from an RDD sample 39.6% male 36.3% with high school education or less 51.0% over age 50 (mean=51.3 years)

Wave 2 Split Ballot design (Spring & Fall 2006) 425 respondents from an RDD sample 37.9% male 29.2% with high school education or less 60.0% over age 50 (mean=53.7 years)

Skin Growth VERSION 1: In the last 12 months, did you see or talk to a doctor about a tumor, cyst or growth of the skin? VERSION 2: In the last 12 months, did you see or talk to a doctor about any kind of growth on your skin?

Skin Growth VERSION 1: In the last 12 months, did you see or talk to a doctor about a tumor, cyst or growth of the skin? VERSION 2: In the last 12 months, did you see or talk to a doctor about any kind of growth on your skin? Version 1 (tumor, cyst, or growth) Version 2 (any growth) YES – had growth8.7% (n=13)18.2% (n=27) p=.015

Problems Breathing VERSION 1: During the past 12 months, did you see or talk with a doctor about any problems with coughing or breathing including asthma, pneumonia, or chronic lung disease? VERSION 2: During the past 12 months, did you see or talk with a doctor about any problems with coughing or breathing?

Problems Breathing Version 1 (with examples) Version 2 (no examples) YES – had problems16.7% (n=35)19.9% (n=43) VERSION 1: During the past 12 months, did you see or talk with a doctor about any problems with coughing or breathing, including asthma, pneumonia, or chronic lung disease? VERSION 2: During the past 12 months, did you see or talk with a doctor about any problems with coughing or breathing? p = n. s.

Arthritis & Joint Pain During the past 30 days, have you had symptoms of pain, aching, or stiffness in or around a joint? Did your joint symptoms first begin more than 3 months ago? Have you ever seen a doctor or other health professional for these joint symptoms? Have you ever been told by a doctor or other health professional that you have some form of arthritis, rheumatoid arthritis, gout, lupus, or fibromyalgia?

Arthritis & Joint Pain 1. During the past 30 days, have you had symptoms of pain, aching, or stiffness in or around a joint? Did your joint symptoms first begin more than 3 months ago?/Have you ever seen a doctor or other health professional for these joint symptoms? 2. Have you ever been told by a doctor or other health professional that you have some form of arthritis, (rheumatoid arthritis, gout, lupus, or fibromyalgia)? 1.Have you ever been told by a doctor or other health professional that you have some form of arthritis, (rheumatoid arthritis, gout, lupus, or fibromyalgia)? 2. During the past 30 days, have you had symptoms of pain, aching, or stiffness in or around a joint? Did your joint symptoms first begin more than 3 months ago?/Have you ever seen a doctor or other health professional for these joint symptoms?

Number of times talked to a doctor VERSION 1: In the past 12 months, how many times have you seen or talked on the telephone with a {DOCTOR} about your {health}? VERSION 2: In the past 12 months, how many times have you seen or talked with a {DOCTOR} about your {health}?

Number of times talked to a doctor VERSION 1: In the past 12 months, how many times have you seen or talked on the telephone with a {DOCTOR} about your {health}? VERSION 2: In the past 12 months, how many times have you seen or talked with a {DOCTOR} about your {health}? nRangeMean Version 1 (phone) times3.34 (SD=6.31) Version 2 (no phone)3410 – 50 times3.72 (SD=4.93)

Number of times talked to a doctor When number of times talking to doctor is top-coded at 12: p=.008 MeanStandard Error of Mean Version 1 (phone)2.71 (SD=3.10).166 Version 2 (no phone)3.36 (SD=3.27).177 VERSION 1: In the past 12 months, how many times have you seen or talked on the telephone with a {DOCTOR} about your {health}? VERSION 2: In the past 12 months, how many times have you seen or talked with a {DOCTOR} about your {health}?

Number of times talked to a doctor The original intent of adding “telephone” was the fear that Rs who only talked on the phone but not say “yes” to “seen or talked with” This test shows that there is a difference – in the opposite way. Respondents seem to be distracted by “telephone” in Version 1 This result was validated by cognitive testing. VERSION 1: In the past 12 months, how many times have you seen or talked on the telephone with a {DOCTOR} about your {health}? VERSION 2: In the past 12 months, how many times have you seen or talked with a {DOCTOR} about your {health}?

Conclusions Hard to know whether the extra “stuff” we add to questions helps broaden a respondent’s thinking or narrows it. There is some evidence that adding more “serious” health conditions as examples in a question may make it harder for respondents to say yes Split Ballot experiments help us figure this out. Cognitive testing, behavior coding and other evaluative techniques can provide us with additional clues.

Thank you