Developing Survey Items Bonnie L. Halpern-Felsher, Ph.D. Professor University of California, San Francisco

From Last Week Find existing measures of interest Create/continue creating/find an instrument with: –At least 2: factual questions frequency and quantity questions “feelings” subjective questions Evaluative questions Scales

Respondents are Able to Respond Items are specific, not ambiguous. Respondents know and remember information. Minimize burden of recall. If recall is needed, help respondent: –Place events or behavior in time –Use memory aids

Respondents are Able to Respond Examples: –Factual Data: SES, Parent Education, Income, Medical Data. Can ask the source (e.g., parents, teachers, chart reviews –Event-Specific Data: Since high school graduation… Since we last surveyed you… Provide calendars…

Respondents are Willing to Respond Items should be written such that respondent does not feel the need to respond inaccurately. Assure confidentiality.

Respondents are Willing to Respond Handling “Don’t Know” and “Not Sure” Responses: –Can provide a screening question –Can provide an option for N/A, None, Don’t Know –Don’t assume or judge

Ambiguous Measurement Items, measures and response sets should be specific enough so as to inform participants regarding the kinds of responses that are acceptable. Do not leave it open to interpretation. Only assess one concept per question.

Ambiguous Measurement Leads to: –Unreliable and invalid information. –Inability to discern age or developmental differences. –Inability to interpret data.

Ambiguous Measurement Can occur with respect to: –Behavior –Time –Events

Ambiguous Measurement Examples: Time In the past week, on how many days did you do homework?

Ambiguous Measurement Examples: Time In the past week, on how many days did you do homework? Better: In the past 7 days, on how many days did you do homework?

Ambiguous Measurement Examples: Behavior Have you ever smoked? Have you ever had sex? What is the chance you will get an STD if have sex?

Ambiguous Measurement Examples: Behavior Have you ever smoked? (amount) Have you ever had sex? (type of sex) What is the chance you will get an STD if have sex? (type of sex, context, partner type)

Factual Questions

Objective: Measure Age

Factual Questions Objective: Measure Age –Depends on how precise you need to be: Exact to the day? General year enough? –Depends on how much participants knows/remembers.

Factual Questions Objective: Measure Age –How old were you on your last birthday? –On what date were you born? –How old are you today?

Factual Questions Objective: Soft drink consumption

Factual Questions Objective: Soft drink consumption –Depends on how precise you need to be: Simply describe general consumption? Assess number of drinks consumed by sample in general? Examine trends over time?

Factual Questions Objective: Soft drink consumption –How many soft drinks did you drink yesterday? –How many soft drinks did you drink in the last 7 days? Should define “soft drink”

Factual Questions Objective: Use of Medical Care

Factual Questions Objective: Use of Medical Care –How are you defining medical care? –Are you counting events? Characterizing individuals?

Factual Questions Objective: Use of medical care –How many times have you seen or talked to a doctor about your health in the past 2 weeks? –How many times have you received any kind of medial care in the last 2 weeks? –How many times have you received any kind of medical care in the last 12 months?

Factual Questions Objective: Measure hospitalizations

Factual Questions Objective: Measure hospitalizations –Depends on how precise you need to be: Exact to the day? General year enough? –How is hospitalizations being defined?

Factual Questions Objective: Measure Age –How many times have you been hospitalized in the past year? Problems?

Factual Questions Objective: Measure Age –How many times have you been hospitalized in the past year? Better: –In the past 12 months, since (DATE) a year ago, how many different times have you been admitted to a hospital as a patient overnight or longer?

Factual Questions Objective: Alcohol consumption

Factual Questions Objective: Alcohol consumption –How many drinks did you have altogether yesterday? Problems?

Factual Questions Objective: Alcohol consumption –How many drinks did you have altogether yesterday? Better: –On days when you have anything alcoholic to drink, how many drinks do you usually have?

Factual Questions Objective: Alcohol consumption –If a teen, or for all people… Likely need to define drink – –E.g., How many times have you had a glass of wine, bottle of beer, or shot of whiskey?

Subjective Questions

Subjective States How would you rate your health?

Subjective States Do you consider crime to be a problem in your neighborhood?

Subjective States Think of an important adult in your life…

Subjective States What is the chance you will get into trouble if you tell a lie?

Ambiguous Measurement Examples: Multiple Concepts I love to play ball sometimes. (True/False)

Ambiguous Measurement Examples: Multiple Concepts I love to play ball sometimes. (True/False) –Is subject responding to “sometimes” or “playing ball?”

Ambiguous Measurement: Some Solutions Provide as much detail as is necessary…without burdening respondent. If construct being measured is too complex for one item, use multiple items. Provide definitions, either before the measure or embedded within the measure.

Ambiguous Measurement: Some Solutions Definition Before Measure: –For the next 10 questions, when we ask about sex, we are talking about consensual intercourse (going all the way…) Within the item: –Have you ever had at least one drink of alcohol (that is, one beer, glass of wine, shot of hard liquor or wine cooler?)

Ambiguous Measurement: Some Solutions Scenario: –Imagine you have been DATING TANYA FOR 3 MONTHS. You both have had sex with two other people but not with each other. Tonight, you and Tanya have sex one time. You DO NOT USE A CONDOM or other safer sex method (e.g., other birth control, withdrawal).

Answering the Questions: Scale Development

General Principles Response set must fit the item/question. Response set should be consistent across types of questions and throughout survey. Balance between detail and accuracy. Responses should fit analytic plan. Consider number and type of response categories.

Response Set Must Fit Items How likely are you to drink alcohol in the next 6 months? –Definitely will to Definitely will not Vs. –Very likely to Very Unlikely

Consistent Response Do not switch order of responses. Do not switch number or type of responses. –Examples: Yes/No  No/Yes Agree/Disagree  Disagree/Agree

Detail vs. Accuracy In general, more detail is best as you can always recode or reduce data, especially with subjective states. But, can reduce accuracy, especially when assessing factual information.

Fits With Analytic Plan How will you analyze the data?

Number of Response Categories In general, between 5 and 7 responses is best to provide meaningful data. Little new, valid data is obtained after 10. If phone survey, best is 3-4 response choices. Odd vs. even number?

Type of Response Sets Do not use both numbers and adjectives (e.g., 1-5 and likely scale). Best: Adjectives, as numbers do not always engender meaning in the same way. –E.g., the mid-point on a 10-point scale is not always the same to people.

Type of Response Sets Disagree/agree Rank order Narrative or open-ended Magnitude estimation techniques

Type of Response Sets Depends on research aims, answers sought, and how will analyze data!

Type of Response Sets For example: –Overall, how would you rate your physical health? Excellent  Poor –On a scale from 0 to 10, where 10=best and 0=worst, how would you rate your physical health? –Overall, would you say you are in good physical health? GOOD/NOT GOOD

Type of Response Sets Likely scales: –Very likely –Likely –Unlikely –Very unlikely