COLLECTING DATA ON A SAMPLE OF RESPONDENTS Designing survey instruments

Remember: The sample survey is a research technique used to estimate population parameters A number of people are asked a series of questions from a standardized instrument concerning a variety of topics, and the answers are analyzed  Interviews v. questionnaires

Options for administration Personal interviews Phone interviews Mail questionnaires Computer-mediated questionnaires Survey design is strongly influenced by administration option chosen

The importance of quality instruments A well-designed instrument is crucial to the success of any sample survey or census With few exceptions, poorly written instruments represent a much greater threat to the validity of your conclusions than do sampling problems or other common threats to survey research validity

How do we go about developing the instrument? The most important thing is knowing what we want to know  What are the most significant questions we have?  What information is crucial to answering these questions?  Guided by literature review  What information can best be obtained in other ways?

How do we go about developing the instrument? Prioritize the importance of the information  Crucial v. important v. useful but not necessary  Getting really good information on a limited set of constructs may be more valuable than a lot of scattered facts and incomplete ideas on many constructs

Don’t go fishing! Too often researchers include many more questions than necessary  Too many ‘interesting but unnecessary’ questions  Tradeoff between additional questions and completion rate Try to avoid a phone survey that lasts more than 15 minutes  Less for questionnaires

Typical instrument format Greeting and introduction Early general questions  Develop rapport  Get the respondent thinking about the topics of study  Generate ‘top-of-mind’ data

Early to mid survey  Focus on topics of interest  Use closed-ended but still somewhat general questions  Move from category to brand questions, medium to station to program questions, etc.

Late survey questions  More open-ended questions  Questions dependent upon deeper review of content  Respondent has been thinking about topics for a while  Edgiest questions  Greatest level of rapport necessary

Final questions  Most embarrassing/difficult questions  Demographics (often lose respondents when you ask about income, etc.)

Closing  Thank your respondents for their time  Reassure them that their answers were helpful and useful regardless of the quality of responses/level of knowledge of the respondent  Any necessary debriefing, especially if you were relatively guarded at the beginning or engaged in some sort of misleading or deceptive instructions, etc.

Impact of data collection method Construct the instrument according to the method of data collection  Self-administered questionnaires require more instructions on the instrument itself than do interviews  When the instrument is administered by an interviewer, written instructions to the interviewer (usually placed in the margin) can be useful in improving response and/or validity of the information gathered

Skip Patterns When you want to avoid asking inappropriate questions, you first ask a ‘screening question’ to identify appropriate respondents  Those for whom the question is inappropriate are not asked the question and ‘skip’ forward to the next appropriate question on the instrument

Example If you want to ask questions about the kinds of beer someone drinks, you would first identify respondents who do not drink beer and have them skip to the next question appropriate to them Example: 1. Do you ever drink beer?  Yes (continue)  No (skip to Q. 3) 2. Which brands do you drink? ______________ 3. Do you ever drink coffee?

Writing the questions Levels of measurement  The higher the level of measurement, the greater the ability to use powerful statistics to analyze your data Scale types Response options  Yes/no  Agree/disagree  (Strongly, somewhat)  Numeric responses v. categories

Question formats Try to develop repetitive response options  Increases speed  Makes respondent feel more comfortable  Reduces errors  Boxes, check-offs, white space

Question wording The wording of questions on any research instrument is crucial. There is usually no more important decision than how a question is asked.  Results from surveys have been shown to be strongly influenced by even subtle differences in question wording.

Instrument format Matters of crowding, use of white space, inclusion of instructions, typeface, and so on are important.

Test the instrument Pre-test the instrument  Have trained interviewers try out your draft on each other,  Determine whether questions are misleading, ambiguous, biased  Determine how long the instrument takes to complete Use this information honestly when recruiting respondents  Ask how the respondent interprets the question  Allow for open-ended responses where closed-ended questions have been written

Developing the instrument Revise and reorganize the instrument based on the feedback from the testing Try it out on respondents from the population being studied Make final adjustments  Always keep track of the changes you made

Developing the instrument Obtain IRB approval (if this is an academic study) Enter the field and gather data  Monitor the success of the instrument and make changes if absolutely necessary  Changes must be run by the IRB

A number of concerns influence the structure of the instrument: First and foremost the essential information needed to answer the research questions/hypotheses Data collection method  Interview, self-administered questionnaire, online v. paper, etc.  Amount and type of instruction, skip patterns, incentives The normal progression of the interview/ questionnaire  Developing rapport/trust Dynamics of respondent memory  Question ordering

A number of concerns influence the structure of the instrument: Respondent facility with language and/or advanced concepts  Kids  Recent immigrants Sensitivity of question topics  Forms and methods of providing confidentiality Efficiency in asking questions