1 MEASUREMENT Measurement Error & Survey Construction

2 Measurement: Measurement Error in Survey Research What is measurement Error? Measurement Error = Observed Score - True Score What are the sources of measurement error in survey research and how do we minimize it? One Requirement of Effective Research Design: Minimizing measurement error (Remember MAXMINCON Principle!)

3 Measurement Error in Survey Research Instrumentation Error/Bias Error that results from lack of fidelity of the measurement instrument (e.g., poor instructions, wording, scales, response options) Response Error/Bias Error that stems from the mentality and/or predisposition of study participants Types of Measurement Error/Bias:

4 Measurement Error in Survey Research-- Instrumentation Bias Sources of Instrumentation Bias -- How to Minimize It? A. Poor Wording of Items/Instructions: –Guideline #1: Are some words in the item ambiguous? Spell out acronyms, define difficult to understand terms, and reword the items using clearer and more commonly recognized vocabulary. –Guideline #2: Does the question state the necessary/ appropriate frame of reference/criterion for answering? If not, the criterion must be clearly indicated. –Guideline #3: Does the question overemphasizes/ exaggerates/dramatizes some condition. If so, it must be toned down and stated in less dramatic terms.

5 Measurement Error in Survey Research-- Instrumentation Bias –Guideline #4: Is the question as specific as it can reasonably be? If the item is too general, state it more specifically. –Guideline #5: When possible, avoid using negatives and double- negatives in statements--they are confusing. – Guideline #6: Avoid putting a blank in the middle of an item.

6 Measurement Error in Survey Research-- Instrumentation Bias –Guideline #7: Use your pronouns carefully when writing both, the items as well as the instructions. –Guideline #8: Choose proper and clear wording and sentence structure for the items and instructions. –Guideline #9: Is the question likely to lead respondents toward a particular answer (i.e., is it a leading question)? If so, the leading phrase must be removed.

7 Measurement Error in Survey Research-- Instrumentation Bias B. Poor Content of Items/Questions: –Guideline#10: Is the question applicable to all respondents? If not, reword it and/or use a detour around it for those to whom it does not apply. –Guideline#11: Avoid branching if at all possible. The two items above can easily be rewritten as one question that applies to everyone. –Guideline #12: Does the item contain an example that is also a possible answer? If so, change or discard the example.

8 Measurement Error in Survey Research-- Instrumentation Bias –Guideline #13: Does the question require respondents to remember too much detail or to recall distant events? If so, it must be modified or generalized to make recall easier. –Guideline #14: Is the question as free from threat to respondents as possible? If not, change it to reduce the threat; consider depersonalizing it and/or asking about hypothetical situations. –Guideline #15: Does the question include more than one issue (i.e., a double- barreled item)? If so, it must be split or modified to include only a single question. –Guideline #16: Is the question “lladed” with a reason for responding in a particular way? If so, the reason must be deleted.

9 Measurement Error in Survey Research-- Instrumentation Bias –Guideline #17: Underline, italicize, or use bold print to draw respondent’s attention to important terms in the instructions. C. Poor Response Format/Options: –Guideline #18: Make sure the list of response options is exhaustive. –Guideline#19: Are “Yea-sayers” or “nay-sayers” likely to always choose one answer? Reword the item to avoid using “Yes” and “No” as choices. –Guideline #20: Use “Always” and “Never” with caution; “Almost Always” and “Almost Never” may be much better substitutes.

10 Measurement Error in Survey Research-- Instrumentation Bias –Guideline #21: Note that not all rating scales are bipolar or have to be balanced/symmetrical with a neutral mid-point (e.g., importance, likelihood, etc.). –Guideline #22: When the respondent is to select only one choice from a list of several options, make sure the choices are mutually exclusive. –Guideline #23: Make sure that the formats of your items and your response options match (are consistent).

11 Measurement Error in Survey Research-- Instrumentation Bias D. Organization and Order: –Guideline #24: Be cognizant of position/order bias. –Guideline #25: When appropriate, use the funnel approach to avoid the potential assimilation effect. That is: The more general first, followed by the more specific –Guideline #26: Use duller demographic questions in a section at the end of the questionnaire (the opposite in interviews) –Guideline #27: Use more interesting and attention-grabbing set of questions earlier in the survey.

12 Measurement Error in Survey Research-- Instrumentation Bias –Guideline #28: Use more sensitive, personal and/or threatening items later in the survey. –Guideline #29: Organize questionnaire by arranging items into subsections (based on content or/and response format), each with a clear and short instruction. –Guideline #30: At the end of the survey, indicate what your respondents are expected to do next—e.g., mail the survey to whom, at what address, by when.

13 Survey Research MEASURMENT ERROR: SOURCES OF RESPONSE ERROR/BIAS

14 Measurement Error in Survey Research-- Response Bias Sources of Response Bias -- How to minimize it? –SOCIAL DESIRABILITY: Response based on what is perceived as being socially acceptable or respectable. Response Error/Bias: Error that stems from the mentality and/or predisposition of study participants.

15 Measurement Error in Survey Research-- Response Bias –YEA- AND NAY-SAYING: Response influenced by the global tendency toward positive or negative answers. –ACQUIESCENCE: Response based on respondent’s perception of what would be desirable to the sponsor.

16 Measurement Error in Survey Research-- Response Bias –THREAT: Response influenced by anxiety or fear instilled by the nature of the question. –PRESTIGE: Response intended to enhance the image of the respondent in the eyes of others.

17 Measurement Error in Survey Research-- Response Bias –AUSPICES: Response dictated by the image or opinion of the sponsor rather than the actual question. –HOSTILITY: Response arising from feelings of anger or resentment engendered by the response task.

18 –ORDER: The sequence in which a series is listed affects the responses to the items. –EXTREMITY: Clarity of extremes and ambiguity of midrange options may encourage extreme responses. –MENTAL SET: Cognitions or perceptions based on previous items influence response to later ones (the “priming” problem).