Questionnaires as Instruments Most frequently used survey instrument Are scientific instrument just like an MRI machine Quality of the questionnaire determines accuracy and precision of the measurements Preferences and attitudes most often these are measured with self-report scales participants respond on rating scales Usually Likert scales from 1-7 or 1-9 For example, an assessment of emotional well-being might include the following items: My mood is generally positive. Strongly disagree 1 ------ 2 ------ 3 ------ 4 ------ 5 Strongly agree I am often sad. Strongly disagree 1 ------ 2 ------ 3 ------ 4 ------ 5 Strongly agree

Questionnaires as Instruments Psychologists measure different types of variables: Demographic: (e.g., age, gender, race, socioeconomic status) To check for response bias To categorize data i.e. differences between young and older adults Use a well know template such as US Census FIGURE 5.5 Although ethnic background is an important demographic variable, accurately classifying people on this variable is not an easy task.

Reliability of Self-Report Measures Reliability refers to the consistency of measurement. Assessed by Test-retest reliability: Administer measure two times to the same sample. Individuals’ scores should be consistent over time. A high correlation between the two scores indicates good test-retest reliability (r >.80) r value ranges from 0 (no correlation) to 1 (perfect correlation)

Reliability of Self-Report Measures How do we improve reliability? Many similar items on the same construct improves reliability Multiple questions that focus on the same construct (operational definition) For example several questions on quality of sleep When the sample is diverse relative to the construct “quality of sleep” Some participants have poor sleep while others have great sleep To avoid restricted range of measurements Testing situation is free of distractions and instructions are clear.

Reliability of Self-Report Measures Reliable measures make us more confident that we are consistently measuring a construct within a sample, but are reliable measures truthful? I could reliably measure your research methods knowledge by measuring your height. The taller you are, the better your score. Is this a truthful or accurate measure of research methods knowledge? Note: We do not expect some measures to produce consistent scores over time. When people change on a particular variable over time we expect the measure to have low test-retest reliability Scores on a math exam before and after taking a math course

Validity of Self-Report Measures Validity refers to the truthfulness of a measure. A valid measure assesses what it is intended to measure. Construct Validity: Does an instrument measure the theoretical construct (concept) it was designed to measure? Such as sleep quality

Validity of Self-Report Measures Construct Validity: This seems like a straightforward question, but consider widely used measures of intelligence which include items such as: comprehension: “Why would people use a secret ballot?” vocabulary: “What does dilatory mean?” similarities: “How are a telephone and a radio alike?” Do these items (and others like them) assess intelligence in a valid manner? The construct validity of intelligence measures is a matter of heated debate.

Validity of Self-Report Measures Establishing the construct validity of a measure depends on convergent validity and discriminant validity. Convergent validity refers to the extent to which two measures of the same construct are correlated (go together). Discriminant validity refers to the extent to which two measures of different constructs are not correlated (do not go together).

Example of Construct Validity Suppose you have developed a new measure of self-esteem (i.e., a person’s sense of self-worth). Which constructs listed below would you expect to show convergent validity ? Which would have discriminant validity? Measures of: Depression Well-being Social anxiety Life satisfaction Grade point average

Example of Construct Validity Table 5.1 Life Satisfaction construct Satisfaction with Life Scale (SWLS) Life Satisfaction (LS-5) Positive Affect (PA) Is life satisfaction the same or different from being happy?

Example of Construct Validity Table 5.1 Check correlation between each of the measures Compare in a correlation matrix Each value is a correlation coefficient SWLS – LS5 correlation of .77 evidence of convergent validity SWLS – PA correlation of .42 evidence of discriminant validity LS5 – PA correlation of .47 evidence of discriminant validity

Constructing a Questionnaire The best choice for selecting a questionnaire is to use one that already has been established as reliable and valid. If a suitable measure cannot be found, researchers choose to create their own questionnaire. It may seem easy, but a lot goes into developing a reliable and valid questionnaire.

Constructing a Questionnaire Important steps for preparing a questionnaire: 1. Decide what information should be sought, demographics, constructs, 2. Decide how to administer the questionnaire 3. Write a first draft of the questionnaire. (borrow from other surveys) Reexamine and revise the questionnaire Pretest the questionnaire using a sample of respondents under conditions similar to the planned administration of the survey. 6. Edit the questionnaire, and specify the procedures for its use.

Guidelines for Effective Wording of Questions Choose how participants will respond: free-response: Open-ended (fill in the blank) questions allow greater flexibility in responses but are difficult to code. closed-response: are quicker to respond to and easier to score may not accurately describe individuals’ responses For example: Multiple choice, True-false, Likert Scale Use simple, direct, and familiar vocabulary keep questions short (20 or fewer words) Respondents will interpret the meaning of words Sensitivity to cultural and linguistics differences in word usage

Guidelines for Effective Wording of Questions Write clear and specific questions: Avoid double-barreled questions (e.g., “Do you support capital punishment and abortion?”). Place any conditional phrases at the beginning of the question (e.g., “If you were forced to leave your current city, where would you live?” Avoid leading questions (e.g., “Most people favor gun control; what do you think?”). Avoid loaded (emotion-laden) questions (e.g., “People who discriminate are racist pigs: T or F”). Avoid response bias with Likert Scale questions Word some of the questions in the opposite direction 1 is Strongly disagree 7 is Strongly disagree Original: I get plenty of sleep Reversed : I do not get enough sleep

Guidelines for Ordering of Questions For self-administered questionnaires, place the most interesting questions first to capture respondents’ attention. For personal and telephone interviews, place demographic questions first to establish rapport with the respondent. Use funnel questions: Start with the most general questions, and move to more specific questions for a given topic. Use filter questions: These questions direct respondents to the survey questions that apply directly to them.

Thinking Critically About Survey Research Correspondence Between Reported and Actual Behavior People’s responses on surveys may not be truthful. Reactivity: People sometimes don’t report truthful responses, because they know the information is being recorded. Social Desirability occurs when people respond to surveys as they think they “should,” rather than how they actually feel or believe.

Thinking Critically About Survey Research Generally, researchers accept people’s responses as truthful, unless there’s reason to suspect otherwise For example, responses aren’t consistent or visual pattern of responses forms a picture. Because behavior doesn’t always match verbal reports of behavior, the multimethod approach to answering questions in psychology is best. FIGURE 5.6 How people say they would respond to this type of situation does not always match what they actually do.

Thinking Critically About Survey Research Correlation and Causality “Correlation does not imply causation.” Example: Correlation between being socially active (outgoing) and life satisfaction Three possible causal relationships: A causes B being outgoing causes people to be more satisfied with their life B causes A being more satisfied with life causes people to be more outgoing C causes A and B A third variable “number of friends” can explain the relationship between socially active (outgoing) and life satisfaction A correlation that can be explained by a third variable is called a “spurious relationship.”

Thinking Critically About Survey Research, continued Diagram of direct and indirect relationships Chaos “chaotic family life” path b path c Poverty Psychological path a (direct) distress “Chaos” mediates the relationship between poverty and psychological distress among children. 20

Thinking Critically About Survey Research, continued Path analysis example A moderator variable may affect the direction and strength of these relationships. Possible moderators: Sex of the child Population density (e.g., rural, urban) Personality features of children (e.g., resilience) 21

Thinking Critically About Survey Research, continued Path analysis Helps us to understand relationships among variables But these relationships are still correlational Cannot make definitive causal statements Other untested variables may be important 22