1. Questionnaires as Instruments
Most frequently used survey instrument
Are scientific instrument just like an MRI machine
Quality of the questionnaire determines accuracy and precision of the measurements
Preferences and attitudes
most often these are measured with self-report scales
participants respond on rating scales
Usually Likert scales from 1-7 or 1-9
For example, an assessment of emotional well-being might include the following items:
My mood is generally positive.
Strongly disagree Strongly agree
I am often sad.
Strongly disagree Strongly agree

2. Questionnaires as Instruments
Psychologists measure different types of variables:
Demographic: (e.g., age, gender, race, socioeconomic status)
To check for response bias
To categorize data i.e. differences between young and older adults
Use a well know template such as US Census
FIGURE 5.5 Although ethnic background is an important demographic variable, accurately classifying people on this variable is not an easy task.

3. Reliability of Self-Report Measures
Reliability refers to the consistency of measurement.
Assessed by Test-retest reliability:
Administer measure two times to the same sample. Individuals’ scores should be consistent over time.
A high correlation between the two scores indicates good test-retest reliability (r >.80)
r value ranges from 0 (no correlation) to 1 (perfect correlation)

4. Reliability of Self-Report Measures
How do we improve reliability?
Many similar items on the same construct improves reliability
Multiple questions that focus on the same construct (operational definition)
For example several questions on quality of sleep
When the sample is diverse relative to the construct “quality of sleep”
Some participants have poor sleep while others have great sleep
To avoid restricted range of measurements
Testing situation is free of distractions and instructions are clear.

5. Reliability of Self-Report Measures
Reliable measures make us more confident that we are consistently measuring a construct within a sample, but are reliable measures truthful?
I could reliably measure your research methods knowledge by measuring your height. The taller you are, the better your score.
Is this a truthful or accurate measure of research methods knowledge?
Note: We do not expect some measures to produce consistent scores over time.
When people change on a particular variable over time
we expect the measure to have low test-retest reliability
Scores on a math exam before and after taking a math course

6. Validity of Self-Report Measures
Validity refers to the truthfulness of a measure.
A valid measure assesses what it is intended to measure.
Construct Validity: Does an instrument measure the theoretical construct (concept) it was designed to measure? Such as sleep quality

7. Validity of Self-Report Measures
Construct Validity:
This seems like a straightforward question, but consider widely used measures of intelligence which include items such as:
comprehension: “Why would people use a secret ballot?”
vocabulary: “What does dilatory mean?”
similarities: “How are a telephone and a radio alike?”
Do these items (and others like them) assess intelligence in a valid manner?
The construct validity of intelligence measures is a matter of heated debate.

8. Validity of Self-Report Measures
Establishing the construct validity of a measure depends on
convergent validity and
discriminant validity.
Convergent validity refers to the extent to which two measures of the same construct are correlated (go together).
Discriminant validity refers to the extent to which two measures of different constructs are not correlated (do not go together).

9. Example of Construct Validity
Suppose you have developed a new measure of self-esteem (i.e., a person’s sense of self-worth).
Which constructs listed below would you expect to show convergent validity ?
Which would have discriminant validity?
Measures of:
Depression
Well-being
Social anxiety
Life satisfaction
Grade point average

10. Example of Construct Validity Table 5.1
Life Satisfaction construct
Satisfaction with Life Scale (SWLS)
Life Satisfaction (LS-5)
Positive Affect (PA)
Is life satisfaction the same or different from being happy?

11. Example of Construct Validity Table 5.1
Check correlation between each of the measures
Compare in a correlation matrix
Each value is a correlation coefficient
SWLS – LS5 correlation of .77 evidence of convergent validity
SWLS – PA correlation of .42 evidence of discriminant validity
LS5 – PA correlation of .47 evidence of discriminant validity

12. Constructing a Questionnaire
The best choice for selecting a questionnaire is to use one that already has been established as reliable and valid.
If a suitable measure cannot be found, researchers choose to create their own questionnaire.
It may seem easy, but a lot goes into developing a reliable and valid questionnaire.

13. Constructing a Questionnaire
Important steps for preparing a questionnaire:
1. Decide what information should be sought, demographics, constructs,
2. Decide how to administer the questionnaire
3. Write a first draft of the questionnaire. (borrow from other surveys)
Reexamine and revise the questionnaire
Pretest the questionnaire using a sample of respondents under conditions similar to the planned administration of the survey.
6. Edit the questionnaire, and specify the procedures for its use.

14. Guidelines for Effective Wording of Questions
Choose how participants will respond:
free-response: Open-ended (fill in the blank) questions allow greater flexibility in responses but are difficult to code.
closed-response:
are quicker to respond to and easier to score
may not accurately describe individuals’ responses
For example: Multiple choice, True-false, Likert Scale
Use simple, direct, and familiar vocabulary
keep questions short (20 or fewer words)
Respondents will interpret the meaning of words
Sensitivity to cultural and linguistics differences in word usage

15. Guidelines for Effective Wording of Questions
Write clear and specific questions:
Avoid double-barreled questions (e.g., “Do you support capital punishment and abortion?”).
Place any conditional phrases at the beginning of the question (e.g., “If you were forced to leave your current city, where would you live?”
Avoid leading questions (e.g., “Most people favor gun control; what do you think?”).
Avoid loaded (emotion-laden) questions (e.g., “People who discriminate are racist pigs: T or F”).
Avoid response bias with Likert Scale questions
Word some of the questions in the opposite direction
1 is Strongly disagree is Strongly disagree
Original: I get plenty of sleep
Reversed : I do not get enough sleep

16. Guidelines for Ordering of Questions
For self-administered questionnaires, place the most interesting questions first to capture respondents’ attention.
For personal and telephone interviews, place demographic questions first to establish rapport with the respondent.
Use funnel questions: Start with the most general questions, and move to more specific questions for a given topic.
Use filter questions: These questions direct respondents to the survey questions that apply directly to them.

17. Thinking Critically About Survey Research
Correspondence Between Reported and Actual Behavior
People’s responses on surveys may not be truthful.
Reactivity: People sometimes don’t report truthful responses, because they know the information is being recorded.
Social Desirability occurs when people respond to surveys as they think they “should,” rather than how they actually feel or believe.

18. Thinking Critically About Survey Research
Generally, researchers accept people’s responses as truthful, unless there’s reason to suspect otherwise
For example, responses aren’t consistent or visual pattern of responses forms a picture.
Because behavior doesn’t always match verbal reports of behavior, the multimethod approach to answering questions in psychology is best.
FIGURE 5.6 How people say they would respond to this type of situation does not always match what they actually do.

19. Thinking Critically About Survey Research
Correlation and Causality
“Correlation does not imply causation.”
Example: Correlation between being socially active (outgoing) and life satisfaction
Three possible causal relationships:
A causes B
being outgoing causes people to be more satisfied with their life
B causes A
being more satisfied with life causes people to be more outgoing
C causes A and B
A third variable “number of friends” can explain the relationship between socially active (outgoing) and life satisfaction
A correlation that can be explained by a third variable is called a “spurious relationship.”

20. Thinking Critically About Survey Research, continued
Diagram of direct and indirect relationships
Chaos “chaotic family life”
path b path c
Poverty Psychological
path a (direct) distress
“Chaos” mediates the relationship between poverty and psychological distress among children. 20

21. Thinking Critically About Survey Research, continued
Path analysis example
A moderator variable may affect the direction and strength of these relationships.
Possible moderators:
Sex of the child
Population density (e.g., rural, urban)
Personality features of children (e.g., resilience) 21

22. Thinking Critically About Survey Research, continued
Path analysis
Helps us to understand relationships among variables
But these relationships are still correlational
Cannot make definitive causal statements
Other untested variables may be important 22