1. Non-Experimental Designs: Survey Methods
Psych 231: Research Methods in Psychology

2. I will aim to have the exams graded and entered into ReggieNet by Tuesday
Remember that piloting is happening in labs this week, so be prepared with everything that you need, and attendance is important
Announcements

3. Non-Experimental designs
Sometimes you just can’t (or don’t want to) perform a fully controlled experiment
Because of the issue of interest
Limited resources (not enough subjects, observations are too costly, etc.).
There are other kinds of designs that may be used
Surveys
Correlational studies
Quasi-Experiments
Developmental designs
Small-N designs
This does NOT imply that they are bad designs
Just remember the advantages and disadvantages of each
Non-Experimental designs

4. Non-Experimental designs
Sometimes you just can’t (or don’t want to) perform a fully controlled experiment
Because of the issue of interest
Limited resources (not enough subjects, observations are too costly, etc.).
There are other kinds of designs that may be used
Surveys
Correlational studies
Quasi-Experiments
Developmental designs
Small-N designs
This does NOT imply that they are bad designs
Just remember the advantages and disadvantages of each
Non-Experimental designs

5. Surveys What are survey methods? Why conduct them? Instruments used
Questionnaires and interviews that ask people to provide information about themselves
Why conduct them?
Best way to collect some kinds of information:
Descriptive, behavioral, and preferential
e.g., demographic information, recreational behavior, personal preferences, and attitudes
To compliment experimental work
Good/common first step, can collect a lot of data about a lot of variables
Do not have to directly observe behaviors
Note: these instruments can be used as DVs in many research designs (observational, experimental)
A good resource:
Check out the Methodology Pages
Surveys

6. Surveys Advantages One can investigate internal events
e.g., attitudes & opinions
Can generalize about an entire population based on relatively small samples of individuals
Large amounts of data can be collected quickly with relatively little cost (effort, time, etc.)
But they’re often not as “cheap” as you may think
Surveys

7. Surveys Disadvantages Correlational: causal claims shouldn’t be made
Often used for descriptive and predictive research questions
Note: Survey instruments are sometimes used as part of an experimental design, which may allow testing of causal claims
Non-response bias
Why doesn’t everybody respond?
Does response rate interact with variables of interest?
Self-reports may not be truthful
Response set - tendency to respond from a particular perspective
Social desirability bias (e.g., how a “moral” person would answer)
Large data sets are sometimes difficult to analyze
Surveys

8. 11 Stages of survey research
Stage 1) Identify the focus of the study and select your research method
What are the objectives of the research?
Is a survey method the best approach?
What kind of survey should be used?
Group administration
Mail surveys
Internet surveys
Telephone surveys
Face-to-face interviews
Focus group interviews
11 Stages of survey research

9. 11 Stages of survey research
Stage 2) Determining the research schedule and budget
Stage 3) Establishing an information base
Find out what’s been done, what’s known
E.g., Find other related surveys
Stage 4) Identify the sampling frame
The actual population that the sample is drawn from (as opposed to the ideal population)
Think of it as operationalizing the conceptual level population
Be aware of potential coverage error – when the sampling doesn’t lead to a good representativeness
11 Stages of survey research

10. 11 Stages of survey research
Stage 5) Determining the sample method and sampling size
Review Probability and Non-Probability methods
Voluntary response method
Importance of sample size
11 Stages of survey research

11. Voluntary response methods
A kind of convenience sampling methods commonly used
Should leftover Easter candy be given out
to students who get an A on the exam?
Call 231-NONO if you think NO
Call 231-4YES if you think YES
Problem: Typically only individuals with strong opinions respond, so the results are often extremely biased (A coverage error issue? Non-response issue?)
Daily show clip
(5:30 mins)
Voluntary response methods

12. Importance of sample size
Sampling error - how is the sample different from the population?
Often focus on this part
But this part is important too
Confidence intervals
An estimate of the mean or percentage of the population, based on the sample data
“John Doe has 55% of the vote, with a margin of error ± 3%”
Margin of error (that “± 3%” part)
The larger your sample size, the smaller your margin of error will be.
Which would you be more likely to believe
“We asked 10 people …”
“We asked 1000 people …”
Importance of sample size

13. Importance of sample size
Sampling error - how is the sample different from the population?
Response rate
What proportion of the sample actually responded to the survey?
Hidden costs here - what can you do to increase response rates
Non-response error (bias)
Is there something special about the data that you’re missing (From the people who didn’t respond)?
Importance of sample size

14. 11 Stages of survey research
Stage 6) Designing the survey instrument
Question construction: How the questions are written is very important
Clearly identify the research objectives
Do your questions really target those research objectives (think Internal and External Validity)?
Take care wording of the questions
Keep it simple, don’t ask two things at once, avoid loaded or biased questions, etc.
How should questions be answered (question type)?
11 Stages of survey research

15. Good and poor questions
Was the FDC negligent by ignoring the
warnings about Vioxx during testing
and approving it for sale?
Yes No
Unsure
Do you favor eliminating the wasteful
excess in the public school budget?
If the FDC knew that Vioxx caused serious side effects during testing,
what should it have done?
Ban it from ever being sold
Require more testing before approving it
Unsure
Do you favor reducing the public school budget?
Yes No
Problem: emotionally
charged words
Good and poor questions

16. Good and poor questions
Should senior citizens be given more
money for recreation centers and
food assistance programs?
Yes No
Unsure
Should senior citizens be given more money for recreation centers?
Yes No
Unsure
Should senior citizens be given more money for food assistance programs?
Problem: asks two
different questions
Good and poor questions

17. Good and poor questions
Are you against same sex marriage
and in favor of a constitutional
amendment to ban it?
Yes No
Unsure
What is your view on same sex marriage?
I think marriage is a matter of personal choice
I’m against it but don’t want a constitutional amendment
I want a constitutional amendment banning it
Problem: Biased in
more than one direction
Problem: Asks two
questions
Good and poor questions

18. Survey Questions Question types
Open-ended (fill in the blank, short answer)
Can get a lot of information, but
Coding is time intensive and potentially ambiguous
Close-ended (pick best answer, pick all that apply)
Easier to code
Same response alternatives for everyone
Take care with your labels
Decide what kind of scale
Decide number/label of response alternatives
What is the best thing about ISU?
What is the best thing about ISU? (choose one)
1. Location
2. Academics
3. Dorm food
4. People who sell things between Milner and the Bone
Survey Questions

19. Survey Questions: Close-ended
Decide what kind of rating scales
Rating:
e.g., Likert scale
PSY 231 is an important course in the major.
Strongly Agree Neutral Disagree Strongly
Agree Disagree
Semantic differential:
Rate how you feel about PSY 231 on these dimensions
Important _____: _____: _____: _____: _____: Unimportant
Boring _____: _____: _____: _____: _____: Interesting
Nonverbal scale for children:
Point to the face that shows how you feel about the toy.
Survey Questions: Close-ended

20. Survey Questions: Close-ended
Decide number/label of response alternatives
Use odd number (mid point and equal # of responses above and below the mid point)
Questions should be uni-dimensional (each concerned with only one thing)
Labels should be clear
Survey Questions: Close-ended

21. 11 Stages of survey research
Stage 7) Pre-testing the survey instrument
Fix what doesn’t seem to be working
Stage 8) Selecting and training interviewers
For telephone and in-person surveys
Need to avoid interviewer bias
Stage 9) Implementing the survey
Stage 10) Coding and entering the data
Stage 11) Analyzing the data and preparing a final report
11 Stages of survey research