Non-Experimental designs: Surveys & Quasi-Experiments Psych 231: Research Methods in Psychology

Lab attendance is critical this week because group projects are being administered Attendance will be taken. Turn in the group project rating sheet 1 after you have run your experiments (Monday is okay) Announcements

Non-Experimental designs Sometimes you just can’t perform a fully controlled experiment Because of the issue of interest Limited resources (not enough subjects, observations are too costly, etc). Surveys Quasi-Experiments Developmental designs Small-N designs This does NOT imply that they are bad designs Just remember the advantages and disadvantages of each Non-Experimental designs

Surveys What are they? Why conduct them? Questionnaires and interviews that ask people to provide information about themselves Why conduct them? To compliment experimental work Good/common first step, can collect a lot of data about a lot of variables Do not have to directly observe behaviors Best way to collect some kinds of information: Descriptive, behavioral, and preferential e.g., demographic information, recreational behavior, and attitudes Surveys

Advantages One can investigate internal events (for example, attitudes & opinions) Can generalize about an entire population based on relatively small samples of individuals Large amounts of data can be collected quickly with relatively little cost (effort, time, etc.) But they’re often not as “cheap” as you may think Surveys

Surveys Disadvantages Correlational: causal claims shouldn’t be made Non-response bias Why doesn’t everybody respond? Does response rate interact with variables of interest? Large data sets are sometimes difficult to analyze Self-reports may not be truthful Response set - tendency to respond from a particular perspective (e.g., how a “moral” person would answer) Surveys

Stages of survey research Stage 1) Identify the focus of the study and select your research method What are the objectives of the research? Is a survey method the best approach? What kind of survey should be used? Group administration Mail surveys Internet surveys Telephone surveys Face-to-face interviews Focus group interviews Stages of survey research

Stages of survey research cont. Stage 2) Determining the research schedule and budget Stage 3) Establishing an information base Find out what’s been done, what’s known E.g., Find other related surveys Stage 4) Identify the sampling frame The actual population that the sample is drawn from (as opposed to the ideal population) Think of it as operationalizing the conceptual level population Stages of survey research cont.

Stages of survey research cont. Stage 5) Determining the sample method and sampling size Review Probability and Non-Probability methods Voluntary response method Importance of sample size Stages of survey research cont.

Voluntary response methods A kind of convenience sampling methods commonly used Should leftover Halloween candy be given out to students who get an A on the exam? Call 123-NONO if you think NO Call 123-4YES if you think YES Problem: Typically only individuals with strong opinions respond, so the results are often extremely biased Voluntary response methods

Importance of sample size Sampling error - how is the sample different from the population? Confidence intervals An estimate of the mean or percentage of the population, based on the sample data “John Doe has 55% of the vote, with a margin of error ± 3%” Margin of error (that “± 3%” part) The larger your sample size, the smaller your margin of error will be. Which would you be more likely to believe “We asked 10 people …” “We asked 1000 people …” Importance of sample size

Importance of sample size Sampling error - how is the sample different from the population? Response rate What proportion of the sample actually responded to the survey? Hidden costs here - what can you do to increase response rates Non-response error (bias) Is there something special about the data that you’re missing (From the people who didn’t respond)? Importance of sample size

Stages of survey research cont. Stage 6) Designing the survey instrument Question construction: How the questions are written is very important Clearly identify the research objectives Do your questions really target those research objectives (think Internal and External Validity)? Take care wording of the questions Keep it simple, don’t ask two things at once, avoid loaded or biased questions, etc. How should questions be answered (question type)? Stages of survey research cont.

Good and poor questions Was the FDC negligent by ignoring the warnings about Vioxx during testing and approving it for sale? Yes No Unsure If the FDC knew that Vioxx caused serious side effects during testing, what should it have done? Ban it from ever being sold Require more testing before approving it Unsure Problem: emotionally charged words Good and poor questions

Good and poor questions Are you against same sex marriage and in favor of a constitutional amendment to ban it? Yes No Unsure What is your view on same sex marriage? I think marriage is a matter of personal choice I’m against it but don’t want a constitutional amendment I want a constitutional amendment banning it Problem: Biased in more than one direction Problem: Asks two questions Good and poor questions

Survey Questions Question types Open-ended (fill in the blank, short answer) Can get a lot of information, but Coding is time intensive and potentially ambiguous Close-ended (pick best answer, pick all that apply) Easier to code Response alternatives are the same for everyone Rating scales Used for “how much” judgments e.g., measures attitudes, agree/disagree Take care with your labels Range of scores, anchors Survey Questions

Stages of survey research cont. Stage 7) Pre-testing the survey instrument Fix what doesn’t seem to be working Stage 8) Selecting and training interviewers For telephone and in-person surveys Need to avoid interviewer bias Stage 9) Implementing the survey Stage 10) Coding and entering the data Stage 11) Analyzing the data and preparing a final report Stages of survey research cont.

Quasi-experiments What are they? General types Almost “true” experiments, but with an inherent confounding variable General types An event occurs that the experimenter doesn’t manipulate Something not under the experimenter’s control (e.g., flashbulb memories for traumatic events) Interested in subject variables high vs. low IQ, males vs. females Time is used as a variable Quasi-experiments

Quasi-experiments Advantages Disadvantages Allows applied research when experiments not possible Threats to internal validity can be assessed (sometimes) Disadvantages Threats to internal validity may exist Designs are more complex than traditional experiments Statistical analysis can be difficult Most statistical analyses assume randomness Quasi-experiments

Quasi-experiments Program evaluation Research on programs that is implemented to achieve some positive effect on a group of individuals. e.g., does abstinence from sex program work in schools Steps in program evaluation Needs assessment - is there a problem? Program theory assessment - does program address the needs? Process evaluation - does it reach the target population? Is it being run correctly? Outcome evaluation - are the intended outcomes being realized? Efficiency assessment- was it “worth” it? The the benefits worth the costs? Quasi-experiments

Quasi-experiments Nonequivalent control group designs with pretest and posttest (most common) (think back to the second control lecture) participants Experimental group Control Measure Non-Random Assignment Independent Variable Dependent Variable But remember that the results may be compromised because of the nonequivalent control group (review threats to internal validity) Quasi-experiments

Quasi-experiments Interrupted time series designs Observe a single group multiple times prior to and after a treatment Obs Obs Obs Obs Treatment Obs Obs Obs Obs Look for an instantaneous, permanent change Variations of basic time series design Addition of a nonequivalent no-treatment control group time series O O O T O O O & O O O _ O O O Interrupted time series with removed treatment If treatment effect is reversible Quasi-experiments

Next time Go to labs this week, attendance will be taken Non experimental designs cont. Reminder, journal summary 2 assignment is coming up Next time