© 2009 by The McGraw-Hill Companies, Inc. Research Methods in Psychology Survey Research

© 2009 by The McGraw-Hill Companies, Inc. Survey Research  Survey results describe  opinions, attitudes, preferences allow predictions about behavior  Survey research uses questionnaires  predetermined set of questions a sample represents a population

© 2009 by The McGraw-Hill Companies, Inc. Survey Research, continued  Surveys can be limited, specific in scope more global in their goals  Are surveys always biased? Don’t assume bias just because a specific organization or company has sponsored the survey Examine the survey procedures and analyses

© 2009 by The McGraw-Hill Companies, Inc. Correlational Research  Assess relationships among naturally occurring variables for example: attitudes, preferences, personality traits, feelings, age, sex  Correlation coefficients strength and direction of predictive relationship between two variables – negativenopositive relationship

© 2009 by The McGraw-Hill Companies, Inc. Correlational Research, continued  Do these relationships indicate positive or negative correlations? As the number of years in which individuals smoke cigarettes increases, the likelihood of lung cancer increases As the frequency of participating in volunteer activities increases, occasions of depressed mood decrease As arousal level increases, the likelihood of retaliation following an offense increases

© 2009 by The McGraw-Hill Companies, Inc. Obtaining a Sample  Researchers are not interested simply in the responses of those who complete a survey They seek to describe the larger population from which the survey was drawn  Careful selection of a sample allows researchers to generalize findings from the sample to the population

© 2009 by The McGraw-Hill Companies, Inc. Basic Terms of Sampling  Population set of all cases of interest examples:  current students at your school  current residents of your city  citizens of the United States  Sampling Frame list of the members of a population  example: registrar’s list of registered students

© 2009 by The McGraw-Hill Companies, Inc. Basic Terms of Sampling, continued  Sample subset of the population used to represent the entire population  example: students in this class as a sample of all students at this school (or this city)‏  Element each member of the population

© 2009 by The McGraw-Hill Companies, Inc. Goal of Sampling  sample should represent the population characteristics of participants in sample should be similar to those of the entire population  example: Which sample represents a population that is 30% freshmen, 30% sophomore, 20% junior, 20% senior? Sample ASample B 30 freshmen, 30 sophomores,60 freshmen, 60 sophomores, 20 juniors, 20 seniors40 juniors, 40 seniors

© 2009 by The McGraw-Hill Companies, Inc. Biased Samples  A biased sample occurs when characteristics of the sample differ systematically from those of the population samples can overrepresent or underrepresent a segment of a population  samples made up of psychology students overrepresent college students and underrepresent people not in college  most research underrepresents individuals from diverse cultures

© 2009 by The McGraw-Hill Companies, Inc. Biased Samples  Two sources Selection bias  occurs when a researcher’s procedures for selecting a sample result in one or more segments of the population being under- or overrepresented Response-rate bias  occurs when individuals selected for the initial sample do not complete and return the survey lack of interest, worried about privacy, don’t have time

© 2009 by The McGraw-Hill Companies, Inc. Approaches to Sampling  “Sampling” refers to procedures used to obtain a sample  Two basic approaches: nonprobability sampling probability sampling

© 2009 by The McGraw-Hill Companies, Inc. Approaches to Sampling, continued  Nonprobability sampling no guarantee each member of population has an equal chance to be in sample “convenience sampling”  researcher selects individuals who are available and willing to respond to the survey  example: magazine surveys, call-in radio surveys

© 2009 by The McGraw-Hill Companies, Inc. Approaches to Sampling, continued  Probability sampling all members of population have an equal chance of being selected for the survey “simple random sample” Two common methods:  choose randomly from the sampling frame of the population  random-digit dialing

© 2009 by The McGraw-Hill Companies, Inc. Approaches to Sampling, continued  Probability sampling Stratified random sample: divide population into subpopulations, called strata Draw random samples from the strata  best to select samples proportional to the strata size Stratified random sampling increases the likelihood the sample will represent the population

© 2009 by The McGraw-Hill Companies, Inc. Survey Methods  Four methods for obtaining survey data: mail surveys personal interviews telephone interviews Internet surveys  Each method has advantages and disadvantages  Choose method based on research question

© 2009 by The McGraw-Hill Companies, Inc. Survey Methods, continued  Mail surveys quick, convenient, self-administered, best for highly personal or embarrassing topics problem of response rate:  people selected for sample don’t complete and return the survey  final sample may be biased—not representative of the population little control over how people respond to the questions

© 2009 by The McGraw-Hill Companies, Inc. Survey Methods, continued  Personal interviews researchers gain more control over how survey is administered interviewers can seek clarification of answers, ask questions  potential problem: interviewer bias occurs when interviewers record only selected portions of answers or changes wording of questions and answers. interviews are costly; interviewers must be highly motivated, carefully trained, supervised

© 2009 by The McGraw-Hill Companies, Inc. Survey Methods, continued  Telephone interviews complete brief surveys efficiently and with greater access to population random-digit dialing to select random samples supervise interviewers easily Problems  selection bias: no phone or multiple phone numbers  response-rate bias: willingness to answer questions on phone  interviewer bias: changes in survey questions and responses

© 2009 by The McGraw-Hill Companies, Inc. Survey Methods, continued  Internet surveys efficient, low-cost, potential for very large samples samples can be very diverse and access typically underrepresented samples problems:  selection bias: access to Internet  response-rate bias: willingness to respond  lack of control over research environment

© 2009 by The McGraw-Hill Companies, Inc. Survey Methods, continued  Ways to increase response rate questionnaire has a “personal touch”  use name, not “Dear student” responding requires minimal effort topic of survey is interesting to respondents respondents identify with organization or sponsor of survey

© 2009 by The McGraw-Hill Companies, Inc. Survey Research Designs  “Research design” a plan for conducting a research project choose research method best suited for answering a particular question  Three types of survey designs cross-sectional design successive independent samples design longitudinal design

© 2009 by The McGraw-Hill Companies, Inc. Survey Research Designs, continued  Cross-sectional survey design select sample from one or more populations at one time  choose population of interest  use probability sampling or convenience sampling probability sampling leads to a more representative sample  respondents complete a survey

© 2009 by The McGraw-Hill Companies, Inc. Survey Research Designs, continued  Cross-sectional survey design Survey responses are used to  describe population (descriptive statistics)‏  make predictions for the population (correlations)‏ at that one point in time If samples are drawn from different populations, compare the populations cannot assess change over time with cross- sectional designs

© 2009 by The McGraw-Hill Companies, Inc. Survey Research Designs, continued  Successive independent samples design a series of cross-sectional designs over time a different sample from the population completes the survey each time each sample is selected from the same population responses from each sample are used to describe the population at each point in time

© 2009 by The McGraw-Hill Companies, Inc. Survey Research Designs, continued  Successive independent samples design compare survey responses from each sample to see how the population changes over time cannot determine whether individuals change over time Problem of noncomparable samples  If different populations are sampled each time, responses may differ because of true changes over time or because different populations were sampled

© 2009 by The McGraw-Hill Companies, Inc. Survey Research Designs, continued  Longitudinal survey design same sample of individuals completes the survey at different points in time can assess how individuals change over time responses from the sample are generalized to describe changes over time in the population

© 2009 by The McGraw-Hill Companies, Inc. Survey Research Designs, continued  Longitudinal survey design: Problems Attrition: people drop out of the study  sample no longer represents population from which it was selected Reactivity: respondents may strive to be consistent over time or become sensitized to the topic longitudinal surveys can’t tell why people change over time (only correlations)‏

© 2009 by The McGraw-Hill Companies, Inc. Measures in Survey Research  Questionnaires most frequently used to collect survey data measure different types of variables  demographic variables using checklists  preferences and attitudes self-report scales respond using rating scales (assume interval level of measurement)‏

© 2009 by The McGraw-Hill Companies, Inc. Measures in Survey Research, continued  All measures must be reliable and valid  Reliability refers to consistency of measurement Test-retest reliability  administer measure two times to same sample  individuals’ scores should be consistent over time  a high correlation between the two sets of scores indicates good test-retest reliability (r >.80)‏  individuals’ scores need not be identical each time, only same place in the distribution of scores

© 2009 by The McGraw-Hill Companies, Inc. Measures in Survey Research, continued  How to improve reliability? more items greater variability among individuals on the factor being measured testing situation free of distractions clear instructions  A measure can be reliable but not valid

© 2009 by The McGraw-Hill Companies, Inc. Measures in Survey Research, continued  Validity refers to the truthfulness of a measure  A valid measure assesses what it is intended to measure Construct validity refers to whether an instrument measures the theoretical construct it was designed to measure

© 2009 by The McGraw-Hill Companies, Inc. Measures in Survey Research, continued  Example of construct validity: Intelligence: Do these questions from a common measure of intelligence assess a person’s intelligence in a valid manner?  comprehension: “Why would people use a secret ballot?”  vocabulary: “What does dilatory mean?”  similarities: “How are a telephone and a radio alike?”

© 2009 by The McGraw-Hill Companies, Inc. Measures in Survey Research, continued  Establishing construct validity: convergent validity  extent to which two measures of the same construct are correlated (go together)‏ discriminant validity  extent to which two measures of different constructs are not correlated (do not go together)‏

© 2009 by The McGraw-Hill Companies, Inc. Measures in Survey Research, continued  Construct validity example new measure of self-esteem Which constructs should show convergent validity with self-esteem measure and which would show discriminant validity? Constructs: depression, well-being, intelligence, extraversion, age, sensation- seeking, social anxiety, life satisfaction, grade point average, reading comprehension, artistic ability

© 2009 by The McGraw-Hill Companies, Inc. Measures in Survey Research, continued  Correlations demonstrating construct validity and reliability are shown in a correlation matrix: (1)(2)(3)(4)‏ (1) new self-esteem, Time (2) new self-esteem, Time (3) measure of positive affect (4) measure of artistic ability test-retest reliability convergent validity discriminant validity

© 2009 by The McGraw-Hill Companies, Inc. Constructing a Questionnaire  Best choice for selecting a measure use measure already shown to be reliable and valid in previous research  If no suitable measure is found create a questionnaire or measure  Creating a reliable and valid questionnaire is not easy

© 2009 by The McGraw-Hill Companies, Inc. Constructing a Questionnaire, continued  Important first steps Decide what information should be sought Decide what type of questionnaire should be used (e.g., self-administered?)‏ Write a first draft of the questionnaire Have experts review questionnaire and then revise it based on their suggestions Pretest the questionnaire using sample and conditions similar to the planned administration of the survey Review results and edit the questionnaire

© 2009 by The McGraw-Hill Companies, Inc. Constructing a Questionnaire, continued  Next steps: Establish reliability and validity Reliability  Test and retest questionnaire using sample and conditions similar to planned survey. Validity  Convergent: Administer questionnaire with measures of theoretically related constructs  Discriminant: Administer questionnaire with measures of theoretically unrelated constructs

© 2009 by The McGraw-Hill Companies, Inc. Constructing a Questionnaire, continued  Guidelines for Writing Survey Questions Choose how participants will respond  free-response (open-ended)‏ greater flexibility in responses difficult to code  closed-response (multiple-choice, true-false)‏ responses are quick and easy easy to score may not accurately describe individuals’ responses Use simple, familiar vocabulary; keep questions short

© 2009 by The McGraw-Hill Companies, Inc. Constructing a Questionnaire, continued  Guidelines for Writing Survey Questions Write clear and specific questions  avoid double-barreled questions  place conditional phrases at the beginning of sentences  avoid leading questions  avoid loaded (emotion-laded) questions

© 2009 by The McGraw-Hill Companies, Inc. Constructing a Questionnaire, continued  Ordering of questions self-administered questionnaires  most interesting questions first personal and telephone interviews  demographic questions first use funnel questions  start with general questions and move to more specific questions on a given topic use filter questions  direct respondents to appropriate questions

© 2009 by The McGraw-Hill Companies, Inc. Thinking Critically About Survey Research  Reported vs. Actual Behavior Survey responses may not be truthful  Reactivity not truthful because the information is being recorded  Social desirability not truthful because responding as they “should”

© 2009 by The McGraw-Hill Companies, Inc. Thinking Critically About Survey Research, continued  Measuring social desirability sample items from Marlowe-Crowne (1964) Social Desirability Scale What is the socially desirable response? 1. No matter who I’m talking to, I’m always a good listenerT F 2.I like to gossip at timesT F 3.I’m always willing to admit it when I make a mistakeT F 4.I have almost never felt the urge to tell someone offT F

© 2009 by The McGraw-Hill Companies, Inc. Thinking Critically About Survey Research, continued  Guidelines for social desirability Accept people’s responses as truthful unless there’s reason to suspect otherwise Because actual behavior doesn’t always match questionnaire responses, use a multimethod approach to answering research questions

© 2009 by The McGraw-Hill Companies, Inc. Thinking Critically About Survey Research, continued  Correlation and causality “correlation does not imply causality” example:  correlation between being socially active (A) and life satisfaction (B)‏  three possible causal relationships A causes B B causes A variable C causes both A and B (e.g., number of friends)‏ = spurious relationship

© 2009 by The McGraw-Hill Companies, Inc. Thinking Critically About Survey Research, continued  Path analysis statistical procedure to tease apart complex correlational relationships among variables Mediators  variables used to explain a correlation between two variables Moderators  variables that affect direction or strength of correlation between two variables

© 2009 by The McGraw-Hill Companies, Inc. Thinking Critically About Survey Research, continued  Example of path analysis Evans et al. (2005) observed a positive correlation between measures of poverty and measures of psychological distress among children Poverty → Psychological distress This is called a direct relationship or path a

© 2009 by The McGraw-Hill Companies, Inc. Thinking Critically About Survey Research, continued  Path analysis example Evans et al. sought to explain why this relationship exists.  level of chaos in the home as possible mediator  chaotic living environment measured using concepts such as unpredictability, confusion, lack of structure, noise, overcrowding, and poor-quality housing

© 2009 by The McGraw-Hill Companies, Inc. Thinking Critically About Survey Research, continued  Path analysis example Evans et al. observed two important correlations  Measures of poverty were positively correlated with greater chaos in the home (Path b)‏  Greater chaos in the home was positively correlated with psychological distress (Path c)‏ These two correlations represent the indirect relationship between poverty and psychological distress

© 2009 by The McGraw-Hill Companies, Inc. Thinking Critically About Survey Research, continued  Path analysis example Diagram of direct and indirect relationships Chaos path bpath c PovertyPsychological path adistress “Chaos” mediates the relationship between poverty and psychological distress

© 2009 by The McGraw-Hill Companies, Inc. Thinking Critically About Survey Research, continued  Path analysis example What if the relationships between poverty, chaos, and psychological distress are not observed for all children? A moderator variable may affect these relationships  moderators affect the direction and strength of relationships

© 2009 by The McGraw-Hill Companies, Inc. Thinking Critically About Survey Research, continued  Path analysis example Possible moderators:  sex of the child the relationships between poverty, chaos, and psychological distress may exist for boys, but not girls  population density the relationships between poverty, chaos, and psychological distress may exist in urban areas, but not rural areas  personality features of children the relationships may exist for low-resilient children, but not high-resilient children

© 2009 by The McGraw-Hill Companies, Inc. Thinking Critically About Survey Research, continued  Path analysis helps us to understand relationships among variables but these relationships are still correlational cannot make definitive causal statements other untested variables are related to children’s psychological distress