Who are the participants? Creating a Quality Sample 47:269: Research Methods I Dr. Leonard March 22, 2010

The importance of a research sample It is rarely possible to study an entire population, or all people the research is focused on, so to be more efficient we draw a sample We use findings from the sample to infer conclusions about the population Therefore, the quality of our conclusions about the population depends on how good, or representative our sample is

Population vs. Sample

Selecting an unbiased sample Ideally, all samples would be without bias, meaning any individual in the population has an equal chance of being in the study To do this a researcher needs to have access to every single member of the population, which is unlikely Therefore, most of our samples have built in biases

Picking a sample

Bias in sampling  When a sample does not reflect some of the similarities or differences present in the population  Chance (by accident) OR Selection Bias (due to your sampling methods) Example: I want to test whether or not a new pedestrian assist device will aid blind individuals’ in navigating busy intersections What is my target population? What is my accessible population? How might I select participants to be in the study?

Gold standard: Random Sampling Increases representativeness of population using PROBABILITY  Characteristics of population are known so likelihood of getting each type of individual can be estimated  Each member of the population has an equal chance of being selected into the sample  Each possible sample of a given size (e.g., n=10) has an equal chance of being selected from the population (N=100) We want our samples to reflect all of the similarities and differences found in our target populations  Age  Gender  Ethnicity  Political or Religious Affiliation  Others specific variables of interest  The range of characteristics or variety in the sample is almost never as big as in the population but it should be close

Simple Random sampling:  Each individual has an equal & independent chance of being selected (like drawing names out of a hat) Stratified Random Sampling:  Divide population into strata, or sub-groups, before randomly selecting participants and then draw representative percentage from each strata Systematic Sampling:  Line up the population, randomly select a starting point, and take every nth (lets say 10th) person Cluster Sampling:  Imagine being interested if learning about high-school students’ attitudes towards military service. You are interested in collecting a sample of 500 students.  Instead of randomly sampling to get 500 students, list the 30 schools and randomly select 5 schools.  Then test 100 students from each of those schools. Types of Random/Probability sampling)

Stratified Random Sampling If we wanted sample to represent the SES breakdown of a given population… Consider need for different recruiting methods

Systematic Sampling

Cluster Sampling 100 n = 500

Nonprobability Sampling Methods Characteristics of entire population not known Therefore, probability of selecting certain types of individuals can not be estimated Sampling is nonrandom, but with an effort to maintain representativeness and avoid bias. Types of nonprobability sampling  Convenience (e.g., Use the next ten people through the door)  Quota (e.g., need a certain number of individuals with a specific trait in each group)  Volunteer (e.g., undergraduate psychology majors)  Purposive (e.g., looking for expert informants)  Snowball (e.g., hard time finding participants so ask them to recruit others they know)

Which kind of research is more likely to use random (probability) sampling?  Experimental research (Any scientific study in which the researcher systematically varies one or more variables, holding all others constant, to see if another variable is affected) Why?  Exerts more control than non-experimental  Seeks to rule out extraneous variables & confounds  More manipulation of independent variable  More likely to use groups of participants

Reminders about experimental research Any scientific study in which the researcher systematically varies one or more variables, holding all others constant, to see if another variable is affected Intervention made or treatment given to observe effects (causal -- does X cause changes in Y?) Independent variable must have two or more levels for comparison  Most often accomplished by having experimental group(s) and control group True experiment if assignment of participants to groups is random

Random Sampling vs. Random Assignment Random sampling is concerned with every individual in the population having an equal chance of being in the study Random assignment is concerned with every participant in the sample having an equal chance of being in an experimental group (rules out bias)

True experimental designs (1) Pretest-posttest randomized control group design  Participants randomly assigned to groups  Results in two “equal” or equivalent groups Experimental and control group  Both groups tested, or observed, before treatment (pretest)  Treatment given (independent variable being manipulated)  Both groups tested, or observed, again (posttest)  Any pre-post gains or differences between groups are likely attributable to a) the treatment or b) random error R O X O R O O

True experimental designs (2) Beware pretest sensitization Sometimes the pretest can cause problems if it also affects the groups in addition to the treatment Posttest only randomized control group design  Participants randomly assigned to groups  Results in two “equal” or equivalent groups  Treatment given (independent variable being manipulated)  Both groups tested, or observed, again (posttest)  Any posttest differences between groups are more likely attributable to a) the treatment or b) random error R X O R O

True experimental designs (3) Best of both designs… Solomon randomized four-group design  Still have participants randomly assigned to groups (may need more participants to get equivalent groups)  Like running two experiments at the same time  Can test for pre-posttest gains and control for pretest sensitization R O X O R O O R X O R O

Problems in True Experimental Designs What if the observed change is NOT due to the treatment but due to random error? We may find a design lacks validity (just like measures can lack validity) Experimental validity can be… Internal - the degree to which an experiment’s methods are controlled and free from confounds External - the degree to which findings from research can be generalized to the real world context (other populations, other settings, other times???)

Threats to internal validity of experiment Any factors that lessens the degree of control Pre- and post-test problems  History - events that occur outside of the study that may affect the outcome  Maturation - changes within individual during the study that may affect the outcome  Instrumentation - changes in the measures between pre- and post-tests; could be capturing different construct  Testing - any feature of the test or task that could change the responses the second time; practice effect  Statistical regression to the mean - over time, scores tend to move toward the average

Threats to internal validity of experiment Participant problems - any problems related to how the individuals participating in the study may challenge the validity of the findings; most common:  Selection effects - participants in samples should be equivalent to each other except for the independent variable but human beings are not clones Also, selection into the study should be totally random AND each participant should have an equal chance of being assigned to groups  Morality/Attrition - every single participant may not complete the study, which can be especially problematic if a certain type of participant is more likely to drop out

Threats to external validity of experiment Was our sample good enough that we can we generalize our findings??? Population A Population B Population C Setting A, Time A Setting B, Time B Setting C, Time C Sample