Sampling & External Validity 1 Chapter 2 part 2

The 65, 95, 99 Percent Rule 1 2

Estimating the Population Using a Sampling Distribution 1 2

The rest of the slides Types of sampling Probability based 1 Types of sampling Probability based Non-probability based 2

Probability Sampling Utilizes some form of random selection All units in the population have equal probability of being chosen Nomenclature: N = number of cases in the sampling frame n = number of cases in the sample NCn = number of combinations of n from N f = n/N and is the sampling fraction 1 2 3 4

Probability Sampling Simple random sampling Stratified random sampling 1 Simple random sampling Stratified random sampling Systematic random sampling Cluster (area) random sampling Multistage sampling

Probability Sampling: Simple Random Sampling Objective: To select n units out of N such that each NCn has an equal chance of being selected Procedure: Use a table of random numbers or computer random-number generator Example: N = 1000 n (desired) = 100 f = n/N = 100/1000 = .10 or 10% Randomly select 100 units (10%) Generalizable; may not be representative of subgroups 1 2 3

Probability Sampling: Stratified Random Sampling Objective: To select n units out of N such that key subgroups of n are representative of subgroups of N Procedure: Divide the population into nonoverlapping groups (strata) N1, N2, N3…Ni, such that N1 + N2 + N3 +… Ni = N. Then do simple random sample of f = n/N in each strata Disproportionate stratified random sampling can be used to oversample small groups. 1 2 3 4

Probability Sampling: Systematic Random Sampling Objective: To systematically select n units out of N such that n is a random sample of N Procedure: Number units in the population from 1 to N (NOTE: Units must be randomly ordered) Decide on the n that you need Calculate k = N/n = the interval size Randomly select an integer between 1 and k Take every kth unit (diagram on next slide illustrates this) 1 2

Probability Sampling: Systematic Random Sampling 1

Probability Sampling: Cluster (Area) Random Sampling Objective: To obtain a representative sample from N when N is spread out over a large geographic area Procedure: Divide the population into clusters Randomly sample clusters Measure all units within sampled clusters Clusters are usually divided along geographical boundaries. 1 2 3

Probability Sampling: Multistage Sampling Objective: To obtain a representative sample from N by combining several sampling techniques to create a more efficient or effective sample than the use of any one sampling type can achieve on its own Example: 1. National sample of school districts stratified by economics 2. Simple random selection of schools within districts 3. Simple random selection of classes within schools 1 2 3

Nonprobability Sampling Does not involve random selection May be representative but cannot depend upon the rationale of probability theory Used when it is not feasible, practical, or theoretically sensible to use random sampling Accidental versus purposive 1 2 3 4

Nonprobability Sampling: Accidental, Haphazard, or Convenience Sampling One of the most common methods of sampling “Man on the street” Volunteers or subjects who are “conveniently” available No evidence that sample is representative 1 2

Nonprobability Sampling: Purposive Sampling Sampling with a “purpose” in mind Useful in reaching a targeted sample quickly Target population is reached but with over-representation of subgroups that are more readily accessible Types: Modal instance Expert Quota Heterogeneity Snowball 1 2 3

Nonprobability Sampling: Purposive Sampling Modal Instance Sampling the most frequent case or typical case Difficult to define what a “typical case” is Probably only useful for informal sampling contexts (or perhaps even more dangerous for those) 1 2 3

Nonprobability Sampling: Purposive Sampling Expert Sampling Assembling of a sample of persons with known or demonstrable expertise in some area “Panel of experts” May be useful for providing evidence as to the validity of another sampling approach you have chosen 1 2

Nonprobability Sampling: Purposive Sampling Quota Sampling Sample selected nonrandomly according to some fixed quota Proportional quota sampling used to represent the major characteristics of the population of interest by sampling a proportional amount of each Nonproportional quota sampling used to supply a minimum number of units in each category but not concerned with proportions 1 2

Nonprobability Sampling: Purposive Sampling Heterogeneity Sampling Used to provide a sample that will include “all” the view or opinions without regard to proportional representation Sampling for diversity Can be thought of as the “opposite of” modal instance sampling 1 2

Nonprobability Sampling: Purposive Sampling Snowball Sampling People meeting the criteria for inclusion in the sample are identified and then they recommend others they know who meet the criteria Useful when trying to reach inaccessible or hard to find populations Examples may include the homeless, drug users, etc. 1 2

Threats to External Validity Interaction of selection and treatment Maybe it is just these people. Interaction of setting and treatment Maybe it is just these places. Interaction of history and treatment Maybe it is just these times.

Guiding Questions for Critiquing the External Validity of Research What are the main results of the study (e.g., positive or negative relationship, group differences, effectiveness of the intervention or treatment)? Do the researchers explicitly state or imply that similar results would hold for other: (a) people, (b) places or situations, and/or (c) times? If so, what is the population/place/time they are attempting to generalize to? If the researchers are generalizing their results, how reasonable are these conclusions given the sample, sampling procedures, and settings used? [This is the key External Validity question] What specifically might lead you to question these conclusions? In other words, if they did suggest the results were generalizable, why might you think otherwise? [The more convincing of a rationale you can generate, the more you should question the external validity]

Use the guiding questions to evaluate the external validity of the following study Prior research has found that (a) intercollegiate athletes are especially “at-risk” for excessive alcohol consumption (e.g., Nelson & Wechsler, 2001), and (b) sport-type differences exist among college athletes in terms of yearly drinking prevalence rates (National Collegiate Athletic Association, 2001). No studies, however, have examined sport-type differences on more specific measures of alcohol consumption (i.e., drinks per week). In the present study, data were analyzed on 298 intercollegiate athletes from two different NCAA Division III universities. Results indicated significant sport type differences on alcohol consumption variables, with athletes from the sports of swimming and diving and wrestling reporting the highest levels of alcohol consumption (M = 5.20, SD = 4.00) and soccer and football reporting the lowest (M = 4.02, SD = 3.25). Results suggest college athletes participating in individual sports are at-risk for future alcohol abuse.