The Scientific Study of Politics (POL 51) Professor B. Jones University of California, Davis

Today Sampling Plans Survey Research

Populations Key Concepts Population – Defined by the research – “All U.S. citizens age 18 or older.” – All democratic countries – Counties in the United States Characteristics of a Population – Bounded and definable – If you can’t define the population, you probably don’t have a well formed research question!

Populations vs. Samples Populations are often unattainable – TOO BIG (U.S. population) – Very Costly to Obtain – May not be necessary The beauty of statistical theory Samples – Simply Defined: a subset of the population chosen in some manner – How you choose is the important question!

Moving Parts of a Sample Units of Analysis – J is the population – i is a member of J – Then i is a “sample element” Sampling Frames – The actual source of the data – Literary Digest Poll (1936) – “Dewey Defeats Truman” (1948) – Exit Polls

More Moving Parts Sampling Unit – Could be same as sample element (Unit of Analysis) – But it could be collections of elements (cluster, stratified sampling) Sampling Plan – Random? Nonrandom?

Kinds of Samples Simple Random Sample – Major Characteristic: Every sample element has an equi-probable chance of selection. – If done properly, maximizes the likelihood of a representative sample. What if your assumptions of randomness goes badly? Nonrandom samples (often) produce nonrepresentative surveys.

Why Randomness is Goodness Nonprobability Sampling – Probability of “getting into” the sample is unknown – All bets are off; inference most likely impossible – Highly unreliable! Simple Random Sampling – Every sample element has the same probability of being selected: Pr(selection)=1/N – In practice, not always easy to guarantee or achieve An Example of a Bad Assumption

Some Data

More Data

Getting Probability Samples Wrong

Draft Lottery Simple random sampling did not exist. – Avg. Lottery Number Jan.-June: 206 – Avg. Lottery Number July-Dec.: 161 – Avg. Deaths Jan.-June: 159 – Avg. Deaths July-Dec.: 111 Differences highly significant. Its absence had profound consequences. Randomness should have ensured an equal chance of draft, invariant to birth date. It didn’t. By analogy, suppose college admissions were based on this kind of lottery… Those of you born later in the year would be less likely to be admitted. Would you consider that fair?

How to Achieve Randomness Random number generation – Modern computers are really good at this. – Assign sample elements a number – Generate a random numbers table – Use a decision rule upon which to select sample. The Key: sampled units are randomly drawn. Why Important? Randomness helps ensure REPRESENTATIVENESS! Absent this, all bets are off: – Convenience Polls – Push Polls – Person-on-the-Street Interviews

A Population and Some “Samples” A “Population” – Striations represent “attitudes” Some “Samples”

Other Kinds of Sampling Strategies Stratified Samples: a probability sample in which elements sharing some characteristic are grouped and then sample elements are randomly chosen from each group. Benefit? Can ensure more representative sample with smaller sample sizes. Why might this be the case?

Sampling come to life in…R!!! Suppose we have a population of 100,000 And in that population, we have 4 groups – Group 1: 13,000 (13 percent) – Group 2: 12,000 (12 percent) – Group 3: 4,000 ( 4 percent) – Group 4: 70,000 (70 percent) Racial/Ethnic Characteristics in the US: US Census – White (69.13 percent) – Black (12.06 percent) – Hispanic (12.55 percent) – Asian (3.6 percent) Some R Code

R #Creating a population of 100,000 consisting of 4 groups set.seed( ) population<- rep(1:4,c(13000, 12000, 4000, 70000)) #Tabulating the population (ctab requires package catspec) ctab(table(population)) #Tabulating the population (ctab requires package catspec) (btw, not sure why percents are not whole numbers) ctab(table(population)) Count Total % population

Sampling What do we expect from random sampling? That each sample reproduces the population proportions. Let’s consider SIMPLE RANDOM SAMPLES. Also, let’s consider small samples (size 100) …which is a.001 percent sample.

R: 3 samples of n=100 #Three Simple Random Samples without Replacement; n=100 which is a.001 percent sample #The set.seed command ensures I can exactly replicate the simulations set.seed(15233) srs1<-sample(population, size=100, replace=FALSE) ctab(table(srs1)) set.seed( ) srs2<-sample(population, size=100, replace=FALSE) ctab(table(srs2)) set.seed(5255) srs3<-sample(population, size=100, replace=FALSE) ctab(table(srs3))

R: Sample Results > set.seed(15233) > srs1<-sample(population, size=100, replace=FALSE) > ctab(table(srs1)) Count Total % srs > set.seed( ) > srs2<-sample(population, size=100, replace=FALSE) > ctab(table(srs2)) Count Total % srs > set.seed(5255) > srs3<-sample(population, size=100, replace=FALSE) > ctab(table(srs3)) Count Total % srs

Implications? Small samples? Variability in proportion of groups. Why does this occur? Let’s understand stratification. What does it do? You’re sampling within strata. Suppose we know the population proportions?

R: Identifying Strata and then Sampling from them. #Stratified Sampling #Creating the Groupings strata1<- rep(1,c(13000)) strata2<- rep(1,c(12000)) strata3<- rep(1,c(4000)) strata4<- rep(1,c(70000)) #Sampling by strata #Selection observations proportional to known population values: Proportionate Sampling set.seed( ) srs4<-sample(strata1, size=13, replace=FALSE) ctab(table(srs4)) set.seed( ) srs5<-sample(strata2, size=12, replace=FALSE) ctab(table(srs5)) set.seed(33325) srs6<-sample(strata3, size=4, replace=FALSE) ctab(table(srs6)) set.seed( ) srs7<-sample(strata4, size=70, replace=FALSE) ctab(table(srs7))

R: Results? Proportional Sampling w/small samples. > srs4<-sample(strata1, size=13, replace=FALSE) > ctab(table(srs4)) Count Total % srs > > set.seed( ) > srs5<-sample(strata2, size=12, replace=FALSE) > ctab(table(srs5)) Count Total % srs > > set.seed(33325) > srs6<-sample(strata3, size=4, replace=FALSE) > ctab(table(srs6)) Count Total % srs > > set.seed( ) > srs7<-sample(strata4, size=70, replace=FALSE) > ctab(table(srs7)) Count Total % srs

Proportionate Sampling What do we see? If we know the proportions of the relevant stratification variable(s)… Then sample from the groups. SMALL SAMPLES can reproduce certain characteristics of the sample. But of course, it is probabilistic.

Disproportionate Sampling Why? “Oversampling” may be of interest when research centers on small pockets in the population. Race is often an issue in this context.

R: Disproportionate Sampling > #Sampling by strata > #Selection observations disproportional to known population values: disproportionate Sampling > #"Oversampling by Race" > set.seed( ) > srs8<-sample(strata1, size=24, replace=FALSE) > ctab(table(srs8)) Count Total % srs > > set.seed( ) > srs9<-sample(strata2, size=22, replace=FALSE) > ctab(table(srs9)) Count Total % srs > > set.seed(103325) > srs10<-sample(strata3, size=14, replace=FALSE) > ctab(table(srs10)) Count Total % srs > > set.seed(11534) > srs11<-sample(strata4, size=70, replace=FALSE) > ctab(table(srs7)) Count Total % srs >

Disproportionate Samples What did I ask R to do? I “oversampled” for some groups. Again, understand why we, as researchers, might want to do this.

Side-trip: Sample Sizes Who is happy with a.001 percent SRS? On the other hand… What do we get from a stratified sample? Suppose we increase n in a SRS? It’s R time!

R: SRS with a 1 percent sample > #Sample Size=1000 > > set.seed( ) > srs1<-sample(population, size=1000, replace=FALSE) > ctab(table(srs1)) Count Total % srs > > set.seed( ) > srs2<-sample(population, size=1000, replace=FALSE) > ctab(table(srs2)) Count Total % srs > > set.seed(52909) > srs3<-sample(population, size=1000, replace=FALSE) > ctab(table(srs3)) Count Total % srs >

Implications? Sample Size MATTERS What do we see? Note, again, what stratification “buys” us. The issues with stratification? Another R example (code posted on website)

R We have again 4 sample elements > set.seed(52352) > urn<-sample(c(1,2,3,4),size=1000, replace=TRUE) > > ctab(table(urn)) Count Total % urn  My Population

R version of a person-on-the-street interview > #Convenience Sample: What shows up > > con<-matrixurn[1:10]; con [1] > > ctab(table(con)) Count Total % con

R and Samples, redux What do we find? Very unreliable sample: we oversample some groups, undersample others. Useless data more than likely. What do you imagine happens when we increase the sample sizes?

R and SRS with samples of size N /*Sample: Sizes 10, 50, 75, 100, 200, 250, 900, 1000*/ set.seed(562) s1<-sample(urn, 10, replace=FALSE) ctab(table(s1)) set.seed(58862) s1a<-sample(urn, 50, replace=FALSE) ctab(table(s1a)) set.seed(562657) s1b<-sample(urn, 75, replace=FALSE) ctab(table(s1b)) set.seed(58862) s2<-sample(urn, 100, replace=FALSE) ctab(table(s2)) set.seed(58862) s3<-sample(urn, 200, replace=FALSE) ctab(table(s3)) set.seed(10562) s4<-sample(urn, 250, replace=FALSE) ctab(table(s4)) set.seed(22562) s5<-sample(urn, 900, replace=FALSE) ctab(table(s5)) set.seed(56882) s6<-sample(urn, 1000, replace=FALSE) ctab(table(s6))

> /*Sample: Sizes 10, 50, 75, 100, 200, 250, 900, 1000*/ Error: unexpected '/' in "/" > > set.seed(562) > s1<-sample(urn, 10, replace=FALSE) > ctab(table(s1)) Count Total % s > > set.seed(58862) > s1a<-sample(urn, 50, replace=FALSE) > ctab(table(s1a)) Count Total % s1a > Sampling and Sample Size

> > set.seed(562657) > s1b<-sample(urn, 75, replace=FALSE) > ctab(table(s1b)) Count Total % s1b > > set.seed(58862) > s2<-sample(urn, 100, replace=FALSE) > ctab(table(s2)) Count Total % s > Sample Sizes

> set.seed(58862) > s3<-sample(urn, 200, replace=FALSE) > ctab(table(s3)) Count Total % s > > set.seed(10562) > s4<-sample(urn, 250, replace=FALSE) > ctab(table(s4)) Count Total % s > Sample Size

> set.seed(22562) > s5<-sample(urn, 900, replace=FALSE) > ctab(table(s5)) Count Total % s > > set.seed(56882) > s6<-sample(urn, 1000, replace=FALSE) > ctab(table(s6)) Count Total % s >

R: What did we learn? Sample size seems to have some impact here. But there are trade-offs.

Important Moving Parts Randomness (covered!) Sampling Frame – Random sampling from a bad sampling frame produces bad samples. Sample Size – What is your intuition about sample sizes? Must they always be large? – Not necessarily so…although…

Bad Sampling Person-on-the-Street Interviews What do these imply? Small samples and inherently nonrandom Likely poor inference. Other examples? Not all non-random samples are necessarily bad Purposive Samples