1. – Sample & Surveys Notes

2. Sample Vocabulary
Population - the entire group of individuals or instances about which we hope to learn
Sample - a representative subset of a population
Sample Survey - descriptive study that asks questions of a sample in hope of learning something about the whole population
Randomization - process by which each individual is given a fair and equal chance of selection for the sample (Best Defense Against Bias)
Sample size - the number of individuals in a sample

3. Survey Vocabulary
Closed-options question- a questions that limits your choices
Ex: What is your favorite class, Math or Science?
Open-option question- a question that allows you to answer freely
Ex: What types of books do you like to read?
Bias - any failure to accurately represent the population in a sample
Ex: Did you hate that movie as much as I did?
Fair question- a question that shows no favoritism
What did you think of that movie?

4. Understanding Samples
Samples are used to “stretch” beyond the data at hand to the entire world or group at large
There are three necessary ideas in order to make this “stretch” or draw this conclusion (ERS)
Examine a part of the whole
Randomize
Sample size

5. Understanding Samples Examining a part of the whole
Researchers often want to know about an entire population but surveying an entire population is often impractical or impossible, therefore …
Researchers often settle for creating a representative sub-group or “sample” from the population

6. Step 1: Sample Surveys
Sample Surveys - Designed to ask questions of a small group of people in order to learn something about the entire population
Sample surveys are everywhere
National polls
Newspaper polls
Internet electronic polls

7. Sample Surveys How can sample surveys truly represent a population?
In order to understand this, let’s look first at a failed sample survey
In 1936, the Literary Digest magazine held a mock election poll with its readers.
The magazine used telephone numbers in order to select a sample of the population.
According to the survey, Alf Landon received 57% of the votes beating F.D. Roosevelt (43%) in a landslide.
When the real election was held, FDR won 62% to 32%

8. What Went Wrong? The magazine used a “biased” sample.
In 1936, the telephone was a luxury afforded only by the affluent so the sample inadvertently was composed of only wealthy individuals.
Roosevelt’s was extremely popular among the less affluent, therefore the sample used under- represented FDR’s support.
How can researchers eliminate biased representation in samples?
The best strategy is ….

9. Bias
What does it mean if someone calls you biased? Bias occurs in statistics if data is skewed by factors that make it inaccurate.

10. Selection Bias
Selection bias involves the way people are chosen for a survey.
Undercoverage - some members of the population are inadequately represented in the sample
Nonresponse bias - individuals chosen for the sample are unwilling or unable to participate in the survey
Be sure to ask for examples.

11. Response Bias
Response bias involves the way people respond to a survey.
Voluntary response bias - sample members are self-selected volunteers
Leading questions - wording of the question may be loaded in some way to unfairly favor one response over another

12. Let’s discuss!
In 1992, a Roper poll conducted for the American Jewish Community of the Holocaust asked:
“Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?”

13. The use of double negatives in this question caused confusion in the way people responded to the survey. 22% of those surveyed said that it was possible that the holocaust did not occur.
This is an example of a leading question!

14. Later, a new survey was conducted in which the question was rephrased:
“Does is seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened?”
In the new survey, only 1% of those surveyed stated that it was possible that the holocaust never occurred.

15. Social Desirability
Most people like to present themselves in a favorable light, so they’ll lie to retain their social desirability.
People won’t admit to racist attitudes, illegal activities, or unpopular opinions, especially if a survey is public.

16. 3. Voluntary Response Bias
Selection Bias
1. Undercoverage
2. Non-response Bias
Response Bias
3. Voluntary Response Bias
4. Leading Questions
5. Social Desirability
Recap: Bias

17. STEP 2. Randomize
Randomization is the best statistical weapon against sampling bias. Why?
it protects researchers by making sure that on average that sample looks like the rest of the population.
Populations have various features that may influence the validity of the findings, sometimes even features that researchers haven’t thought about. Randomization accounts for this by giving every one an equal chance of selection and representation in the sample.
it also allows researchers to make inferences from their sample to the population from which it was drawn because the sample represents the population accurately.

18. Example 2: What kind of bias?
A uniformed policeman interviews a group of high school students. He asks for the student’s name and then asks if the student has used drugs in the last 30 days.
A study on coronary problems was conducted using Duke University students.
A survey about cafeteria food was conducted by leaving forms at the cash register.

19. Random Sampling
To guard against bias, we use random sampling, which simply means that the sample being surveyed is based on chance.

20. STEP 3. Sample Size
The fraction of the population that you’ve sampled does NOT matter! The only thing that is important is the sample size itself!
Samples need to be representative
In order to see the proportion of a population that fall into a category, it is necessary to see several respondents in each category in order to say anything precise enough to be useful. (usually several hundred respondents)

21. Census
Hey! Wouldn’t it be easier to just survey the whole population, then there is no need to worry about any of the sampling stuff. Right??
NO! A census may appear to really represent the population but it may actually not for three main reasons.

22. A Census Doesn’t Make Sense
It can be difficult to complete a census
there are always some individuals who are hard to locate (e.g., homeless) or hard to survey (e.g., people with limited ability to communicate)
Populations are always changing
Deaths and births are constantly happening and constantly changing the population
By the time the census is completed, an event could have changed everyone’s opinion regarding the questions in the census.
A census is more complicated than a survey
Census’s often require a team effort and the help of the population being surveyed.

23. Probability Samples Simple Random Sample
gives each combination of people within the population an equal chance to be selected for the survey.
Ex: Pull a name from that hat to win a prize.
However, when we draw a sample at random, each sample will be different. We call these sample to sample differences, or sampling variability.
Stratified Random Sampling
Used when a population is already broken up into groups individuals chosen by common properties
Ex: Two students were chosen at random from each English class to attend the event.

24. Non-Probability Sampling
Not based on random process
There are three types or sample types that can cause bad samples
Convenience
Judgment
Questionnaire

25. Convenience Sample Convenience sample
Decision made by the most convenient method
Only those individuals who are at hand are included
Why is this Bad?
Leads to bias in response because the people at hand often have a common tie and are not representative of the whole population

26. Judgment
Judgment
Decision made by one or more experts using subjective judgment
Ex: Mr. Jones chooses two students to attend the conference
Why is this Bad?
It doesn’t allow for an accurate representation of the population, therefore no accurate predictions or inferences can be made from the data.

27. Questionnaire Sample Voluntary response sample
In this approach, a large group of individuals are eligible to participate but only those who respond to the survey are counted.
Why is this bad?
Leads to a bias because only those who care strongly enough about the survey will respond; therefore, the results from the sample are not representative of the entire population

28. What can go wrong (BIAS)
Nonresponse bias
Nonresponse to surveys can be a source of bias because those who do not respond to a survey could differ from those who do.
To prevent this bias:
Don’t bore people with long surveys
Don’t send out a lot of surveys; send out fewer random surveys in scenarios which you can ensure a high response level

29. What can go wrong (BIAS)
Response Bias
Refers to a bias brought about by survey questions which influences responses
This influence is often referred to as a “leading question”
In leading questions the surveyor uses influential words to “lead” a person to a certain answer.
E.g.
Do you think that the evil companies who destroy animals’ habitats should be allowed to continue destroying the rain forest when they harvest trees?- biased
Do you think companies should be allowed to harvest trees from the rain forest? - not biased

30. Rules for Eliminating Bias in Your Surveys
Look for bias in any survey you encounter
there is no way to recover from a sample or survey that asks biased questions. All of your data becomes useless when you have a biased question included in your survey!
Spend your time and resources reducing biases
If possible, test your survey before you use it
Always report you sampling methods in detail

31. Practice Problems
Consumers Union asked all subscribers whether they had used alternative medical treatments and, if so, whether they had benefited from them. For almost all of the treatments, approx. 20% of those responding reported cures or substantial improvement.
A) Population - All US Adults
B) Sample - those who responded
C) Method - a nonrandom questionnaire
D) Bias - Voluntary response sample causes the bias. Only those who cared strongly enough about the question responded. This sample can not represent the whole population because those who did not respond could have different opinions or answers then those who did respond