1. Chapter 4 Gathering Data
Section 4.1
Experimental and Observational Studies

2. Observational study vs Experiment
Observational study : the investigator observes individuals and measures variables of interest but does not attempt to influence the response.
Example: Based on observations you make in nature, you suspect that female crickets choose their mates on the basis of their health.  Observe health of male crickets that mated.
Experiment (study) : the investigator observes how a response variable behaves when the researcher manipulates one or more factors.
Example: Deliberately infect some males with intestinal parasites and see whether females tend to choose healthy rather than ill males.

3. Observational Study – Sample Survey
A sample survey selects a sample of people from a population and interviews them to collect data.
A sample survey is a type of observational study.
A census is a survey that attempts to count the number of people in the population and to measure certain characteristics about them.

4. Example: Drug Testing & Student Drug Use
A headline read: “Student Drug Testing Not Effective in Reducing Drug Use” in a news release from the University of Michigan. Facts about the study:
76,000 students nationwide from 497 high schools and 225 middle schools
Schools selected for the study included schools that tested for drugs and schools that did not test for drugs
Each student filled out a questionnaire asking about his/her drug use
Questions:
What were the response and explanatory variables?
Was this an observational study or an experiment?
Response: Drug Use
Explanatory: Student Drug Testing
Answer: 2.This study was an observational study.
In order for it to be an experiment, the researcher would had to have assigned each school to use or not use drug testing rather than leaving this decision to the school.

5. Comparing Experiments & Observational Studies
An experiment reduces the potential for lurking variables to affect the result. Thus, an experiment gives the researcher more control over outside influences.
Only an experiment can establish cause and effect. Observational studies can not.
Experiments are not always possible due to ethical reasons, time considerations and other factors.

6. Example
Study: Do smaller classes in elementary school really benefit students in areas such as scores on standard tests, staying in school, and going to college?
The Tennessee STAR program: each students of 6,385 students who were beginning kindergarten was assigned to three types of classes:
(1) regular class with one teacher;
(2) regular class with one teacher and a full-time aide;
(3) small class.
Four years later, they returned to regular classes. The only systematic difference was the type of class. In later years, the students from small classes had higher scores on standard tests.
Q1: What is the treatment?
Q2: Is it an observational study or an experiment? Why?
Q3: Explanatory variable? Response variable?
Q4: What is the only systematic difference within the students?
Q5: Can it prove that class size made the difference?

7. Example:
A study comparing the reading skills of 200 boys with shoe size 6, and the reading skills of another group of 300 boys with shoe size 2. The results suggests that boys with bigger shoe size tend to have better reading skills.
Q1: Identify the response variable and the explanatory variable.
Q2: Is this study an observational study or an experiment?
Q3: Can we conclude that the shoe size is responsible for the reading skill?
response variable: reading skills
explanatory variable: shoes size
observational study

8. Drawbacks of Observational Studies
The effect of explanatory variable on the response may be confounded (mixed up) with lurking variables.
Lurking variables – the variable(s) associated with the response, but are not of interest; effects cannot be separated from the effect of the explanatory variable on the response
Observational studies: Often, the effect of one variable on another often fail because the explanatory variable is confounded with lurking variables.

9. The Strength of Experiments (compared with observational studies)
Experiments provide good evidence for causation (able to control lurking variables)
Lurking variables – the variable(s) associated with the response, but are not of interest; effects cannot be separated from the effect of the explanatory variable on the response

10. Chapter 4 Gathering Data
Section 4.2
Good and Poor Ways to Sample

11. Sampling Frame and Sampling Design
The sampling frame is the list of subjects in the population from which the sample is taken, ideally it lists the entire population of interest.
Eg: phone book, school list, et al
The sampling design determines how the sample is selected.

12. Simple Random Sampling (SRS)
Random Sampling is the best way of obtaining a sample that is representative of the population.
A simple random sample of ‘n’ subjects from a population is one in which each possible sample of that size has the same chance of being selected.
A simple random sample is often just called a random sample. The “simple” adjective distinguishes this type of sampling from more complex random sampling designs presented in Section 4.4.

13. SRS Example: selecting two cartoon characters
With four cartoon characters: Tom, Jerry, Mickey and Minnie, we want to select a simple random sample made of 2 characters.
Questions:
List the possible samples:
List of Possible Samples: {Tom, Jerry}, {Tom, Mickey }, {Tom, Minnie},
{Jerry, Mickey }, {Jerry, Minnie},
{Mickey, Minnie}.
2. What is the chance that a particular sample of size 2, such as {Jerry, Minnie} will be drawn?
3. What is the chance that Tom will be selected?

14. SUMMARY: Using Random Numbers to select a SRS
To select a simple random sample:
number the subjects in the sampling frame using numbers of the same length (number of digits).
select numbers of that length from a table of random numbers or using a random number generator.
include in the sample those subjects having numbers equal to the random numbers selected.

15. Using Random Numbers to select a SRS with TI84
To use TI84 to generate number and randomly select 2 subjects out of 4.
Step1: Put the subjects in an order, and determine to select the subjects with the largest assigned numbers.
Step2:From the main screen press [MATH] and use the arrow keys to scroll to PRB Step3:Select 1:rand and rand will be displayed on the main screen Step4:Press [(] [4] [)] and [ENTER] Step5:The calculator will display the 4 randomly generated numbers
Step6:Match each subject from Step1 with a number.
Step7: the two subjects associated with the 2 largest numbers is our random choice.
Q1: How do we randomly select two names from {Tom, Jerry, Micky, Minnie} ?
Q2: How do we randomly divide {Tom, Jerry, Micky, Minnie} into two groups?

16. SUMMARY: Types of Bias in Sample Surveys
Bias: When certain outcomes will occur more often in the sample than they do in the population.
Sampling bias occurs from using nonrandom samples or
having under-coverage.
Nonresponse bias occurs when some sampled subjects
cannot be reached or refuse to participate or fail to answer
some questions.
Response bias occurs when the subject gives an incorrect
response (perhaps lying) or the way the interviewer asks the
questions (or wording of a question in print) is confusing or
misleading.
A Large Sample Does Not Guarantee An Unbiased Sample!

17. Poor Ways to Sample
Convenience Sample: a type of survey sample that is easy to obtain.
Unlikely to be representative of the population.
Often severe biases result from such a sample.
Results apply ONLY to the observed subjects.
Volunteer Sample: most common form of convenience sample.
Subjects volunteer for the sample.
Volunteers do not tend to be representative of the entire population.
Eg1: A magazine publisher inserts a postage-paid survey into an issue of its magazine in order to collect information regarding reader satisfaction with the magazine.
Eg2: Ann Landers summarizing responses of readers 70% of (10,000) parents wrote in to say that having kids was not worth it—if they had to do it over again, they wouldn’t.

18. Review: Design Of Experiments (Bias in Comparative Experiments)
Ann Landers summarizing responses of readers
70% of (10,000) parents wrote in to say that having kids was not worth it—if they had to do it over again, they wouldn’t.
Bias: Most letters to newspapers are written by disgruntled people. A random sample showed that 91% of parents WOULD have kids again.

19. Find the bias (sampling bias, nonresponse bias, response bias) that a study may have
Eg1: A magazine publisher inserts a postage-paid survey into an issue of its magazine in order to collect information regarding reader satisfaction with the magazine. Eg2: Ann Landers summarizing responses of readers 70% of (10,000) parents wrote in to say that having kids was not worth it—if they had to do it over again, they wouldn’t. Eg3: A headline read: “Student Drug Testing Not Effective in Reducing Drug Use” in a news release from University of Michigan. Facts about study:
Sampling bias, response bias, non response bias
Sampling bias
76,000 students nationwide from 497 high schools and 225 middle schools
Schools selected for the study included schools that tested for drugs and schools that did not test for drugs
Each student filled out a questionnaire asking about his/her drug use
Response bias
Exercise ?

20. SUMMARY: Key Parts of a Sample Survey
Identify the population of all subjects of interest.
Construct a sampling frame which attempts to list all subjects in the population.
Use a random sampling design to select n subjects from the sampling frame.
Be cautious of sampling bias due to nonrandom samples (such as volunteer samples) and sample undercoverage, response bias from subjects not giving their true response or from poorly worded questions, and nonresponse bias from refusal of subjects to participate.
We can make inferences about the population of interest when
sample surveys that use random sampling are employed.

21. Chapter 4 Gathering Data
Section 4.3
Good and Poor Ways to Experiment

22. Elements of an Experiment
Experimental units: The subjects of an experiment; the entities that we measure in an experiment.
Treatment: A specific experimental condition imposed on the subjects of the study; the treatments correspond to assigned values of the explanatory variable.
Explanatory variable: Defines the groups to be compared with respect to values on the response variable.
Response variable: The outcome measured on the subjects to reveal the effect of the treatment(s).

23. Design of Experiments
Experimental units (subjects for human)– individual on which experiment is done.
Treatment (or factor)– specific experimental condition (e.g.: certain real medicine).
Placebo – false treatment to control for psychological effects (e.g.: sugar pills)
Types of variables:
Response variable – variable that measures the outcome of the study.
Explanatory variable (Factors) – variable(s) that explains or causes changes in the response variable.
In a study of sickle cell anemia, 150 patients were given the drug hydroxyurea, and 150 were given a placebo (dummy pill). The researchers counted the episodes of pain in each subject. Identify:
The subjects
The factors / treatments
And the response variable
Examples: 1. Smoking and lung cancer; 2.Running on a treadmill and heart rate; (p193)
HWQ: 3.26(a) 3.24, 3.25.
(patients, all 300)
(hydroxyurea and placebo)
(episodes of pain)
Examples: 1. Smoking and lung cancer; 2.Running on a treadmill and heart rate;

24. Experiments An experiment deliberately imposes treatments on the
experimental units in order to observe their responses.
The goal of an experiment is to compare the effect of the treatment on the response.
Experiments that are randomized occur when the
subjects are randomly assigned to the treatments;
randomization helps to eliminate the effects of lurking
variables.

25. 3 Components of a Good Experiment
Control/Comparison Group: allows the researcher to analyze the effectiveness of the primary treatment.
Randomization: eliminates possible researcher bias, balances the comparison groups on known as well as on lurking variables.
Replication: allows us to attribute observed effects to the treatments rather than ordinary variability.
Double Blind (if possible):Only the statistician knows which subject in the study is in which group, not the researcher(doctor or evaluator) and not the subjects themselves (patients).

26. Component 1: Control or Comparison Group
A placebo is a dummy treatment, i.e. sugar pill. Many subjects respond favorably to any treatment, even a placebo.
Placebo effect (power of suggestion). The “placebo effect” is an improvement in health due not to any treatment but only to the patient’s belief that he or she will improve.
A control group typically receives a placebo. A control group allows us the analyze the effectiveness of the primary treatment.
A control group need not receive a placebo. Clinical trials often compare a new treatment for a medical condition, not with a placebo, but with a treatment that is already on the market.

27. Component 1: Control or Comparison Group
Experiments should compare treatments rather than attempt to assess the effect of a single treatment in isolation.
Is the treatment group better, worse, or no different than the control group?
Example: 400 volunteers are asked to quit smoking and each start taking an antidepressant. In 1 year, how many have relapsed? Without a control group (individuals who are not on the antidepressant), it is not possible to gauge the effectiveness of the antidepressant.

28. Component 2: Randomization
To have confidence in our results we should randomly assign subjects to the treatments. In doing so, we
eliminate bias that may result from the researcher assigning the subjects.
balance the groups on variables known to affect the response.
balance the groups on lurking variables that may be unknown to the researcher.

29. Component 3: Replication
Replication is the process of assigning several experimental units to each treatment.
The difference due to ordinary variation is smaller with larger samples.
We have more confidence that the sample results reflect a true difference due to treatments when the sample size is large.
Since it is always possible that the observed effects were due to chance alone, replicating the experiment also builds confidence in our conclusions.

30. Blinding the Experiment
Ideally, subjects are unaware, or blind, to the treatment
they are receiving.
If an experiment is conducted in such a way that neither the subjects nor the researcher (evaluators) working with them know which treatment each subject is receiving, then the experiment is double-blinded.
A double-blinded experiment controls response bias
from the subject and experimenter.

31. Define Statistical Significance
If an experiment (or other study) finds a difference in two (or more) groups, is this difference really important?
If the observed difference is larger than what would be expected just by chance, then it is labeled statistically significant.
Rather than relying solely on the label of statistical significance, also look at the actual results to determine if they are practically significant.

32. Generalizing Results
Recall that the goal of experimentation is to analyze the association between the treatment and the response for the population, not just the sample.
However, care should be taken to generalize the results of a study only to the population that is represented by the study.

33. SUMMARY: Key Parts of a Good Experiment
A good experiment has a control comparison group, randomization in assigning experimental units to treatments, blinding, and replication. The experimental units are the subjects—the people, animals, or other objects to which the treatments are applied.
The treatments are the experimental conditions imposed on the experimental units. One of these may be a control (for instance, either a placebo or an existing treatment) that provides a basis for determining if a particular treatment is effective. The treatments correspond to values of an explanatory variable.

34. SUMMARY: Key Parts of a Good Experiment
Randomize in assigning the experimental units to the
treatments. This tends to balance the comparison groups
with respect to lurking variables.
Replicating the treatments on many experimental units
helps, so that observed effects are not due to ordinary
variability but instead are due to the treatment. Repeat
studies to increase confidence in the conclusions.