1. Methods of Comparison: Controlled Experiments and Observational Studies
Math 1680

2. Overview Introduction Controlled Experiments Minimizing Bias
Observational Studies
Association, not Causation
Comparing Rates
Salk Vaccine Trial
Breast Cancer Screening
Summary

3. Introduction Suppose a company wanted to market a weight-loss pill
After design is complete, the company still needs to answer one basic question
Does the pill actually work?
How can the company determine this?

4. Introduction
One method would be to give the pill to everyone who wants to lose weight and see if they actually do
What’s wrong with this approach?
First of all, the cost of giving everyone the pill is prohibitive, and there may be dangerous side effects for the untested pill.
Secondly, people who want to lose weight may be exercising or eating more healthily, which could overshadow the effects of the pill (This is an example of a confounding factor.)

5. Controlled Experiments
To test the pill, two groups are created from a sample of people
Control group is given a placebo
Treatment group is given the pill

6. Controlled Experiments
Why is it important that the people don’t know which group they are in?
It has been shown repeatedly that a patient's symptoms can be alleviated by an otherwise ineffective treatment if the individual expects or believes that it will work. Conversely, members of a control group knowingly receiving an inert substance tend to report a worsening of symptoms. This phenomenon is called the placebo effect.

7. Controlled Experiments
The most ideal form of experiment is one where neither the subjects nor the evaluators know who received the treatment
This is a double-blind experiment
This design provides the least opportunity for biased results

8. Controlled Experiments
After a period of time, the company checks to see how much weight people in each group have lost
If there is a significant difference between the groups, assume the pill was the reason
If there is no significant difference between the groups, assume the pill had no effect

9. Minimizing Bias
What if the control group and the treatment group differ in some way other than the treatment?
Then we have confounding factors in the study, and the results will be biased and/or invalid
Confounding factors can include…
Age
Race
Gender
Socio-economic status (SES)
Location
Political/religious orientation

10. Minimizing Bias
In the pill example, suppose all of the men were placed into the treatment group and all of the women were placed into the control group
What is wrong with this design?
The pill may interact differently with people based on gender. For example, suppose that high levels of testosterone diminish the effect of the pill. Then the results would be biased against the pill’s effectiveness.

11. Minimizing Bias
In order to minimize the risk of bias, researchers assign people to each group randomly from the pool of subjects
If enough people are in the pool, the groups will look similar in terms of age, race, gender, or any other demographic which could be a confounding element

12. Observational Studies
How would you design a controlled experiment to test if second-hand smoke causes lung cancer?
Answer: You can’t, really. The time scale is too large and it would be virtually impossible to ensure that your control group never got second-hand smoke exposure.

13. Observational Studies
In many studies, designing a controlled experiment is simply not possible
In such cases, researchers must settle for observational studies
Observational studies let the subjects select which group to join
Observational studies should be placed under great scrutiny because the control and treatment groups may be quite different in many ways
These differences may introduce confounding influences on the study results

14. Observational Studies
A TV commercial observes that volunteers who switched to having to Cheerios for breakfast reduced their cholesterol level by 4%. Does this show that Cheerios cereal is effective in reducing your cholesterol level?
Not necessarily. People who volunteer for such a study may already be interested in improving their health and may be taking steps such as exercising daily. This could lower their cholesterol level.

15. Observational Studies
One form of observational study involves using historical controls to compare a modern treatment with
The difference in conditions between past and present can itself be a confounding factor
For example, in testing a polio vaccine, one would not want to compare infection rates in the 1950’s with infection rates in the 1900’s because of the large differences in standard of living and health care quality in the two periods

16. Association, not Causation
An important limitation of all studies is that they can only show the strength of association between two variables
Association may point to causation
For example, if exposure to a virus causes a disease, then people who are exposed should be sicker than similar people who are not exposed
However, a study alone cannot show causation
Human inference is needed to connect causal links
Generally, a casual link is accepted if
A scientific theory is derived which explains the association
Enough studies controlling all potential confounding factors are performed, and all of them show the same association

17. Association, not Causation
Cervical cancer is more common among women who have been exposed to the herpes virus, according to many observational studies. Is it fair to conclude that the virus causes cervical cancer?
No. Both herpes and cervical cancer have been shown to be sexually transmitted diseases. Women who have many partners are at higher risk for both diseases than women with few partners, so the number and type of partners is a confounding influence which explains the association.

18. Association, not Causation
Some studies find an association between liver cancer and smoking. Does this mean smoking causes liver cancer, or is there a confounding influence at play?
People who smoke also tend to drink, so drinking is associated with smoking. It has been shown repeatedly that over consumption of alcohol can cause liver cancer.

19. Comparing Rates
Sometimes the groups in comparison are of different sizes
Rather than comparing the actual numbers, use rates or proportions such as percents
Adjusting to rates allows for a more direct comparison

20. Comparing Rates
In the U.S. in 1990, there were 2.1 million deaths from all causes compared to 1.7 million in 1960
According to Census data, there were about 177 million Americans in 1960 and about 250 million in 1990
True or false: The public’s health got worse over the period
False. Convert the death counts to percentage rates. The death rate in 1960 was 0.96%, while the death rate in 1990 was 0.84%. If anything, public health improved.

21. Salk Vaccine Trial
In 1954, researchers tested the effectiveness of a vaccine for polio
Approximately 400,000 children were selected
All children in the study had parental consent to be vaccinated (Why?)
Children randomly divided into equal sized treatment and control groups
Control group was given a placebo
Is this a controlled experiment or an observational study?

22. Salk Vaccine Trial Results Was the vaccine effective? Study Group
Population
Polio Cases
Rate per 100,000
Vaccinated
200,745 57
28.4
Placebo
201,229
142
70.6

23. Breast Cancer Screening
In 1963, an insurance company used 62,000 women (all insured by the plan) as subjects
Women ranged in age from 40 to 64
31,000 of the women were used as controls
The others were encouraged to undergo annual screening
Roughly 20,200 women from the treatment group came for annual screenings, the other 10,800 refused
After 5 years, the death rates were checked

24. Breast Cancer Screening
Is this a controlled experiment or an observational study?
It depends on what questions you are trying to answer. If you are only interested in the difference between the treatment and control groups, then you are using the results of a controlled experiment. However, when you start drawing inferences between the screened and refused groups, you are looking at an after-the-fact observational study.

25. Breast Cancer Screening
Deaths (breast cancer)
Deaths (all other causes)
Number
Rate per 1,000
Treatment
31,000 39
1.3
837 27
Examined
20,200 23
1.1
428 21
Refused
10,800 16
1.5
409 38
Control 63 2
879 28
Researchers noted three things: 
1.    Screening had little impact on diseases other than breast cancer.
2.    Poorer women were less likely to accept screening than richer ones.
3.    Most diseases fall more heavily on the poor than on the rich.

26. Breast Cancer Screening
Deaths (breast cancer)
Deaths (all other causes)
Number
Rate per 1,000
Treatment
31,000 39
1.3
837 27
Examined
20,200 23
1.1
428 21
Refused
10,800 16
1.5
409 38
Control 63 2
879 28
Does screening save lives? Which numbers prove your point?

27. Breast Cancer Screening
Deaths (breast cancer)
Deaths (all other causes)
Number
Rate per 1,000
Treatment
31,000 39
1.3
837 27
Examined
20,200 23
1.1
428 21
Refused
10,800 16
1.5
409 38
Control 63 2
879 28
Why is the death rate from all other causes in the whole treatment group about the same as the rate for the control group?

28. Breast Cancer Screening
Deaths (breast cancer)
Deaths (all other causes)
Number
Rate per 1,000
Treatment
31,000 39
1.3
837 27
Examined
20,200 23
1.1
428 21
Refused
10,800 16
1.5
409 38
Control 63 2
879 28
Why is the death rate from all other causes higher for the “refused group” than the examined group?

29. Breast Cancer Screening
Deaths (breast cancer)
Deaths (all other causes)
Number
Rate per 1,000
Treatment
31,000 39
1.3
837 27
Examined
20,200 23
1.1
428 21
Refused
10,800 16
1.5
409 38
Control 63 2
879 28
Unlike most diseases, breast cancer afflicts the rich more than the poor. Which numbers in the table confirm this association?

30. Case Study: Child care and behavior
Question: If parents choose to use child care, are they more likely to see undesirable behaviors in their children?
Question: How should one find out? By use of controlled experiments, or by observational study?

31. Case Study: Child care and behavior
Answer: In 1991, a study commenced on 1364 infants (“subjects”) and followed them through their sixth year in school. 12 years later, an article was published. “the more time children spent in child care from birth to age four-and-a-half, the more adults tended to rate them, both at age four-and-a-half and at kindergarten, as less likely to get along with others, as more assertive, as disobedient, and as aggressive.”

32. Case Study: Child care and behavior
A summary of the study noted, “The study authors noted that their study was not designed to prove a cause and effect relationship. …” (that more child care time causes more aggressive behavior)
Possible confounding factors? Perhaps… perhaps … perhaps…

33. Case Study: Child care and behavior
Question: How about a controlled experiment instead? (i.e. the researchers select children to place in child care)
What are the advantages?
What problems might arise?

34. Summary: Basic Principles of Statistical Experimental Design
Raise question to answer: Explanatory variable(s) & Response variable(s).
Leading principle: Comparison.
Other principles: randomize, repeat (or involve many subjects to reduce chance variation)
Why:

35. Summary: Principles of Statistical Experimental Design
Limitations: Statistical analysis of an experiment cannot tell us how far the results will generalize to other settings. (Experiment run in Massachusetts; how about in the entire USA?)
However, the randomized comparative experiment, because of its ability to give convincing evidence for causation, is one of the most important ideas in statistics.

36. More Elaborate Statistical Designs
Matched Pairs Design: Pair subjects with the same age, sex, income, etc. and compare their responses
Example: Compare two advertisements for the same product
Common variation of MPD: impose both treatments on the same subjects, so that each subject serves as his or her own control. (See HW2, Q9)

37. More Elaborate Statistical Designs
Question: How does randomization work here?
Answer: Which one of a matched pair sees the first ad is decided at random. Or, which treatment a person gets first is decided at random if he or she also serves as his or her own control.
Additional Example: Matched Pairs for the cell phone experiment. (Details: trained in using simulator, order in which a subject drives with and without the phone be random, two drives be on separate days)

39. Summary Statisticians use the method of comparison
To determine the effectiveness of a “treatment”, they usually compare the responses of a treatment group with a control group
If the control group is comparable to the treatment group (apart from the treatment), then a significant difference in the responses of the two groups is likely to be due to the effect of the treatment

40. Summary
If the treatment group is different from the control group with respect to other factors, the effects of these other factors are likely to be confounded with the effect of the treatment
To make sure the treatment group is like the control group, investigators put subjects into either group at random

41. Summary Whenever possible, the control group is given a placebo
The response should be to the treatment itself rather than to the idea of the treatment
In a double-blind experiment, the subjects do not know whether they are in treatment or in control, and neither do those who evaluate the responses
Guards against bias, both in the responses and in the evaluations

42. Summary
In an observational study, the investigators do not assign the subjects into treatment or control groups
Subjects which have the condition are the treatment group and the ones which do not are the control
Observational studies can establish association, which is not necessarily causation

43. Summary
Observational studies are particularly susceptible to confounding factors
This can make them very misleading about cause and effect relationships
With observational studies, try to find out how the subjects came to be in treatment or control
Are the groups comparable?
What factors are confounded with the treatment?
What adjustments (if any) did the investigators make to control for these factors?