2.1 Observe or Experiment?

Should we use an observational study or an experiment? What’s the difference? In an observational study, we record data or survey results WITHOUT INTERFERING with the subjects responses In an experiment, the researcher IMPOSES TREATMENTS on the subjects, and records data for each group.

Which is best? A well-controlled experiment is the ONLY WAY to establish CAUSE AND EFFECT An observational study with a well-chosen sample group can give us quick results with little expense Often, the results of an observational study will cause interest in designing an experiment to follow up.

Observational Study Advantages 1. Quick results 2. Not too expensive 3. Few ethical restraints 4. Can cover a wide variety of factors at once Disadvantages 1. Can only show CORRELATION, NOT CAUSATION 2. Sometimes, it’s difficult to observe without affecting the data

Experiment Advantages 1. Can establish CAUSE and EFFECT 2. Can eliminate many lurking variables Disadvantages 1. Ethical restrictions, especially on human testing 2. Financial restrictions 3. Time constraints

Either study can be done well or done poorly Sample size Poor questions Bad design Lurking variables Overreaching population Overreaching inference

2.2 What makes a good observational study?

Types of Bias to Avoid Sampling Bias: When the sample does not reflect the population of interest. There are several sampling errors that might cause this: sample size is too small (called undercoverage), the sample is not random, the sample is not chosen from the population of interest. Response Bias: When the members of the sample group are giving unusable or unmeaningful responses. This could be for a number of reasons: confusing questions, leading questions, unprotected anonymity Nonresponse Bias: When significant numbers of members of the sample group fail to respond to the survey. Phone-in and reply-by-mail are highly likely to suffer from this type of bias.

So What Size Sample is Needed? Recall the formula for Margin of Error: MOE = Suppose we want our results to have a margin of error of only +/- 3%. What sample size will insure this? Suppose we would be satisfied with a MOE of 7.5%. What sample size would be needed?

Are these results statistically significant? A randomly chosen group of 250 dentists were polled about whether they would recommend brand A or brand B as the first choice for toothpaste. 120 chose brand A, 110 chose brand B, and the rest had no opinion. What is the margin of error for this study? Is the preference for brand A statistically significant? How can we tell?

2.3 Types of Observational Studies

Some Basic Types Sample Survey: Collects data now, about present. Most opinion and political surveys are this type. Retrospective Study: Collects data now, but about the past. We get the data immediately, but must trust people’s memories of past experiences Case-Control Study: many medical studies are this type. It avoids the ethical restrictions on experimenting on dangerous medical concerns. Prospective Study: set up now, but must wait to collect data in the future. Very reliable data, but the waiting is lengthy and not always practical.

Cautions for Observational Studies Beware the LURKING VARIABLE Beware the CONFOUNDING VARIABLE Be cautious in making claims about CAUSE CORRELATION does not imply CAUSATION!!!

2.4 What makes a good experiment?

4 Components of a good Experiment Replication: Does the experiment have enough subjects? Remember, we can use MOE to measure this Randomization: Were the subjects divided into groups by a random method? Control/Comparison Group: One group should receive no treatment, or perhaps the current standard of care. Not all groups should receive the new treatment being tested. Blinding: Subjects should be unaware of which treatment group they are a part of. This is critical because of the placebo effect.

What’s wrong here? 1. PharmaB wants to test their new medication for migraines. Their 2700 subjects are randomly placed into three groups. Group A receives a does of 50 mg. Group B receives a dose of 100 mg. Group C receives a dose of 200 mg. Data is collected from each group.

What’s wrong here? 2. Science Sam wants to know if vitamin C will help students improve their math scores. His 2000 subjects choose to be in Group A, which receives the vitamin, or in Group B, which receives the placebo. Sam collects the data from both groups.

What’s wrong here? 3. Nervous Nelly wants to test a new organic form of Xanax. She randomly assigns her 21 subjects to three groups. Group A receives a placebo. Group B receives their regular Xanax. Group C receives the new treatment. Nelly records the data from all three groups.

Do these results really matter? Are the results statistically significant? Use MOE Are the results practically significant? Harder to compute, but also important. For example, if the new medication will end a migraine 30 minutes sooner, but costs 5 times as much as the current treatment, would consumers by likely to buy it?

Testing Statistical Significance 1. At St. Luke's hospital in 1998, 674 women were diagnosed with breast cancer. Five years later, 88% of the Caucasian women and 83% of the African American women were still alive. A researcher concludes that being Caucasian causes women with breast cancer to have an increased chance of surviving five years. Use MOE to determine whether this conclusion is statistically significant.

2.5 Types of Experiments Multifactor: Sometimes we want to test more than one factor. Especially in medical situations, there is often more than one contributing factor. This design is also used to determine whether prescriptions affect each other, and how. They do require a larger group of subjects, and are usually limited to two or three factors at the most.

2.5 Types of Experiments Matched Pair: Matched Pair design simply means that each subject is part of a pair. It could be “married couple” or “plant from the same field”, or any logical reason to pair the subjects and track their data together. Crossover Design: is a type of matched pair in which each subject is part of the treatment group, and at a different time, also part of the control group. This is a very strong model, as it eliminates nearly all lurking variables.

2.5 Types of Experiments Block Design: This design is similar to stratified random sampling for observational studies. By blocking the subjects first by gender, for example, the researcher insures that each group will contain both genders. Blocking can be done by gender, age, economic status, ethnicity, or any variable of interest.