Experimental Design

HW3

Harry Potter Best selling book series in history. Highest grossing film series in history. “Harry Potter and the Prisoner of Azkaban” won the Hugo Award for best Novel.

Argument from Ignorance “Absence of evidence is not evidence of absence.” – Carl Sagan Just because there is no evidence that something is true, does not mean that it is false. Just because there is no evidence that something is false, does not mean that it is true.

Shifting the Burden of Proof A similar fallacy is “shifting the burden of proof”. It goes: “God exists. If you think otherwise, prove that he doesn’t!” Here, you make a claim (“God exists”) but instead of giving evidence for it, you require that your opponent give evidence for the opposite.

The Christian Devil

Appeal to Popularity “Millions of people around the world believe that we have been visited in the past by extraterrestrial beings.” – Narrator from “Ancient Aliens”

Antonin Scalia Longest serving justice on US Supreme Court. Taught at the University of Chicago. First Italian-American SC justice. Extremely powerful and influential

Interview with Scalia Isn’t it terribly frightening to believe in the Devil? You’re looking at me as though I’m weird. My God! Are you so out of touch with most of America, most of which believes in the Devil? I mean, Jesus Christ believed in the Devil! It’s in the Gospels! You travel in circles that are so, so removed from mainstream America that you are appalled that anybody would believe in the Devil! Most of mankind has believed in the Devil, for all of history. Many more intelligent people than you or me have believed in the Devil.

RECAP: SCIENTIFIC METHOD

The Water Cycle

Earthquakes & Tsunamis

Earth’s Seasons

Science We’ve learned a lot of bad ways (fallacies) for figuring out whether claims are true. There is a good way of finding things out: science! Science tries to discover the causal structure of the world, so it can predict, explain, and control nature for the benefit of humankind.

Science We cannot directly observe the causal structure of the world, we can only observe correlations, and theorize about them. The goal of science is to test our theories about the causal structure of the world. We try to show that they are false. If we try very, very hard to show they are false, and we keep failing, then we can accept them as true.

EXPERIMENTAL DESIGN

Types of Scientific Studies There are two basic types of scientific studies (the stuff that gets published in scientific journals and reported in the “science” section of the newspaper): Observational studies Controlled experiments

Observational Studies An observational study looks at data in order to determine whether two variables are correlated.

Case Study In science, we want to know about the effects of something (exposure to radiation, living through a certain political crisis…) or the causes of something (a disease, having certain beliefs…). A case study finds people who have the condition we want to know about (they were exposed to radiation, or they have the disease) and looks back at their histories.

Example Suppose I want to know why people gamble. I might find a group of gamblers and give them all a survey: When did you first have sex? Do you smoke? Did your parents divorce? When you win money, how do you spend it? Do you eat meat?

Problems with Case Studies Suppose I find that 27% of the gamblers I survey have divorced parents. Does that mean divorce is significant cause of gambling? No. We need to know if this is more or less than the divorce rate among non-gamblers. (In fact, it’s about the same: HK divorce rate is 20%- 30%.)

Case Control Study

In a case control study we find not just a group of people we’re interested in (gamblers) but also a group of people we’re not interested in (the control group, non-gamblers). The goal is to compare the histories of one group to the histories of the other group.

Problems with Case Control Studies Correlation is not causation! What if I discover that more gamblers smoke than non-gamblers. I still don’t know: Maybe smoking causes gambling. Maybe gambling causes smoking. Maybe poverty causes gambling & smoking. Maybe it’s just a coincidence.

Unreliable Histories

Why Do We Do Them? Case control studies can be done very easily, very fast, and with very little expense. Scientists will use them to suggest things to study more seriously, or to rule out certain hypotheses. After all, if gamblers smoke less than non-gamblers, smoking probably does not cause gambling!

Cohort Studies

Cohort studies are more reliable than case control studies. In a cohort study, you follow two groups over time. One group, the cohort, has a certain condition (for example, smokes) and the other group doesn’t. Then you see what happens and compare the results.

Cohort Studies For example, an cohort study might ask women to record how much wine they drink, and also to report if they develop breast cancer. After many years, a correlation may be found between wine consumption and cancer.

Advantages over Case Control Avoids recall bias. Lets us study changes over time. Useful for studying rare conditions. Lets us investigate many effects. Allows us to calculate the relative risk (the amount that a condition increases or decreases your risk of something.)

Problems: Confounding Variables Suppose my cohort is a group of smokers. Smokers tend to have more in common with one another than just smoking: The poor smoke more than the rich. The uneducated smoke more than the educated. People who drink alcohol smoke more than people who do not.

Problems: Confounding Variables Anything that we discover in the cohort that is correlated with smoking will also be correlated with all the confounding variables! So if smokers get more cancer, is it because they smoke, or because they don’t have money to go to a hospital for checkups?

Observational Studies Importantly, observational studies can only show whether two variables A and B are correlated. They cannot show whether A causes B, or B causes A, or some third cause causes both, or if the correlation is accidental.

EXPERIMENTAL STUDIES

Controlled Experiments The first recorded controlled experiment occurs in the Book of Daniel, part of the Jewish Torah and the Christian Bible.

Daniel 1:1-16 In the book, Daniel is forced into the service of King Nebuchadnezzar of Babylon. He is fed the king’s meat and wine, but he refuses – the Jews have special laws about how things like meat and wine are prepared.

Daniel’s Experiment “Please test your servants for ten days: Give us nothing but vegetables to eat and water to drink. Then compare our appearance with that of the young men who eat the royal food, and treat your servants in accordance with what you see.” Daniel 1:12-13

Daniel’s Experiment “At the end of the ten days they looked healthier and better nourished than any of the young men who ate the royal food. So the guard took away their choice food and the wine they were to drink and gave them vegetables instead.” Daniel 1:15-16

Controlled Experiments In a controlled experiment there are two groups who get separate treatments. One group, the “control group” gets the standard treatment. For example, all of the king’s servants ate meat and wine before Daniel suggested a different diet might be better.

Controlled Experiments The other group, the “experimental group”, gets the treatment we plan to test. If the test group has better results than the control group, we have good evidence that our new treatment should be adopted.

Why are They Better? Observational studies only reveal correlations, they can’t reveal causation. Controlled experiments are also only studies of correlation: correlation between the control group and outcomes, and correlation between the experimental group and outcomes.

Why are They Better? But controlled experiments are better than observational studies. Why? In observational studies, people are not randomly assigned to conditions. For example, an observational study might find a correlation between using a cane and dying within a year.

Canes This is because old people are more likely to use a cane and more likely to die (than young people). If you randomly assigned young and old people to cane or no-cane conditions, the correlation would go away. Canes don’t cause death.

Confounding Variables A confounding variable is a variable that affects the variables you want to study. For instance, if you want to study whether canes cause death, age is a confounding variable, because age influences your chances of death.

Confounding Variables A controlled experiment lets you “control for” confounding variables. You can make the control group and the experimental group have equal numbers of people from each age group. Then you know that if more people in your experimental group die, it wasn’t due to their age (the other group had similar ages).

Controlling In an observational study, there is no way to rule out a common cause for two correlated variables A and B. In an experimental study, the common cause is ruled out, because the experimenter is the one who causes (“controls”) whether people have A or not.

Controlling In an observational study, there is no way to rule out B causing A rather than A causing B. Does wine reduce the risk of cancer, or does a lowered risk of cancer increase wine consumption? If experimenters control who gets wine, then we can rule out the hypothesis that in our study, lowered cancer risk causes wine drinking.

Next Time We’ll talk more next time about other things that can bias an experiment and how to “control for” them.

What about Observational Studies? Why do scientists still conduct observational studies, if controlled experiments are considered better evidence? 1.Moral reasons 2.Practical reasons

Moral Reasons Sometimes performing a controlled experiment would be unethical. For example, suppose we want to know whether vaccines cause autism (NOTE: they do not). We cannot simply stop vaccinating people.

Moral Reasons If you stopped vaccinating children, you’d effectively be killing lots of children (and adults). Vaccines prevent lots and lots of otherwise deadly infectious diseases.

Moral Reasons Thus you must conduct an observational study. Find people who (for whatever reason) chose not to vaccinate their children, and compare their rates of autism to those of the vaccinated children. When you do this you find that vaccines do not cause autism. (No correlation, hence no causation.)

Practical Reasons Some controlled experiments are also simply impractical. Does being smart make you rich? Well, we can’t make a random group of people smart. That’s impossible. Does being rich make you smart? Well, we can’t give a random bunch of people a lot of money– we’re just poor scientists!