Experiments Textbook 4.2

Observational Study vs. Experiment Observational Studies observes individuals and measures variables of interest, but does not attempt to influence the responses. A sample survey is a type of observational study The goal is to describe some group or situation, compare groups, examine relationships, etc. Experiments deliberately impose some treatment on individuals to measure their responses. The goal is to determine whether the treatment causes a change in response.

Confounding When our goal is to understand cause and effect, experiments are the only source of fully convincing data. For example: we want to see if taking hormones will help prevent heart attacks. Explanatory Variable = whether or not a woman takes hormones Response Variable = whether or not that woman has a heart attack DANGER: Confounding variables exist when their effects on the response variable cannot be distinguished from each other Example: What if the woman already has high cholesterol, or family history of heart disease, or is very healthy…

Check Your Understanding Does eating dinner with their families improve students’ academic performance? According to an ABC News Article, “Teenagers who eat with their families at least five times a week are more likely to get better grades in school.” This finding was based on a sample survey conducted by researchers at Columbia University.  Was this an observational study or an experiment?  What are the explanatory and response variables?  Explain clearly why such a study cannot establish a cause- and-effect relationship. Suggest a variable that may be confounded with whether families eat dinner together.

The Language of Experiments An experiment is a statistical study in which we actually do something (a treatment) to people, animals or objects (the experimental units) to observe the response. The experimental units are the smallest collection of individuals to which treatments are applied. When the units are human beings, they are often called subjects. Experiments can give good evidence for causation. We are able to test the effects of certain variables while trying to control others. The experimental variables in an experiment are called factors. When the factors take on different values (called levels), we can study the interaction of several factors.

Practice What are the effects of repeated exposure to an advertising message? The answer may depend on both the length of the ad and how often it is repeated. An experiment investigated this question using 120 undergraduate students who volunteered to participate. All subjects viewed a 40-minute television program that included ads for a digital camera. Some subjects saw a 30-second commercial; others a 90- second commercial. The same commercial was shown either 1, 3, or 5 times during the program. After viewing, the subjects answered questions about their recall of the ad, their attitude toward the camera, and their intention to purchase it.  What are the subjects in this experiment?  What are the factors?  What are the levels of the factors?  How many total treatments are being imposed?

How to Experiment Badly A high school regularly offers a review course to prepare students for the SAT. This year, budget cuts will allow the school to offer only an online version of the course. Suppose the group of students who take the online course earn an average increase of 45 points in their math scores from a pre-test to the actual SAT test. Can we conclude that the online course is effective? What’s wrong with this experiment?

How to Experiment Well We need to compare two or more treatments. Random assignment to treatments is also essential for good experimental design. How can we improve the previous experiment? Select 50 students, half of which will participate in the online SAT course, and the other half will participate in a classroom course. Number the subjects. How can we randomize the assignment to the two treatments?

Principles of Experimental Design  Comparison. Use a design that compares two or more treatments.  Random Assignment. Use chance to assign experimental units to the treatments. Doing so helps create roughly equivalent groups of experimental units by balancing the effects of other variables among the treatment groups.  Control. Keep other variables that might affect the response the same for all groups.  Replication. Use enough experimental units in each group so that any differences in the effects of the treatments can be distinguished from chance differences between the groups.

Completely Randomized Design Music students often don’t evaluate their own performances accurately. Can small-group discussions help? The subjects were 29 students preparing for end-of-semester performance that is an important part of their grade. The 15 students in one group each videotaped a practice performance, evaluated it themselves, and then discussed the tape with a small group of other students. The remaining 14 students watched and evaluated their tapes alone. At the end of the semester, the discussion group students evaluated their final performance more accurately.  Describe a completely randomized design for this experiment.  What is the purpose of the control group in this experiment?

What can go wrong? The logic of experimentation depends on our ability to treat all experimental units the same in every way except for the actual treatments being compared. Particularly, in medical trials, we must be aware of the placebo effect in which a patient actually responds to “fake” medicine. In these experiments, a treatment including placebo must be included. There are also times when an experimenter may subconsciously treat subjects differently based upon prior information. Double Blind studies are important in this case – in which neither the subject NOR the experimenter knows which treatment is being administered.

Inference for Experiments In an experiment, researchers usually hope to see a difference in the responses so large that it is unlikely to happen merely by chance. We use the laws of probability to determine if the effects are larger than expected. If the results are larger than expected, we call them statistically significant. If we observe statistically significant results in a well- designed, randomized experiment, we can say we have evidence of a cause-and-effect relationship.

Blocking Like with Simple Random Samples, sometimes a slightly more complex experimental design is necessary for more precise results. Suppose that a phone company is considering two different keyboard designs (A and B). The company decides to perform an experiment to compare the two keyboards using 10 volunteers. 4 of the 10 volunteers already own smartphones, though, so their typing ability will already be better… We can use a randomized block design to more accurately compare the keyboards. 10 Volunteers Smart phone users Keyboard A (2 ppl) Keyboard B (2 ppl) Non-smart phone users Keyboard A (3 ppl) Keyboard B (3 ppl)

Matched Pairs Design A common type of blocked design for comparing two treatments is a matched pairs design. The idea is to create blocks by matching pairs of similar experimental units. Then we randomly decide which member of the pair receives each treatment. Get your heart beating activity (p. 255)