Experiments: What Can Go Wrong? The logic of a randomized comparative experiment depends on our ability to treat all the subjects the same in every way except for the actual treatments being compared. Good experiments, therefore, require careful attention to details to ensure that all subjects really are treated identically. A response to a dummy treatment is called a placebo effect. The strength of the placebo effect is a strong argument for randomized comparative experiments. Placebo effect: Mommy’s kiss on a child’s “boo-boo” when hurt. Even though the kiss has no “active treatment”, it makes the child feel better!

Blinding - method used so that units do not know which treatment they are getting Double blind - neither the units nor the evaluator know which treatment a subject received. Whenever possible, experiments with human subjects should be double-blind.

Inference for Experiments In an experiment, researchers usually hope to see a difference in the responses so large that it is unlikely to happen just because of chance variation. We can use the laws of probability, which describe chance behavior, to learn whether the treatment effects are larger than we would expect to see if only chance were operating. If they are, we call them statistically significant. Definition: An observed effect so large that it would rarely occur by chance is called statistically significant. A statistically significant association in data from a well-designed experiment does imply causation.

Units should be blocked on a variable that effects the response!!! Experiment Designs Randomized block design – units are blocked into groups (homogeneous) and then randomly assigned to treatments Random assignment Units should be blocked on a variable that effects the response!!!

Randomized block design Treatment B Treatment A Treatment A Treatment B Randomly assign experimental units to treatments Put into homogeneous groups

Blocking Experiments Definition Completely randomized designs are the simplest statistical designs for experiments. But just as with sampling, there are times when the simplest method doesn’t yield the most precise results. Experiments Definition A block is a group of experimental units that are known before the experiment to be similar in some way that is expected to affect the response to the treatments. In a randomized block design, the random assignment of experimental units to treatments is carried out separately within each block. Form blocks based on the most important unavoidable sources of variability (lurking variables) among the experimental units. Randomization will average out the effects of the remaining lurking variables and allow an unbiased comparison of the treatments. Control what you can, block on what you can’t control, and randomize to create comparable groups.

Men, Women, and Advertising Women and men respond differently to advertising. Researchers would like to design an experiment to compare the effectiveness of three advertisements for the same product. PROBLEM: (a) Explain why a randomized block design might be preferable to a completely randomized design for this experiment. (a) A completely randomized design considers all subjects, both men and women, as a single pool. The random assignment would send subjects to three treatment groups without regard to their gender. This ignores the differences between men and women, which would probably result in a great deal of variability in responses to the advertising in all three groups. For example, if an ad appealed much more to men, you would get a wide range of reactions to that ad from the two genders. That would make it harder to compare the effectiveness of the ads. A randomized block design would consider women and men separately. In this case, the random assignment would occur separately in each block. blocking will control for the variability in responses to advertising due to gender. This will allow researchers to look separately at the reactions of men and women, as well as to more effectively assess the overall response to the ads.

Men, Women, and Advertising Women and men respond differently to advertising. Researchers would like to design an experiment to compare the effectiveness of three advertisements for the same product. (b) Outline a randomized block design using 300 volunteers (180 men and 120 women) as subjects. Describe how you would carry out the random assignment required by your design. We randomly assign the 120 women into three groups of 40, one for each of the advertising treatments. Give each woman a distinct label between 1 and 120. Use a computer’s random number generator to sort the numbers from 1 to 120 in a random order. The women with labels corresponding to the first 40 numbers in the list will view Ad 1, the next 40 will view Ad 2, and the last 40 will view Ad 3. randomly assign the 180 men into three groups of 60 using a similar process. After each subject has viewed the assigned ad, compare reactions to the three ads within the gender blocks.

Matched pairs - a special type of block design Two types: 1.) Match up experimental units according to similar characteristics & randomly assign one to one treatment & the other automatically gets the 2nd treatment 2.) Have each unit do both treatments in random order -the assignment of treatments is dependent

Pair experimental units according to specific characteristics. Treatment A Treatment B Next, randomly assign one unit from a pair to Treatment A. The other unit gets Treatment B. Pair experimental units according to specific characteristics. This is one way to do a matched pairs design – another way is to have the individual unit do both treatments (as in a taste test).

Treatment & group are confounded Treatment B Treatment A Confounding does NOT occur in a completely randomized design! One group is assigned to treatment A & the other group to treatment B. Treatment A Treatment B

Is this an experiment? Why or why not? Example 4: An article from USA Today reports the number of victims of violent crimes per 1000 people. 51 victims have never been married, 42 are divorced or separated, 13 are married, and 8 are widowed. Is this an experiment? Why or why not? What is a potential confounding variable? No, no treatment was imposed on people. Age – younger people are more at risk to be victims of violent crimes

Is this an experiment? Why or why not? Example 5: Four new word-processing programs are to be compared by measuring the speed with which standard tasks can be completed. One hundred volunteers are randomly assigned to one of the four programs and their speeds are measured. Is this an experiment? Why or why not? Yes, a treatment is imposed. Yes, a treatment was imposed Completely randomized one factor, word processing program & 4 levels, the four new programs Speed at which standard tasks can be done What type of design is this? Factors? Levels? Response variable? Completely randomized one factor: word-processing program with 4 levels speed

Can this design be improved? Explain. Example 5: Four new word-processing programs are to be compared by measuring the speed with which standard tasks can be completed. One hundred volunteers are randomly designed to one of the four programs and their speeds are measured. Is there a potential confounding variable? Can this design be improved? Explain. a) Speed/expertise of each individual b) Use a matched pairs design where each volunteer uses all four programs in random order You could do a block design where each person uses each program in random order. NO, completely randomized designs have no confounding

What type of design is this? Why use this method? Example 6: Suppose that the manufacturer wants to test a new fertilizer against the current one on the market. Ten 2-acre plots of land scattered throughout the county are used. Each plot is subdivided into two subplots, one of which is treated with the current fertilizer, and the other with the new fertilizer. Wheat is planted and the crop yields are measured. What type of design is this? Why use this method? When does randomization occur? Matched - pairs design Randomly assigned treatment to first acre of each two-acre plot

Bias is a systematic error in measuring the estimate Randomization reduces bias by spreading any uncontrolled confounding variables evenly throughout the treatment groups. Bias is a systematic error in measuring the estimate Blocking also helps reduce variability. Variability is controlled by sample size. Larger samples produce statistics with less variability.

High bias & high variability High bias & low variability Low bias & high variability Low bias & low variability