1. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 1 Chapter 10: Experimental Research Objectives: Briefly state the purpose of experimental research and list the basic steps involved in conducting and controlling an experiment. Briefly define internal validity and describe eight major threats to the internal validity of an experiment.

2. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 2 Chapter 10: Experimental Research Objectives: Briefly define external validity and describe six major threats to the external validity of an experiment. Identify and briefly describe five ways to control extraneous variables.

3. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 3 Chapter 10: Experimental Research Objectives: Define and provide examples of single- variable designs (i.e., pre-experimental, true experimental, and quasi- experimental designs) and factorial designs, and explain how they differ.

4. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 4 Experimental Research Experimental research is the only type of research that can test hypotheses to establish cause-effect relations. The researcher manipulates at least one independent variable and controls other relevant variables, and observes the effect on one or more dependent variables. The researcher manipulates the treatment. The researcher has control over selection and assignment.

5. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 5 Experimental Research In experimental research studies the independent variable is also called the treatment, causal, or experimental variable. In experimental research studies the dependent variable is also called the criterion, effect, or posttest variable.

6. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 6 Experimental Research Experimental research is the most structured of all research. When conducted well, experimental research studies can provide evidence for cause-effect relations. Several experimental studies taken together can provide support for generalization of results.

7. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 7 Experimental Research The experimental process The steps in the experimental research process are the same as in other types of research. Selecting and defining a problem Selecting participants and measuring instruments Preparing a research plan Executing procedures Analyzing the data Formulating conclusions

8. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 8 Experimental Research The experimental process In experimental studies, the researcher controls selection and assignment. Experimental studies often examine comparisons between or among groups. Comparison of approaches (A versus B) Comparison of an approach to an existing approach (A versus no A) Comparison of different amounts of a single approach (A little of A versus a lot of A)

9. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 9 Experimental Research The experimental process In experimental research studies the group that receives the treatment is the experimental group and the group that does not receive the treatment is called the control group. Sometimes groups are comparison groups that receive alternative treatments (e.g., two types of instruction in a content area).

10. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 10 Experimental Research Manipulation and control In experimental studies, researchers control or remove the influence of extraneous variables. Participant variables Organismic (e.g., age) Intervening (e.g., interest) Environmental variables (e.g., school or teacher effects)

11. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 11 Threats to Experimental Validity Internal validity refers to the degree to which observed differences in the dependent variable are a direct result of manipulation of the independent variable and not some other variable. Internal validity is concerned with rival explanations for an effect.

12. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 12 Threats to Experimental Validity External validity, sometimes referred to as ecological validity, is the degree to which the results from a study are generalizable to other groups. When researchers increase the internal validity of their study, they decrease their external validity.

13. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 13 Threats to Experimental Validity When researchers are concerned with external validity, their ability to control important extraneous variables suffers. When there is a choice, researchers should err on the side of control and maximize internal validity.

14. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 14 Threats to Internal Validity History Maturation Testing Instrumentation Statistical regression Differential selection of participants Mortality Selection-maturation and interactive effects

15. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 15 Threats to Internal Validity History Any event occurring during a study that is not part of the experimental treatment but that may effect the dependent variable represents a history threat. Longer-lasting studies are more prone to history threats.

16. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 16 Threats to Internal Validity History threat example In a study of the effects of instructional simulations in learning chemistry content, a history threat would be demonstrated if students in the study were exposed to simulations in a different setting, such as when learning geography, while the study was being conducted.

17. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 17 Threats to Internal Validity Maturation Maturation refers to physical, intellectual, and emotional changes that naturally occur within participants over a period of time.

18. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 18 Threats to Internal Validity Maturation threat example In studies of interventions that are designed to increase children’s theory of mind, if the interventions lasted more than a couple of weeks at critical time points, participants may gain critical theory of mind awareness simply due to cognitive development and not due to the treatment.

19. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 19 Threats to Internal Validity Testing Testing as a threat to internal validity is demonstrated when taking a pretest alters the result of a posttest.

20. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 20 Threats to Internal Validity Instrumentation Instrumentation is a threat to internal validity when the instrumentation is either unreliable or is changed between pre- and posttesting.

21. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 21 Threats to Internal Validity Statistical regression Extremely high or low scores tend to regress to the mean on retesting. Statistical regression example If students perform poorly on a pretest it is difficult to determine if the gain in their scores is due to treatment effects.

22. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 22 Threats to Internal Validity Differential selection of participants Participants in the control and experimental groups differ in ways that influence the dependent measure.

23. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 23 Threats to Internal Validity Mortality Participants drop out of the study at differential rates across conditions.

24. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 24 Threats to Internal Validity Selection can interact with other threats to internal validity (i.e., history, maturation, instrumentation). Participants selected into the treatment and control conditions have different experiences or maturation rates or instrumentation varies across conditions.

25. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 25 Threats to External Validity External validity threats can be divided into two categories: ‘Generalizing to whom’ threats Threats affecting groups to which the study can be generalized ‘Generalizing to what’ threats Threats affecting the settings, conditions, variables, and contexts to which the results can be generalized

26. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 26 Threats to External Validity Pretest-treatment interaction Multiple-treatment interference Selection-treatment interaction Specificity of variables Experimenter effects Reactive arrangements

27. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 27 Threats to External Validity Reactive arrangements Hawthorne effect: Any situation in which participants’ behavior is affected because they are in a study. John Henry effect (Compensatory rivalry): Members of the control group compete with the experimental group.

28. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 28 Threats to External Validity Reactive arrangements Placebo effect: To combat compensatory rivalry, researchers attempt to give control groups a placebo, not the experimental treatment, but something to decrease the perception that they are in the control group. Participants should perceive they are all getting the same thing. Novelty effect: When participants are engaged in something different this may increase attention, interest, behavior, learning, etc., just because it is something new.

29. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 29 Group Experimental Designs The validity of an experiment is a function of the degree to which extraneous variables are controlled. Randomization is the best mechanism to control for extraneous variables. Randomization distinguishes experimental designs. Randomization should be used whenever possible. If groups cannot be randomly formed, variables should be held constant when at all possible (e.g., time of day, which researcher is present).

30. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 30 Group Experimental Designs Participant variables can be controlled and held constant Matching can equate groups through random assignment of pairs. Comparing homogeneous groups allows the researcher to control for extraneous variables.

31. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 31 Types of Group Designs Single-variable designs are any design that involves one manipulated variable. Pre-experimental designs do not adequately control for extraneous variables and should be avoided. True-experimental designs offer a very high degree of control and are always preferred designs. Quasi-experimental designs do not control as well as experimental designs but are preferable over pre-experimental designs.

32. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 32 Pre-Experimental Designs The one-shot case study involves a single group that is exposed to a treatment (X) and then posttested (O). XOXO Threats to validity are not adequately controlled with this design. Do not use this design.

33. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 33 Pre-Experimental Designs The one-group pretest-posttest design involves a single group that is pretested, exposed to treatment, and then tested again. OXOOXO The success of the treatment is determined by comparing pretest and posttest scores. This design does not control for history, testing, instrumentation, regression, or maturation. Statistical regression is not controlled nor is pretest-treatment interaction.

34. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 34 Pre-Experimental Designs The static-group comparison design involves at least two nonrandomly formed groups. One group receives an experimental treatment and the other group receives the traditional treatment. Both groups are posttested. X1OX1O X2OX2O

35. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 35 Pre-Experimental Designs The number of groups can be expanded beyond two. The groups are better described as comparison, not experimental and control. This design does not control for maturation, selection effects, selection interactions, and mortality.

36. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 36 Pre-Experimental Designs There is some control for history in this design. This design is sometimes used in exploratory studies.

37. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 37 True Experimental Designs The pretest-posttest control group design requires at least two groups. Groups are formed by random assignment. Both groups are administered a pretest, each group receives a different treatment and both groups are posttested. The design may be extended to include additional groups.

38. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 38 True Experimental Designs ROX1OROX1O ROX2OROX2O ROX3OROX3O The combination of random assignment and the presence of a pretest and a control group serve to control for all threats to internal validity.

39. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 39 True Experimental Designs The only potential weakness in this design is a possible interaction between the pretest and the treatment. Researchers should report assess and report the probability of a pretest-treatment interaction.

40. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 40 True Experimental Designs There are a few variations on the basic pretest-posttest control group design. One variation includes random assignment of matched pairs to the treatment groups. There is little advantage to this variation. Another variation of this design involves one or more additional posttests. ROX1OOROX1OO ROX2OOROX2OO

41. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 41 True Experimental Designs The posttest-only control group design is the same as the pretest- posttest control group design except that it lacks a pretest. This design is often expanded to include more than two groups. RX1ORX1O RX2ORX2O

42. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 42 True Experimental Designs The posttest-only control group design is best used when there is likelihood of a pretest-treatment interaction threat. As with the pretest-posttest control group design, the addition of a matched random assignment does not represent an increased advantage.

43. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 43 True Experimental Designs The Solomon Four-Group Design is a combination of the pretest-posttest control group design and the posttest- only control group design. ROX1OROX1O ROX2OROX2O RX1ORX1O RX2ORX2O

44. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 44 True Experimental Designs The analysis of the Solomon four-group design is a 2 x 2 factorial analysis of variance. This analysis tests if those who received the treatment performed differently than those who did not. This analysis can assess if there is a testing effect. This analysis assesses for pretest-interaction effects.

45. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 45 True Experimental Designs The Solomon four-group design requires a large number of participants. The Solomon four-group design may not always be the best design. The design selected should be based upon potential threats and the nature of the proposed study.

46. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 46 Quasi-Experimental Designs When it is not possible to assign participants to groups randomly, researchers can use quasi-experimental studies. In the nonequivalent control group design, two or more treatment groups are pretested, administered a treatment, and posttested. OX1OOX1O OX2OOX2O

47. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 47 Quasi-Experimental Designs The nonequivalent control group design involves the random assignment of groups not individuals. The lack of random assignment introduces validity threats (e.g., regression, and selection interaction effects). To reduce threats when using this design researchers often assure groups are as equivalent as possible (e.g., use ANCOVA).