1. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 1 Chapter 10: Experimental Research Objectives: Briefly state the purpose of experimental research and list the basic steps involved in conducting and controlling an experiment. Briefly define internal validity and describe eight major threats to the internal validity of an experiment.

2. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 2 Chapter 10: Experimental Research Objectives: Briefly define external validity and describe six major threats to the external validity of an experiment. Identify and briefly describe five ways to control extraneous variables.

3. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 3 Chapter 10: Experimental Research Objectives: Define and provide examples of single- variable designs (i.e., pre-experimental, true experimental, and quasi- experimental designs) and factorial designs, and explain how they differ.

4. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 4 Experimental Research Experimental research is the only type of research that can test hypotheses to establish cause-effect relations. The researcher manipulates at least one independent variable and controls other relevant variables, and observes the effect on one or more dependent variables. The researcher manipulates the treatment. The researcher has control over selection and assignment.

5. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 5 Experimental Research In experimental research studies the independent variable is also called the treatment, causal, or experimental variable. In experimental research studies the dependent variable is also called the criterion, effect, or posttest variable.

6. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 6 Experimental Research Experimental research is the most structured of all research. When conducted well, experimental research studies can provide evidence for cause-effect relations. Several experimental studies taken together can provide support for generalization of results.

7. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 7 Experimental Research The experimental process The steps in the experimental research process are the same as in other types of research. Selecting and defining a problem Selecting participants and measuring instruments Preparing a research plan Executing procedures Analyzing the data Formulating conclusions

8. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 8 Experimental Research The experimental process In experimental studies, the researcher controls selection and assignment. Experimental studies often examine comparisons between or among groups. Comparison of approaches (A versus B) Comparison of an approach to an existing approach (A versus no A) Comparison of different amounts of a single approach (A little of A versus a lot of A)

9. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 9 Experimental Research The experimental process In experimental research studies the group that receives the treatment is the experimental group and the group that does not receive the treatment is called the control group. Sometimes groups are comparison groups that receive alternative treatments (e.g., two types of instruction in a content area).

10. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 10 Experimental Research There are numerous challenges for experimental studies in educational settings. A lack of sufficient exposure to treatments (i.e., treatments are too short or diffuse). Failure to make treatments significantly different from one another (e.g., an experimental instructional program in math may not be different enough from the comparison math instructional program).

11. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 11 Experimental Research Manipulation and control In experimental studies, researchers control or remove the influence of extraneous variables. Participant variables Organismic (e.g., age) Intervening (e.g., interest) Environmental variables (e.g., school or teacher effects)

12. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 12 Threats to Experimental Validity Internal validity refers to the degree to which observed differences in the dependent variable are a direct result of manipulation of the independent variable and not some other variable. Internal validity is concerned with rival explanations for an effect.

13. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 13 Threats to Experimental Validity External validity, sometimes referred to as ecological validity, is the degree to which the results from a study are generalizable to other groups. When researchers increase the internal validity of their study, they decrease their external validity.

14. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 14 Threats to Experimental Validity When researchers are concerned with external validity, their ability to control important extraneous variables suffers. When there is a choice, researchers should err on the side of control and maximize internal validity.

15. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 15 Threats to Internal Validity History Maturation Testing Instrumentation Statistical regression Differential selection of participants Mortality Selection-maturation and interactive effects

16. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 16 Threats to Internal Validity History Any event occurring during a study that is not part of the experimental treatment but that may effect the dependent variable represents a history threat. Longer-lasting studies are more prone to history threats.

17. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 17 Threats to Internal Validity History threat example In a study of the effects of instructional simulations in learning chemistry content, a history threat would be demonstrated if students in the study were exposed to simulations in a different setting, such as when learning geography, while the study was being conducted.

18. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 18 Threats to Internal Validity Maturation Maturation refers to physical, intellectual, and emotional changes that naturally occur within participants over a period of time.

19. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 19 Threats to Internal Validity Maturation threat example In studies of interventions that are designed to increase children’s theory of mind, if the interventions lasted more than a couple of weeks at critical time points, participants may gain critical theory of mind awareness simply due to cognitive development and not due to the treatment.

20. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 20 Threats to Internal Validity Testing Testing as a threat to internal validity is demonstrated when taking a pretest alters the result of a posttest.

21. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 21 Threats to Internal Validity Instrumentation Instrumentation is a threat to internal validity when the instrumentation is either unreliable or is changed between pre- and posttesting.

22. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 22 Threats to Internal Validity Statistical regression Extremely high or low scores tend to regress to the mean on retesting. Statistical regression example If students perform poorly on a pretest it is difficult to determine if the gain in their scores is due to treatment effects.

23. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 23 Threats to Internal Validity Differential selection of participants Participants in the control and experimental groups differ in ways that influence the dependent measure.

24. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 24 Threats to Internal Validity Mortality Participants drop out of the study at differential rates across conditions.

25. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 25 Threats to Internal Validity Selection can interact with other threats to internal validity (i.e., history, maturation, instrumentation). Participants selected into the treatment and control conditions have different experiences or maturation rates or instrumentation varies across conditions.

26. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 26 Threats to External Validity External validity threats can be divided into two categories: ‘Generalizing to whom’ threats Threats affecting groups to which the study can be generalized ‘Generalizing to what’ threats Threats affecting the settings, conditions, variables, and contexts to which the results can be generalized

27. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 27 Threats to External Validity Pretest-treatment interaction Multiple-treatment interference Selection-treatment interaction Specificity of variables Experimenter effects Reactive arrangements

28. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 28 Threats to External Validity Pretest-treatment interaction This threat occurs when participants respond differently to a treatment because they have been exposed to a pretest. Pretest may alert participants. Self-report measures are often susceptible to pretest-treatment interaction effects.

29. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 29 Threats to External Validity Pretest-treatment interaction At times, unobtrusive measures can be used as pretests, to limit this threat to validity (e.g., using previously administered standardized assessments to measure ability in science instead of using a pretest).

30. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 30 Threats to External Validity Multiple-treatment interference This threat occurs when previous treatments cross-over into a current experiment. This makes it challenging to determine the effectiveness of the later treatment. This threat may occur in studies that access participants who have been exposed to other research studies (e.g., university participant pools).

31. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 31 Threats to External Validity Selection-treatment interaction When a study’s findings only apply to the groups selected and are not representative of other groups. This may happen in non-randomly assigned studies where a treatment is less or more effective for certain demographics (e.g., ability levels).

32. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 32 Threats to External Validity Specificity of variables When researchers do not adequately define their variables, instruments, or population, it makes it difficult to determine how well the findings will generalize to an alternative population.

33. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 33 Threats to External Validity Experimenter effects Experimenter effects occur when characteristics or behaviors of the experimenter influence the participants’ responses.

34. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 34 Threats to External Validity Reactive arrangements These threats are also referred to as participant effects. These threats are associated with differences in participants’ behavior, feelings, and attitudes because they are in a study.

35. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 35 Threats to External Validity Reactive arrangements Hawthorne effect: Any situation in which participants’ behavior is affected because they are in a study. John Henry effect (Compensatory rivalry): Members of the control group compete with the experimental group.

36. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 36 Threats to External Validity Reactive arrangements Placebo effect: To combat compensatory rivalry, researchers attempt to give control groups a placebo, not the experimental treatment, but something to decrease the perception that they are in the control group. Participants should perceive they are all getting the same thing. Novelty effect: When participants are engaged in something different this may increase attention, interest, behavior, learning, etc., just because it is something new.

37. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 37 Group Experimental Designs The validity of an experiment is a function of the degree to which extraneous variables are controlled. Randomization is the best mechanism to control for extraneous variables. Randomization distinguishes experimental designs. Randomization should be used whenever possible. If groups cannot be randomly formed, variables should be held constant when at all possible (e.g., time of day, which researcher is present).

38. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 38 Group Experimental Designs Participant variables can be controlled and held constant Matching can equate groups through random assignment of pairs. Comparing homogeneous groups allows the researcher to control for extraneous variables.

39. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 39 Group Experimental Designs Participants can serve as their own controls to control for participant differences. Analysis of covariance (ANCOVA) is a statistical procedure that can be used to control for participant variables.

40. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 40 Types of Group Designs Single-variable designs are any design that involves one manipulated variable. Pre-experimental designs do not adequately control for extraneous variables and should be avoided. True-experimental designs offer a very high degree of control and are always preferred designs. Quasi-experimental designs do not control as well as experimental designs but are preferable over pre-experimental designs.

41. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 41 Types of Group Designs Factorial designs are any design that involves two or more independent variables, at least one that is manipulated.

42. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 42 Pre-Experimental Designs The one-shot case study involves a single group that is exposed to a treatment (X) and then posttested (O). XOXO Threats to validity are not adequately controlled with this design. Do not use this design.

43. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 43 Pre-Experimental Designs The one-group pretest-posttest design involves a single group that is pretested, exposed to treatment, and then tested again. OXOOXO The success of the treatment is determined by comparing pretest and posttest scores. This design does not control for history, testing, instrumentation, regression, or maturation. Statistical regression is not controlled nor is pretest-treatment interaction.

44. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 44 Pre-Experimental Designs The static-group comparison design involves at least two nonrandomly formed groups. One group receives an experimental treatment and the other group receives the traditional treatment. Both groups are posttested. X1OX1O X2OX2O

45. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 45 Pre-Experimental Designs The number of groups can be expanded beyond two. The groups are better described as comparison, not experimental and control. This design does not control for maturation, selection effects, selection interactions, and mortality.

46. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 46 Pre-Experimental Designs There is some control for history in this design. This design is sometimes used in exploratory studies.

47. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 47 True Experimental Designs The pretest-posttest control group design requires at least two groups. Groups are formed by random assignment. Both groups are administered a pretest, each group receives a different treatment and both groups are posttested. The design may be extended to include additional groups.

48. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 48 True Experimental Designs ROX1OROX1O ROX2OROX2O ROX3OROX3O The combination of random assignment and the presence of a pretest and a control group serve to control for all threats to internal validity.

49. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 49 True Experimental Designs The only potential weakness in this design is a possible interaction between the pretest and the treatment. Researchers should report assess and report the probability of a pretest-treatment interaction.

50. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 50 True Experimental Designs There are a few variations on the basic pretest-posttest control group design. One variation includes random assignment of matched pairs to the treatment groups. There is little advantage to this variation. Another variation of this design involves one or more additional posttests. ROX1OOROX1OO ROX2OOROX2OO

51. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 51 True Experimental Designs The posttest-only control group design is the same as the pretest- posttest control group design except that it lacks a pretest. This design is often expanded to include more than two groups. RX1ORX1O RX2ORX2O

52. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 52 True Experimental Designs The posttest-only control group design is best used when there is likelihood of a pretest-treatment interaction threat. As with the pretest-posttest control group design, the addition of a matched random assignment does not represent an increased advantage.

53. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 53 True Experimental Designs The Solomon Four-Group Design is a combination of the pretest-posttest control group design and the posttest- only control group design. ROX1OROX1O ROX2OROX2O RX1ORX1O RX2ORX2O

54. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 54 True Experimental Designs The analysis of the Solomon four-group design is a 2 x 2 factorial analysis of variance. This analysis tests if those who received the treatment performed differently than those who did not. This analysis can assess if there is a testing effect. This analysis assesses for pretest-interaction effects.

55. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 55 True Experimental Designs The Solomon four-group design requires a large number of participants. The Solomon four-group design may not always be the best design. The design selected should be based upon potential threats and the nature of the proposed study.

56. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 56 Quasi-Experimental Designs When it is not possible to assign participants to groups randomly, researchers can use quasi-experimental studies. In the nonequivalent control group design, two or more treatment groups are pretested, administered a treatment, and posttested. OX1OOX1O OX2OOX2O

57. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 57 Quasi-Experimental Designs The nonequivalent control group design involves the random assignment of groups not individuals. The lack of random assignment introduces validity threats (e.g., regression, and selection interaction effects). To reduce threats when using this design researchers often assure groups are as equivalent as possible (e.g., use ANCOVA).

58. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 58 Quasi-Experimental Designs The time-series design is an elaboration of the pretest-posttest design. One group is repeatedly pretested until pretest scores are stable. The group is then exposed to a treatment and after treatment is repeatedly posttested.

59. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 59 Quasi-Experimental Designs O O O O X OO O O A variation of the time-series design is the multiple time-series design that includes a control group. This variation eliminates the history and instrumentation threats. O O O O X 1 OO O O O O O O X 2 OO O O

60. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 60 Quasi-Experimental Designs History is a threat with time-series designs. Instrumentation may also be a threat if testing changes. Pretest threats are problematic with time- series designs. However, it is relatively easy to establish the degree of the threat given data from repeated testing.

61. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 61 Quasi-Experimental Designs In a counterbalanced design, all groups receive all treatments but in a different order and all groups are posttested after each treatment. Counterbalanced designs can include any number of groups. The number of groups is equal to the number of treatments. Treatment order is randomly assigned.

62. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 62 Quasi-Experimental Designs A unique threat with a counter-balanced design is multiple treatment interaction. X1OX2OX3OX1OX2OX3O X3OX1OX2OX3OX1OX2O X2OX3OX1OX2OX3OX1O

63. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 63 Factorial Designs Factorial Designs are elaborations on single-variable experimental designs to permit investigation of two or more variables, at least one of which is manipulated by the researcher. Factorial designs are often employed after an independent variable has first been investigated individually.

64. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 64 Factorial Designs The purpose of a factorial design is to determine whether the effects of an independent variable are generalizable across all levels. One example of a factorial design is the 2 X 2 design. Type of instruction (computer-based or paper and pencil) by gender.

65. Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved. 65 Factorial Designs Many factors (independent variables) studies are possible to address specific research questions.