1. Educational Action Research Todd Twyman Summer 2011 Week 1

2. Agenda Discuss Internal and External Validity Discuss Internal and External Validity Review Threats to Internal Validity Review Threats to Internal Validity Consider Precautions for Minimizing Threats to Internal Validity Consider Precautions for Minimizing Threats to Internal Validity

3. Why are you here? How can you use the results from your action research project to inform your practices? How can you use the results from your action research project to inform your practices? Why is action research important for improving schools? Why is action research important for improving schools?

4. The internal validity of a study relies on the researcher’s ability to rule out factors other than the intervention as possible causes of changes in the dependent variable. Internal Validity

5. Improving Internal Validity Maximize systematic variance Maximize systematic variance Make the groups as different as possible in regard to the intervention Make the groups as different as possible in regard to the intervention Minimize error variance Minimize error variance Reduce the differences between the participants Reduce the differences between the participants Control extraneous variance Control extraneous variance Manipulate the environment to maximize the controls Manipulate the environment to maximize the controls

6. Improving Internal Validity Focus is on trying to “control” extraneous factors (or confounds). Identify threats and then either delete or reduce threats to validity through your research design. Focus is on trying to “control” extraneous factors (or confounds). Identify threats and then either delete or reduce threats to validity through your research design.

7. Thinking about Internal Validity… What could have caused these findings? What could have caused these findings? What could have influenced the results besides the intervention? What could have influenced the results besides the intervention? What are alternative hypotheses for the results? What are alternative hypotheses for the results? What decisions would you make based on these results? What decisions would you make based on these results? What are the implications of the findings? What are the implications of the findings?

8. Threats to Internal Validity History History Maturation Maturation Testing Testing Instrumentation Instrumentation Statistical regression Statistical regression Differential selection Differential selection Experimental mortality Experimental mortality Selection-maturation interaction Selection-maturation interaction Experimental treatment diffusion Compensatory rivalry by the control group Compensatory equalization of treatments Resentful demoralization of the control group

9. Threats to Internal Validity History: Events may occur that influence the observed effects. History: Events may occur that influence the observed effects. Maturation: Physical/psychological changes that influence the observed effects. Maturation: Physical/psychological changes that influence the observed effects. Testing: Pre-test influences the post-test. Testing: Pre-test influences the post-test. Instrumentation: Changes in the measurement tool influence the observed effects. Instrumentation: Changes in the measurement tool influence the observed effects.

10. Identifying Threats A tenth-grade biology teacher institutes a new approach to the teaching of one- celled animal life based on a series of seven-minute single-concept films. Prior to starting the new unit, she develops a test covering the material and administers it to her class before and after the unit, noting with satisfaction a dramatic improvement by her pupils on the test. A tenth-grade biology teacher institutes a new approach to the teaching of one- celled animal life based on a series of seven-minute single-concept films. Prior to starting the new unit, she develops a test covering the material and administers it to her class before and after the unit, noting with satisfaction a dramatic improvement by her pupils on the test.

11. Threats to Internal Validity Statistical regression : Test scores move toward the mean. Statistical regression : Test scores move toward the mean. Differential selection : Participants are differentially selected into groups. Differential selection : Participants are differentially selected into groups. Attrition : Participants differentially drop out of the study. Attrition : Participants differentially drop out of the study. Selection-maturation interaction : Participants differ based on maturation. Selection-maturation interaction : Participants differ based on maturation.

12. Identifying Threats A school principal has received a federal grant to institute a new mathematics enrichment program. Using school records, he computes the average mathematics achievement score on a nationally standardized examination for his fifth-, ninth-, and twelfth-grade pupils during each of the preceding three years. He plans to have the same tests administered for the two years following the institution of the new program. He wishes to compare the relative performance of the three classes before and after the implementation of the new program. A school principal has received a federal grant to institute a new mathematics enrichment program. Using school records, he computes the average mathematics achievement score on a nationally standardized examination for his fifth-, ninth-, and twelfth-grade pupils during each of the preceding three years. He plans to have the same tests administered for the two years following the institution of the new program. He wishes to compare the relative performance of the three classes before and after the implementation of the new program.

13. Threats to Internal Validity Treatment diffusion : Control group seeks access to treatment. Treatment diffusion : Control group seeks access to treatment. Compensatory rivalry : Control group works harder to make up for group placement. Compensatory rivalry : Control group works harder to make up for group placement. Compensatory equalization : Outsiders provide other treatment to compensate. Compensatory equalization : Outsiders provide other treatment to compensate. Demoralization of control group: Control group performs worse due to discouragement. Demoralization of control group: Control group performs worse due to discouragement.

14. Identifying Threats A researcher uses a table of random numbers to subdivide 15 junior high school classes into three groups of five classes each (30 pupils per class) as follows: “Treatment A,” “Treatment B,” and “Control.” The treatments are administered during the first four months of the school year, and at mid-semester a posttest is given to all of the 450 pupils involved. A researcher uses a table of random numbers to subdivide 15 junior high school classes into three groups of five classes each (30 pupils per class) as follows: “Treatment A,” “Treatment B,” and “Control.” The treatments are administered during the first four months of the school year, and at mid-semester a posttest is given to all of the 450 pupils involved.

15. 3 Steps to Strengthening Treatment Fidelity Write precise specifications for dependent and independent variables. Write precise specifications for dependent and independent variables. TRAIN extensively to ensure treatment is delivered as intended. TRAIN extensively to ensure treatment is delivered as intended. Collect data on fidelity of treatment implementation. Collect data on fidelity of treatment implementation.

16. Internal Validity Review

17. Threats to Internal Validity 1. History (H) 2. Maturation (M) 3. Testing (T) 4. Instrumentation (I) 5. Statistical regression (SR) 6. Differential selection (DS) 7. Experimental mortality (EM) 8. Selection-maturation interaction (SMI) 9. Experimental treatment diffusion (ETD) 10. Compensatory rivalry by the control group (CR-control) 11. Compensatory equalization of treatments (CE) 12. Resentful demoralization of the control group (Rdem)

18. Control group works harder to make up for group placement Control group works harder to make up for group placement Pre-test influences the post-test Pre-test influences the post-test Control group performs worse due to discouragement Control group performs worse due to discouragement Test scores move toward the mean Test scores move toward the mean Events may occur that influence the observed effects Events may occur that influence the observed effects

19. Control group seeks access to treatment Control group seeks access to treatment Participants are differentially selected into groups Participants are differentially selected into groups Physical/psychological changes that influence the observed effects Physical/psychological changes that influence the observed effects Participants differentially drop out of the study Participants differentially drop out of the study

20. Participants differ based on maturation Participants differ based on maturation Changes in the measurement tool influence the observed effects Changes in the measurement tool influence the observed effects Outsiders provide other treatment to compensate Outsiders provide other treatment to compensate

21. Quiz

22. Steps to Reducing Threats to Validity Debrief with peers: identify strengths/ weaknesses/ biases. Debrief with peers: identify strengths/ weaknesses/ biases. Align data sources with research questions. Align data sources with research questions. Triangulate data sources. Triangulate data sources. Record data accurately and without bias. Record data accurately and without bias. Embed fidelity checks. Embed fidelity checks. Thoroughly describe the setting and study. Thoroughly describe the setting and study. Engage in continuous and ongoing reflection. Engage in continuous and ongoing reflection.

23. Read with an Informed Eye Assignment 3 will ask you to critique an Action Research article of your choice. To prepare: Look carefully at the methodology of research articles. Look carefully at the methodology of research articles. Evaluate the degree to which the authors have controlled threats to internal validity. Evaluate the degree to which the authors have controlled threats to internal validity. Question everything! Question everything!

24. Critiquing Research Are the problems and their educational importance explained clearly? Are the problems and their educational importance explained clearly? Does the literature review sufficiently set up a rationale for the need for this study? Does the literature review sufficiently set up a rationale for the need for this study? Are the research variables easily identified from the hypothesis or purpose statement? Are the research variables easily identified from the hypothesis or purpose statement?

25. Critiquing Research What instruments are used in the study? Are they appropriate for the research question and for the participants? What instruments are used in the study? Are they appropriate for the research question and for the participants? Is the research design appropriate to the research question? Is the research design appropriate to the research question? How does the study control for threats to internal or external validity? How does the study control for threats to internal or external validity? Are the conclusions logically related to the questions posed in the research question? Are the conclusions logically related to the questions posed in the research question?

26. Your own study Identify the goals for conducting your study Identify the goals for conducting your study Identify the types of validity that are applicable to your study (consider the purpose of the study, activities, audience, data sources) Identify the types of validity that are applicable to your study (consider the purpose of the study, activities, audience, data sources) Create a plan for reducing the threats to internal validity that is aligned with the types of validity addressed above. Create a plan for reducing the threats to internal validity that is aligned with the types of validity addressed above.