1. Experimental Design making causal inferences

2. Causal and Effect The IV precedes the DV in time The IV precedes the DV in time The IV and DV are correlated The IV and DV are correlated There are no plausible additional variables that could reasonably explain the correlation There are no plausible additional variables that could reasonably explain the correlation

3. The Gold Standard What are the differences between these types of studies? What are the differences between these types of studies? True experiments True experiments Quasi experiments Quasi experiments Observational studies Observational studies Surveys Surveys Why is it so hard to do in education? Why is it so hard to do in education?

4. The Gold Standard True experiments versus observational studies, or quasi- experiments True experiments versus observational studies, or quasi- experiments The Randomized Clinical Trial model The Randomized Clinical Trial model Why is it so hard to do in education? Why is it so hard to do in education?

5. The Gold Standard True experiments True experiments –Random assignment to treatment and control conditions –The Gold Standard for causal inferences –The treatment is under the control of the researcher

6. The Gold Standard Quasi experiments Quasi experiments –The treatment is under the control of the researcher –As many of the characteristics of a true experiment as possible –Intact groups, random assignment is not possible

7. The Gold Standard Observational studies Observational studies –Intact groups, not randomly assigned –If there is a treatment, it may not be under the control of the researcher –Correlations between variables, not causality

8. The Gold Standard Surveys Surveys –The focus is on estimation of parameters, not the effects of a treatment –Snap shot in time –Self-report, subjective perceptions of respondents, not objective or external –Not designed to provide hard outcomes

9. The Essential Characteristics Random Selection Random Selection Random Assignment Random Assignment Manipulation of the IV Manipulation of the IV Control Condition Control Condition

10. Additional Factors to Consider Placebo or Comparison Group(s) Placebo or Comparison Group(s) Multiple Measurements Over Time Multiple Measurements Over Time Control Over Confounding Variables Control Over Confounding Variables

11. Controlling Confounds The experimental environment The experimental environment Admissibility criteria Admissibility criteria Blocking Blocking

12. Blocking Creates homogeneous subsets Creates homogeneous subsets Builds potential confounds into the design Builds potential confounds into the design Reduces error term Reduces error term Makes a more sensitive and therefore powerful experiment Makes a more sensitive and therefore powerful experiment

13. Additional Design Features Measurement of Compliance or Implementation Effects Measurement of Compliance or Implementation Effects Blinding Blinding Measurement of Outcomes Measurement of Outcomes Integrity of Data Collection, Entry, and Reporting Procedures Integrity of Data Collection, Entry, and Reporting Procedures

14. Ethical Considerations Informed consent Informed consent Privacy Privacy Do no harm Do no harm Denying services Denying services Inducements Inducements Dual roles Dual roles

15. 2003 #4

16. Rubric for Part A Identify a plausible example of a problem Identify a plausible example of a problem –“Because a deadline has been moved back...” Relate the identified problem to the change in stress level Relate the identified problem to the change in stress level –“...the stress levels of those working in the department have been lowered...” State that the problem effects can not be distinguished from the treatment effects State that the problem effects can not be distinguished from the treatment effects –“...which could be mistakenly attributed to the treatment.”

17. Rubric for Part A Give a reason for the necessity of random assignment. Give a reason for the necessity of random assignment. State that randomization is relied upon to create comparable groups. State that randomization is relied upon to create comparable groups. State that randomization helps reduce the influence of potential confounding variables. State that randomization helps reduce the influence of potential confounding variables.

18. Rubric for Part A “Without random assignment of volunteers to the two programs, it is possible that the two treatment groups could differ in some way that affects the outcome of the experiment. Randomization “evens out” the possible effects of potentially confounding variables.” “Without random assignment of volunteers to the two programs, it is possible that the two treatment groups could differ in some way that affects the outcome of the experiment. Randomization “evens out” the possible effects of potentially confounding variables.”

19. Rubric for Part B Indicate that a control group does provide additional information Indicate that a control group does provide additional information Explain that the control group allows the company to determine if either or both treatments are effective in reducing stress Explain that the control group allows the company to determine if either or both treatments are effective in reducing stress Explain that the control group provides a baseline for comparison, an indication of what might have happened anyway, even without the treatment Explain that the control group provides a baseline for comparison, an indication of what might have happened anyway, even without the treatment

20. Rubric for Part B “Without the control group, the company could compare the two treatments, but would not be able to say whether the observed reduction in stress was attributable to participation in the programs. For example, a change in the work environment during this period might have reduced the stress level of all employees. The addition of a control group would enable the company to assess the magnitude of the mean reduction attributable to each treatment, as opposed to just determining if the two programs differ.” “Without the control group, the company could compare the two treatments, but would not be able to say whether the observed reduction in stress was attributable to participation in the programs. For example, a change in the work environment during this period might have reduced the stress level of all employees. The addition of a control group would enable the company to assess the magnitude of the mean reduction attributable to each treatment, as opposed to just determining if the two programs differ.”

21. Rubric for Part C Indicate that one cannot generalize, and give a plausible reason, such as... Indicate that one cannot generalize, and give a plausible reason, such as... The participants were volunteers and volunteers my not be representative of the population The participants were volunteers and volunteers my not be representative of the population The participants were not randomly selected from the population The participants were not randomly selected from the population

22. Rubric for Part C “No it is not, for this experiment we took volunteers but the problem with it is that the people who volunteered are very likely the ones who needed the stress reduction the most...Therefore, it is not reasonable to generalize because most likely the people who volunteered are not representative of the population.” “No it is not, for this experiment we took volunteers but the problem with it is that the people who volunteered are very likely the ones who needed the stress reduction the most...Therefore, it is not reasonable to generalize because most likely the people who volunteered are not representative of the population.”

23. Common Student Errors Did not understand the difference between random allocation of subjects and random sampling. Did not understand the difference between random allocation of subjects and random sampling. Often used the word "confounding" in part (a), but did not explain how the treatment results were mixed up with some other variable. Often used the word "confounding" in part (a), but did not explain how the treatment results were mixed up with some other variable.

24. Common Student Errors Seemed to think that a larger sample size would fix any problem in the experiment, rather than recognizing that the major problem of the experiment was that there was no random sampling of employees. Seemed to think that a larger sample size would fix any problem in the experiment, rather than recognizing that the major problem of the experiment was that there was no random sampling of employees. Incorrectly stated that random allocation "eliminates" bias. Incorrectly stated that random allocation "eliminates" bias.

25. 2001 #4

26. Rubric for Part A Blocking Scheme A is preferable Blocking Scheme A is preferable Creates homogeneous blocks with respect to forest exposure Creates homogeneous blocks with respect to forest exposure Plots will have similar forest exposure Plots will have similar forest exposure

27. Rubric for Part B Randomization within blocks should reduce bias due to the influence of confounding variables Randomization within blocks should reduce bias due to the influence of confounding variables (fertility of soil, moisture, etc.) (fertility of soil, moisture, etc.) on the productivity of the trees. on the productivity of the trees. 2001 FR #4 Rubric 2001 FR #4 Rubric

28. Extension Questions How would you randomize trees within blocks? How would you randomize trees within blocks? What other confounding variables might impact the results? What other confounding variables might impact the results?

29. Extension Questions How does this concept of blocking apply to educational evaluation? How does this concept of blocking apply to educational evaluation? Students Students Classrooms Classrooms Schools Schools