1. Evaluating Impact: Use of Yin’s Partial Comparisons Case Study Approach Michael S. Trevisan, trevisan@wsu.edu Jennifer E. LeBeau, jlebeau@wsu.edu Washington State University

2. Overview 1. Introduction 2. Partial Comparisons Approach (Yin, 1995; 2000) 3. Rival Explanations 4. Two Examples 5. Discussion

3. Introduction Randomized Controlled Trials (RCTs) given priority for impact evaluation Randomized Controlled Trials (RCTs) given priority for impact evaluation Examples for which RCTs are inappropriate for evaluating effectiveness: Examples for which RCTs are inappropriate for evaluating effectiveness: –Programs in the early stages of development with emerging or less defined outcomes –Programs with multi-dimensional outcomes –Programs whose outcomes will not occur in the near term

4. Introduction Many STEM projects funded by NSF are difficult to evaluate Many STEM projects funded by NSF are difficult to evaluate Introduction of a new strategy is imperative Introduction of a new strategy is imperative Partial Comparisons Approach (PCA) Partial Comparisons Approach (PCA)

5. Partial Comparisons Approach Developed by Yin (1995) Developed by Yin (1995) Seldom used to evaluate impact of educational programs but shows promise Seldom used to evaluate impact of educational programs but shows promise Aim: “multiple partial comparisons instead of imposing a singular research design in carrying out an evaluation” (p. 29) Aim: “multiple partial comparisons instead of imposing a singular research design in carrying out an evaluation” (p. 29) Possibly less expensive and more generalizable than RCTs Possibly less expensive and more generalizable than RCTs

6. Partial Comparisons Approach Suggested Partial Comparisons (Yin, 1995) Suggested Partial Comparisons (Yin, 1995) –Outcomes-only comparisons –Process-only comparisons –Causal interpretation –Rival interpretations –Policy analyses

7. Rival Explanations Strengthen impact claim Strengthen impact claim Two categories of rivals (Yin, 2000) Two categories of rivals (Yin, 2000) Real-life rivals (pp. 250-258) Real-life rivals (pp. 250-258) –Direct (practice or policy) rivals –Commingled (practice or policy) rivals –Implementation rivals –Rival theory –Super rivals –Societal rivals

8. Example One Math-Science Partnership, Building Science Teaching Capacity Math-Science Partnership, Building Science Teaching Capacity Need based on analysis of science test score data Need based on analysis of science test score data Five components Five components PCA and Rival Explanations PCA and Rival Explanations

9. Example One: Rival Explanations Component One Component One –Participants came prepared to work on strategic plans and, thus, were more productive and got more out of the SPI.

10. Example One: Rival Explanations Components Two, Three, and Four Components Two, Three, and Four –Teaching science in previous years has increased teachers’ knowledge, confidence, and skills.

11. Example One: Rival Explanations Component Five Component Five –Teachers learned the strategies in their formal education and are, therefore, more likely to use them.

12. Example One: Summary MSP’s performance trend is in the desired direction. MSP’s performance trend is in the desired direction. Further discussions with project personnel needed. Further discussions with project personnel needed. Policy analyses (Yin, 1995) Policy analyses (Yin, 1995)

13. Example Two Acceso la Ciencia (Access Science) Acceso la Ciencia (Access Science) NSF-funded NSF-funded Brings informal science activities to Latino communities in Eastern Washington Brings informal science activities to Latino communities in Eastern Washington PCA and Rival Explanations PCA and Rival Explanations

14. Example Two: Rival Explanations Interest and involvement in and awareness of STEM-related careers would have occurred regardless of a program implemented specifically for that purpose. Interest and involvement in and awareness of STEM-related careers would have occurred regardless of a program implemented specifically for that purpose. Participants had a prior knowledge of or interest in STEM-related careers. Participants had a prior knowledge of or interest in STEM-related careers.

15. Example Two: Summary Acceso la Ciencia made a difference Acceso la Ciencia made a difference Policy analyses Policy analyses Future discussions Future discussions

16. Discussion PCA is a viable alternative to the RCT method to evaluate STEM programs. PCA is a viable alternative to the RCT method to evaluate STEM programs. Future considerations Future considerations

17. Discussion Important Considerations Important Considerations –Important users of evaluation information may not come away with the same understanding. –Using the PCA to distill information without losing meaning is difficult.

18. Discussion Important Considerations Important Considerations –Partial comparisons need to be revisited in relation to the original project goals. –Rival explanations need to be further explored.

19. Thank You Evaluating Impact: Use of Yin’s Partial Comparisons Case Study Approach Michael S. Trevisan, trevisan@wsu.edu trevisan@wsu.edu Jennifer E. LeBeau, jlebeau@wsu.edu jlebeau@wsu.edu Washington State University PO Box 642136, Pullman WA 99164-2136 (509) 335-9117