1. A Framework for Studying Variation in Program Effects June 14, 2015 Michael J. Weiss Based on: Weiss, M. J., Bloom, H. S., & Brock, T. (2014). A Conceptual Framework for Studying the Sources of Variation in Program Effects. Journal of Policy Analysis and Management, 33(3). 1

2. Outline Section I: Introduction Section II: Proximal Sources of Variation in Program Effects Section III: Connecting the Treatment Contrast to Program Implementation & Treatment Fidelity 2

3. Section 1 Introduction 3

4. Consequences of Effect Variation Targeting services (types of clients / locations) Equity / fairness Best case / worst case scenarios Generalizability / Scaling Research design – What research questions to address? – Sample size – Etc. 4

5. Three Motivating Studies After School Programs – certain types “work” (sequenced activities, active learning, limited # of goals, activities tied to goals) Charter Schools – certain types “work” (clear behavior standards, zero- tolerance, coaching new teachers) Welfare-To-Work – certain types “work” (counseling to get jobs quick, personalized attention) – context mattered (low unemployment rate  larger effects) 5

6. DEFINING A PROGRAM EFFECT FOR AN INDIVIDUAL Section 1: Introduction 6

7. Weight Loss Program No Program Potential Outcome Program Effect On Neyman 7

8. ESTIMATING THE AVERAGE PROGRAM EFFECT FOR A GROUP OF INDIVIDUALS Section 1: Introduction 8

9. Estimating the Average Program Effect Randomize individuals (or groups) to: 1.Receive program offer 2.Not receive program offer Compare two group’s average future outcomes 9

10. DEFINING VARIATION IN PROGRAM EFFECTS Section 1: Introduction 10

11. Weight Loss Program No Program Potential Outcome Program Effect On Neyman Weight Loss Program No Program Potential Outcome Program Effect On Rubin 11

12. The Framework Outcomes Treatment Contrast Program Effect 12 Treatment Offered Take-up Treatment Offered Treatment Fidelity Implementation Process Client Characteristics and Context Characteristics of Implementing Organization Implementation Plan Treatment Planned Program Model Take-up Outcomes Treatment Received

13. Section 2 Proximal Sources of Variation in Program Effects 13

14. The Three C’s 1.Treatment Contrast 2.Client Characteristics 3.Program Context 14

15. 1. THE TREATMENT CONTRAST Section 2: Proximal Sources of Variation in Program Effects 15

16. Weight Loss Program No Program Potential Outcome Program Effect On Neyman Contrast 1 Neyman Weight Loss Program Alternate Program Potential Outcome Program Effect On Neyman Contrast 2 Neyman 16

17. Treatment Contrast and Program Effect Treatment Received Outcomes Treatment Received Treatment Contrast Program Effect 17 Client Characteristics and Context Characteristics of Implementing Organization Treatment Fidelity Implementation Process Implementation Plan Treatment Planned Program Model Treatment Offered Take-up Treatment Offered Treatment Fidelity Implementation Process Implementation Plan Treatment Planned Program Model Outcomes - Employment - Achievement - Recidivism - etc. Outcomes - Employment - Achievement - Recidivism - etc. Treatment Received - Content - Quantity - Quality - Conveyance Mediators Treatment Received - Content - Quantity - Quality - Conveyance Mediators

18. 2. CLIENT CHARACTERISTICS Section 2: Proximal Sources of Variation in Program Effects 18

19. Weight Loss Program No Program Potential Outcome Program Effect On High-risk Type High-Risk Weight Loss Program No Program Potential Outcome Program Effect On Low-risk Type Low-Risk 19

20. Client Characteristics as Moderators Treatment Received Outcomes Treatment Contrast Program Effect Outcomes Treatment Received 20 Client Characteristics and Context Characteristics of Implementing Organization Treatment Fidelity Implementation Process Implementation Plan Treatment Planned Program Model Treatment Offered Take-up Treatment Offered Treatment Fidelity Implementation Process Implementation Plan Treatment Planned Program Model

21. Author’s Suggestions (Client Characteristics as Moderators) 1.Risk E.g., pre-test, pre-RA income level 2.Readiness E.g., acknowledgment of problem (substance abuse) 3.Program Theory E.g., age (adult education that relies on technology) 21

22. 3. PROGRAM CONTEXT Section 2: Proximal Sources of Variation in Program Effects 22

23. Weight Loss Program No Program Potential Outcome Program Effect On Neyman Context 1 Neyman Weight Loss Program No Program Potential Outcome Program Effect On Neyman Context 2 Neyman 23

24. Context as a Moderator Treatment Received Outcomes Treatment Received Treatment Contrast Program Effect 24 Client Characteristics and Context Characteristics of Implementing Organization Treatment Fidelity Implementation Process Implementation Plan Treatment Planned Program Model Treatment Offered Take-up Treatment Offered Treatment Fidelity Implementation Process Implementation Plan Treatment Planned Program Model

25. Section 3 Connecting the Treatment Contrast to Program Implementation and Treatment Fidelity 25

26. FROM TREATMENT OFFERED TO TREATMENT RECEIVED: CLIENT TAKE-UP Section 3: Connecting the Treatment Contrast to Program Implementation and Treatment Fidelity 26

27. Client Take-Up Treatment Received Outcomes Treatment Received Treatment Contrast Program Effect 27 Client Characteristics and Context Characteristics of Implementing Organization Treatment Fidelity Implementation Process Implementation Plan Treatment Planned Program Model Treatment Offered Take-up Treatment Offered Treatment Fidelity Implementation Process Implementation Plan Treatment Planned Program Model

28. Author’s Suggestions (Strategies to Increase Client Take-Up) 1.Incentives / Inducements ASAP’s MetroCard tied to attending advising sessions 2.Requirements / Mandates Opening Doors required student success course 3.Remove Barriers H&R Block’s FAFSA simplification 401(k) saving plans “opt-out” 4.Improved Outreach High school to college transition, summer outreach 28

29. FROM TREATMENT PLANNED TO TREATMENT OFFERED: PROGRAM IMPLEMENTATION Section 3: Connecting the Treatment Contrast to Program Implementation and Treatment Fidelity 29

30. Program Implementation Treatment Received Outcomes Treatment Received Treatment Contrast Program Effect 30 Client Characteristics and Context Characteristics of Implementing Organization Treatment Fidelity Implementation Process Implementation Plan Treatment Planned Program Model Treatment Offered Take-up Treatment Offered Treatment Fidelity Implementation Process Implementation Plan Treatment Planned Program Model 1 1 2 2 3 3

31. Treatment Plan Definition: Planned content, quantity, quality & conveyance of treatment services for clients The plans for “what, how much, how, and by whom?” of treatment Varies in specificity 31 1 1

32. Implementation Plan Definition: Set of instructions for how the treatment plan is to be realized Worth Considering: Staff recruitment and selection Staff training Staff monitoring and supports 32 2 2

33. Implementing Organization (Author’s Suggestions ) 1.Strong Leadership 2.Resources / Capacity 3.Supportive Culture / Climate 4.Involvement of Outside Monitor or “Fixer” 33 3 3

34. Treatment Fidelity Treatment Received Outcomes Treatment Received Treatment Contrast Program Effect 34 Client Characteristics and Context Characteristics of Implementing Organization Treatment Fidelity Implementation Process Implementation Plan Treatment Planned Program Model Treatment Offered Take-up Treatment Offered Treatment Fidelity Implementation Process Implementation Plan Treatment Planned Program Model

35. A Framework for Studying Variation in Program Effects Michael J. Weiss Howard S. Bloom Thomas Brock michael.weiss@mdrc.org howard.bloom@mdrc.org thomas.brock@ed.govmichael.weiss@mdrc.orghoward.bloom@mdrc.orgthomas.brock@ed.gov 35