1. WREC May 29, 2014 Washington, D.C. Learning “What Works” in Career Pathways Programming: The ISIS Evaluation. David Judkins Abt Associates, Inc

2. Abstract 2  Introductory review of new methodology for the study of the mediating pathways in the context of randomized experiments of social interventions –Pathways may contain both exposure to program components and early outcomes of that exposure –Other talks in this session talk about randomizing to program components –This talk is about how to analyze when randomization of components is not selected as the evaluation methodology  Plans to apply this new methodology to ISIS

3. What is ISIS?  Random assignment evaluation of nine “career pathways” programs –Impact, implementation, cost-benefit studies  Evaluation funded by ACF; Abt Associates heading research team –Additional funding for program enhancements/scale up from Open Society Foundations, Joyce Foundation, Kresge Foundation, Meadows Foundation, OFA/ACF (3 HPOG sites)  RA still underway; early impact results in 2-3 years  Programs vary, but all promote access to and completion of post-secondary education, targeting low-income, low-skilled adults and youth  Conducting nine separate studies –Overlap in research questions, measurement, analysis plans 3

4. Core Ideas of Career Pathways Programs  Address the wide range of skill and other needs of economically disadvantaged participants  Key “inputs” or elements: Assessment (academic and non-academic); basic and vocational skills training; supports; employment connections  Create manageable, well-articulated training steps  Provide credentials valued in high demand occupations/sectors  Build effective partnerships –Community colleges, employers, CBOs, WIBs 4

5. Challenge: Getting inside the “Black Box”  How to determine more effective program components? –Counseling –Tutoring –Financial assistance –Internships  How to determine intermediate outcomes on vital causal paths? –Self-efficacy –Strength of social network –Basic numeracy and literacy skills –Stress management skills 5

6. Benefits  Effective components are built into future interventions  Ineffective components are dropped  New components are invented to impact vital early outcomes even more strongly  Future evaluation findings can be obtained more quickly if good early indicators are available 6

7. Familiar but Inadequate  Baron and Kenny (1986) mediation triangle  The indirect effect of treatment on Y mediated by M estimated as ab.  The direct effect of treatment on Y is d. 7

8. Why Do We Need Better Methodology?  Baron and Kenny assume independence of errors in two equations.  This will be violated if there are any common causes of the two errors such as: –Measured baseline characteristics –Unmeasured baseline characteristics –Other measured mediators –Other unmeasured mediators  Also, Baron and Kenny does not generalize well to categorical outcomes 8

9. Pearl’s Method  Judea Pearl has developed a much more powerful and general method.  However, he developed it in a graphical framework unnatural to statisticians in the Neyman-Rubin tradition.  Recently, Kosuke Imai and co-authors have recast it in the language and traditions of Neyman-Rubin.  Dramatically increased popularity  Still very difficult literature to penetrate  Sketch some features today  We plan to use on ISIS 9

10. Pearl’s Method Capabilities 10 ChallengePearl can Handle Confounding due to measured moderators Confounding due to other measured mediators Confounding due to unmeasured moderators Confounding due to unmeasured other mediators Nonnormal outcomes such as binary outcomes

11. ISIS Nomenclature  Blend of nomenclatures proposed by Pearl and by Imai and coworkers.  Key is the concept of forcing an environmental stimulus while “blocking” one type of response to the stimulus  Three potential outcomes per person Y0: person randomly assigned to control Y1: person randomly assigned to treatment and no changes are blocked Y2: person randomly assigned to treatment but change in mediator M is “blocked” 11

12. Three Potential Outcomes 12

13. Example  Y0 = person’s degree attainment at 36 months if assigned to control  Y1 = person’s degree attainment at 36 months if assigned to treatment and no “natural” changes are “blocked”  Y2 = person’s degree attainment at 36 months if assigned to treatment but somehow we blocked change to self-confidence 13

14. Potential Outcomes Framework for Mediation  Total effect = average of Y1-Y0  Indirect effect mediated by M = average of Y1-Y2  Direct effect = average of Y2-Y0  Then total = indirect + direct  Continuing example: –Y1-Y0 = total effect of treatment on degree attainment –Y1-Y2 = indirect effect of treatment on degree attainment via boosted self-confidence –Y2-Y0 = direct effect of treatment on degree attainment  Note: no need to reference linear models to define the estimands 14

15. Mediating Pathways  In ISIS, we are interested in multiple mediating pathways –Trying to decide which are the vital pathways for successor programs to emulate  Multiple mediation can take two forms: –Serial (T causes M1 causes M2 causes Y) –Parallel (T causes both M1 and M2, both of which jointly cause Y)  Pearl’s framework flexible enough to handle simultaneous parallel and serial mediation 15

16. Service Exposure and Early Outcomes  Both are mediators  Let D1 and D2 be exposure levels to two program components  Let W1 and W2 be two early outcomes  Treatment causes D1 and D2 to change, which causes W1 and W2 to change (in addition to direct effects of T on them)  All of which lead to changes in Y 16

17. Example Graph for Parallel and Serial Mediation 17

18. Estimation Process  Build a series of models for every mediator and outcome in the system in terms of causally prior variables  Draw simulated values from system for each pathway of interest –What happens if some services are blocked and some early outcomes are blocked  Pathways with highest simulated Y values are the most promising for future program developers to consider 18

19. Assumptions  Strong assumptions required  Considerable debate about best way to express them  Oversimplified version –Variation in M within each level of Y –No unmeasured prime joint causes of M and Y –This includes post-randomization latent outcomes 19

20. Example Violation  M = Self-confidence  Y = Degree attainment  Academic skill at unmeasured intermediate point mediates treatment effects on both self-confidence and degree attainment  Easy to come to the mistaken conclusion that shortcut methods work that build self-confidence through means other than skill improvement  Need to measure skill to prevent this incorrect mediational inference 20

21. Challenges  Curse of dimensionality  Need to identify graphs to be tested that correspond to program-specific theories of change  Also need to believe that we have measured the necessary set of variables to unconfound the mediator(s) and outcome(s) in the graph of interest 21

22. Additional Information David Judkins Principal Scientist Abt Associates David_judkins@abtassoc.com Website www.projectisis.org Brendan Kelly Federal Co-Project Officer, ISIS ACF/OPRE Brendan.Kelly@ACF.hhs.gov Molly Irwin Federal Co-Project Officer, ISIS ACF/OPRE Molly.Irwin@ACF.hhs.gov