1. Abt Associates | pg 1 Using Impact Evaluation Tools to Unpack the Black Box and Learn What Works Laura R. Peck Principal Scientist, Abt Associates Inc. presentation to American Evaluation Association Conference panel on “Getting Real About Causality and Complexity: Evaluation Methods and Approaches for Complex Interventions” Washington, DC, October 19, 2013

2. Abt Associates | pg 2 Agenda  Using Randomization to Identify Endogenous Subgroups (primer on Peck, 2003)  Programmatic & Temporal Complexity  Responses to the Complexity Problem –Continuously-measured mediators –Multiple mediators –Complexly-measured mediators –Design options

3. Abt Associates | pg 3 Using Randomization… When exposed to treatment… used program feature Z (or not) experienced high dosage of intervention followed treatment path W-X-Y behaved a particular way If exposed to treatment, would have… used program feature Z (or not) experienced high dosage of intervention followed treatment path W-X-Y behaved a particular way Treatment Group Control Group

4. Abt Associates | pg 4 Using Randomization… In the absence of the treatment… dropped out of school (or not) experienced long-term unemployment had less favorable LM outcomes behaved a particular way If not exposed to treatment, would have… dropped out of school (or not) experienced long-term unemployment had less favorable LM outcomes behaved a particular way Control Group Treatment Group

5. Abt Associates | pg 5 In Practice  Step 1: Use baseline (exogenous) characteristics to predict subgroup membership –To capitalize on the internal validity of the experimental design, use prediction subsample (to ensure symmetric identification of T and C subgroups)  Step 2: Estimate impacts on predicted subgroups –Impacts estimates are unbiased  Step 3: Convert estimated impacts for predicted subgroups to represent actual subgroups –Conversion rests (1) on an assumption of the homogeneity of impacts among those predicted to be in a subgroup; and (2) on the foundation of an experimental impact estimate

6. Abt Associates | pg 6 The Motivating Problem  Programmatic complexity (a snap-shot) –Theory –Administration –Implementation –Program design –Participants –Targets  Temporal complexity (the movie)

7. Abt Associates | pg 7 Program Complexity as Cafeteria On the buffet (what to offer): results of the administrative and management decisions about what to offer and how to offer it In the kitchen (the chefs): funding, administrative and management decisions about moving theory (design and operational possibilities) into the field (and on to the buffet) Local management: at the buffet, decide what to bring to their program enrollees and targets need, to bring to the dining room/table Targets: getting in in line to come to a table Participants: in the dining area, at the table, consuming the varied programs that local management assembles from the buffet

8. Abt Associates | pg 8 Possible Responses  Continuously-measured mediators  Multiple mediators  Complexly-measured mediators  Design options –multi-arm (for programmatic complexity): A, B, C v. A, A+B, C –mega-multi-arm (for programmatic & temporal complexity) –why not factorial for these questions?

9. Contact: Laura R. Peck Principal Scientist Laura_Peck@abtassoc.com Laura_Peck@abtassoc.com (301) 347-5537