1. Pushing Forward from BHR through Random Individual-Level Variation in Program Components within Sites: The HPOG Impact Study Stephen H. Bell APPAM Research Conference November 7, 2013

2. Abt Associates | pg 1 Levels and Mechanisms of Program Variation in an RCT Context MECHANISM LEVELRandomNatural SiteCluster randomization Multi-level estimation Multi-level explanatory models (e.g., BHR) IndividualThree-way within-site randomization Standard experiment- al estimation Analysis of symmetrically predicted endogenous subgroups (ASPES; Peck, 2003; Bell & Peck, 2013)

3. Abt Associates | pg 2 Strengthen multi-level explanatory model using three-way random assignment of individuals in a subset of sites (e.g., Health Profession Opportunity Grants evaluation) _______________________________________ 1. Omitted variable bias in BHR 2. How third experimental arm removes bias for the randomized element 3. How third experimental arm reduces bias for BHR- style cross-site comparisons 4. Implications / next steps Goal and Outline

4. Abt Associates | pg 3 HPOG and Its Impact Evaluation  Career Pathways framework-based training for TANF and low-income individuals to pursue healthcare sector careers  HPOG-Impact is part of a rich research “portfolio” at ACF  Impact Evaluation involves an experimental design, with randomization of eligibles to control and treatment groups, with randomization to enhanced treatment in some locations

5. Abt Associates | pg 4 Study Sample and Data Collection  Sample size –Individuals: about 10,500 overall: 7,000 T; 3,500 C –Study sites: 38 study sites programs across 20 grantees –Planned variation sample (TBD) Peer support Emergency financial assistance Non-cash incentives  Data collection –At baseline (before RA), from PRS & supplement –Quarterly wage data (NDNH) –Follow-up surveys at 15 months post-randomization –Implementation study site visits –Grantee, staff/management and other surveys

6. Abt Associates | pg 5 Two-Level Model of Site-Level Determinants of Intervention Impacts

7. Abt Associates | pg 6 Omitted-Variable Bias Concern

8. Abt Associates | pg 7 Two-Level Model with Three-Arm Random Assignment

9. Abt Associates | pg 8 How the Third Arm Removes Bias

10. Abt Associates | pg 9 Using the Experimental Evidence to REDUCE the Bias Risk in Non-Experimental Estimates

11. Abt Associates | pg 10 A)Vary as many individual program features as possible through 3-arm random assignment within sites (e.g., HPOG) B)Better yet... –get many dozens of sites –randomize program features across sites Use A to approximate B by adding lots of natural variation sites to a small number of three-arm random assignment sites ______________________________________ Next: Determine statistical power of the design for HPOG Impact Then: Determine whether site-focused methods can support / enhance individual-focused methods like ASPES... or vice versa Implications for Design / Extensions

12. Abt Associates | pg 11 Levels and Mechanisms of Program Variation in an RCT Context MECHANISM LEVELRandomNatural SiteCluster randomization Multi-level estimation Multi-level explanatory models (e.g., BHR) IndividualThree-way within-site randomization Standard experiment- al estimation Analysis of symmetrically predicted endogenous subgroups (ASPES; Peck, 2003, Bell & Peck, 2013)

14. Further Information Molly Irwin Federal Project Officer, HPOG HHS/ACF/OPRE molly.irwin@acf.hhs.gov Stephen Bell Principal Scientist Abt Associates Inc. stephen_bell@abtassoc.com

15. Further Information Molly Irwin Federal Project Officer, HPOG HHS/ACF/OPRE molly.irwin@acf.hhs.gov Stephen Bell Principal Scientist Abt Associates Inc. stephen_bell@abtassoc.com