1. The Disability Employment Initiative (DEI): Impact Evaluation Design October 21, 2015 Sung-Woo Cho, Ph.D.

2. Abt Associates | pg 2 Introducing Myself  Associate at Abt Associates  Project Director of the DEI impact evaluation  Most of my work is in impact evaluation of programs in community colleges and K-12  Past work has been in community college students and their outcomes  Teach applied statistics at The George Washington University

3. Abt Associates | pg 3 DEI Round 5 Overview  Six grantees in six different states  Intervention consists of services that are designed to assist people with disabilities who are seeking employment  Flagged DEI Round 5 participants: TREATMENT group  Using a rigorous evaluation design, we would want to compare these people with a comparison group, tracking their outcomes along the same points in time

4. Abt Associates | pg 4 Design Phase  Not randomized – we are using two quasi- experimental designs (QEDs) to determine impact of DEI interventions as a whole, and impact of a career pathways component (part of the intervention)  Outcomes include postsecondary credentials, employment, wages  Will incorporate a survey (Abt SRBI) to determine disability type and ADL (adult daily living) information  Treatment and comparison – at the LWIA level. Services provided at AJCs (One-Stops)

5. Abt Associates | pg 5 DEI Round 5 Interventions  The actual interventions across the six grantees vary  However, career pathways component is consistent throughout the grantees  Some of the interventions are based on previous rounds’ interventions  Examples of interventions for DEI Round 5: –Wrap-around services (South Dakota) –Remedial skill development (California) –Disabilities resource coordinator (Minnesota)

6. Abt Associates | pg 6 Quasi-Experimental Design (QED)  The basic idea is to match a treatment group of customers to a comparison group of similar customers –Match by using their characteristics: gender, age, ethnicity, other demographic and wage information, and disability type –Pre-enrollment wage information would be very good to have, since there’s a good amount of variation here Wages are also an outcome of interest  In the end, you have a treatment group and comparison group of customers that look similar to one another on key characteristics – except only the treatment group received the DEI Round 5 services

7. Abt Associates | pg 7 Comparison Groups  Our calls with the grantees indicated that the participants in other local LWIAs could serve as our comparison group  All else equal, similar participants in the comparison LWIAs received a different set of services compared to participants in treatment LWIAs  We ruled out trying to create a comparison group directly from community and technical colleges –The data are difficult to collect directly from these institutions, and general population may be hard to match

8. Abt Associates | pg 8 Disability Type and Matching  Comparison group – would need to match on people with self-reported disabilities  The treatment LWIAs will have a participant tracking system (PTS) that will collect information on disability type  However, the comparison LWIAs will not have a PTS –Solution: A survey that will collect information on disability type among comparison LWIA participants who disclose a disability

9. Abt Associates | pg 9 QED and Baseline Equivalence  Once you have a treatment group and comparison group that look similar to one another, we want to measure their baseline (that is, pre-intervention) characteristics –Ex: using wage information prior to start of the intervention to measure baseline equivalence  If you can show that the treatment and comparison groups are very similar at baseline, you can look at the difference in the two groups’ outcomes to measure the impact of the intervention

10. Abt Associates | pg 10 QED Visual Outcome Time Start of services Treatment Comparison Baseline equivalence established Impact estimate

11. Abt Associates | pg 11 QED Matching Strategy 10 Treatment Customers 30 Comparison Customers

12. Abt Associates | pg 12 Match Based on Characteristics 10 Treatment Customers 30 Comparison Customers

13. Abt Associates | pg 13 20 Comparison Customers are Not Matched (in red) 10 Treatment Customers 20 Comparison Customers

14. Abt Associates | pg 14 And They are Left Out of the Sample in a 1:1 Matching Strategy 10 Treatment Customers 10 Comparison Customers

15. Abt Associates | pg 15 Important Notes on Matching  The matching is done at the person (participant) level –We should use information about the LWIAs when we do the matching – but keep in mind that we are not matching LWIAs themselves –However, we keep in mind geography of the LWIAs  Each analysis would first be run within each state –We will eventually group the six states together, to determine the overall impact of the Round 5 interventions

16. Abt Associates | pg 16 Measuring Outcomes  Outcomes will be measured by WIASRD and Wagner-Peyser reported outcomes  These can include, but are not limited to, the following: –Employment –Wages –Credential completion (self-reported)

17. Abt Associates | pg 17 Potential Limitations  Interventions vary in strength and have different approaches to operationalizing career pathways programs. Grantees typically focus on adding services/components to existing services.  May be difficult to detect impacts on outcomes given nature of the interventions  Small sample sizes could contribute to issues in power –Makes it harder to determine if impacts are actually present

18. Abt Associates | pg 18 Why is this Important?  People with disabilities who are seeking employment – an important group to help  DOL has demonstrated this with large financial support Want to figure out if the initiatives are making an impact on education and employment outcomes  Survey information will give us more info on disability type, which is not collected in enough detail at AJCs  Through the evaluation, we are setting up data collection systems that will allow for easier information retrieval on this population and the types of services they receive

19. Abt Associates | pg 19 Concluding Remarks  A rigorous QED is possible, assuming that we will have information on an appropriate comparison group of non-Round 5 customers in other local WIAs  We will use data from WIASRD and Wagner-Peyser, with information on demographic characteristics and outcomes  We would use a matching strategy to create a comparison group for each treatment group –A survey will help collect information on comparison group disability type  Implementation study will be important for understanding how to provide career pathways programs to this population