1. 2014 Welfare Research and Evaluation Conference Washington, D.C. A Statistical Portrait of Populations in Nine Career Pathways Programs: Preliminary Findings David J. Fein With David Judkins, Azim Shivji, & Peter Honnef Abt Associates, Inc

2. What is ISIS?  Random assignment evaluation of nine career pathways programs –Programs vary, but all promote access to and completion of one or more post-secondary education steps, target economically disadvantaged youth & adults –Nine separate studies set within a common conceptual framework  ISIS is sponsored by the Office for Planning, Research and Evaluation at ACF –With support for programs from Open Society Foundations, Joyce Foundation, Kresge Foundation, Meadows Foundation, OFA/ACF (3 HPOG sites) –RA ends late 2014; early impact results in 2015-16  Today preview preliminary findings from a forthcoming working paper on the ISIS sample 2

3. Programs ISIS Is Testing  Des Moines Area Community College (DMACC) “WTA Connect”  Integrated Basic Education and Skills Training (I-BEST) Program (Washington State: Bellingham, Everett, Whatcom colleges)  Instituto del Progreso Latino, Chicago, “Carreras en Salud”  Madison Area Technical College (MATC) Patient Care Pathway Program  Pima Community College “Pathways to Healthcare Program” (Tucson, AZ)  San Diego Workforce Partnership (SDWP) “Bridge to Employment in the Healthcare Industry”  Workforce Development Council of Seattle-King County (WDC) “Health Careers for All”  Valley Initiative for Development and Advancement (VIDA) Program, South Texas  Year Up (8 offices nationally) 3

4. Paper Addresses Three Descriptive Questions  Who volunteers for diverse ISIS programs? –Basic characteristics, measured through forms administered at study intake  To what degree does this population access post-secondary training on its own? –Analyze patterns of college enrollment and persistence in the control group  What characteristics predict college enrollment and persistence absent ISIS programs? –“Road test” ISIS baseline covariates (various implications for later impact analysis) –Identify potential mediators and subgroups that programs may want to target 4

5. Baseline Characteristics of ISIS Sample Enrolled Through March 2014 CharacteristicDMACCI-BESTInstitutoMATCPimaSDWPVIDAWDCYear Up Explicitly Target on: Economic Disadvantage?YesNo Yes Low skill?Yes No YesNo Economic Disadvantage Income under $15,000 (%)564935274953516639 Received WIC/SNAP (%)676043356848688234 Received TANF/welfare (%)1521548206447 Financial hardship (%)644839345954686131 Educational Background Parents attended college (%)284316524344264256 Mostly As in high school (%)671661920192210 Mostly Cs/below in h.s. (%)655068423736543635 Some college, <1 year (%)101113251619151623 Some college, 1+ year (%)81018222623532521 Associates degree+ (%)4810614175183 Sample size 684526598499121710069294192007

6. Baseline Characteristics of ISIS Sample Enrolled Through March 2014 (Cont.) CharacteristicDMACCI-BESTInstitutoMATCPimaSDWPVIDAWDCYear Up Explicitly Target on: Youth?No Yes Race-ethnicity/ geography?No LatinoNo TX borderNoUrban Age Under 25 (%)313645442132362099 25-34 (%)2730333032 41451 35+ (%)42342225473523350 Race-ethnicity Hispanic origin (%)17279995646961230 Black, non-Hispanic (%)448021122215255

7. Other Characteristics  Working paper provides similar comparisons in other domains of career pathways theory of change: –Psycho-social factors –Career knowledge –Time constraints –Personal and life challenges  Explores how characteristics in these domains predict college enrollment and persistence 7

8. College Outcomes by Site  Most ISIS programs focus on college, a few promote college as well as other training options  College outcomes for control group  “where is the bar”?  Analyze college records from match to the National Student Clearinghouse –The only national source of individual-level data on college enrollments and completions –Covers >90% enrollments nationally –Captures enrollments and completions –Under-coverage of students: at for-profits, refusing to share records, subject to matching error –Little/no evaluation of data quality for: certificates, non- credit enrollments, FT/PT status 8

9. Percent of Control Group Members Ever Enrolled in College Since Random Assignment

10. Average Total Months Control Group Members Enrolled in College Since Random Assignment

11. What Characteristics Predict College Enrollment and Persistence Absent ISIS?  Focus on key substantive domains in career pathways theory of change  Sample: control group pooled X-site, RA’d by 12/13  Two outcomes (measured through 3/14) –Any college enrollment after random assignment (logistic regression, n=3,018) –Number of months enrolled after random assignment (linear model, n=1,387 ever-enrolled)  Adding each domain as a set to base model (site dummies, months exposed, demographic characteristics—estimates not shown)  Bold = estimate statistically significant at p<.10 11

12. Estimates for Educational Background 12 Domain, Covariates (Domain Added as Set to Base Model) Ever Enrolled (Logit) Months Enrolled Educational background Either parent attended college -0.0660.610 Usual high school grades (vs. mostly Cs or below) Mostly As 0.5220.339 Mostly Bs 0.3440.464 Educational attainment (vs. high school/below) Some college, less than 1 year 0.758-0.426 Some college, 1 year + 1.3000.874 Associates degree+ 0.8450.021 Vocational certificate -0.120-0.301

13. Estimates for Psycho-Social Factors and Career Knowledge 13 Domain, Covariates (Each Domain Added Separately to Base Model) Ever Enrolled (Logit) Months Enrolled Psycho-social Student Readiness Indices (ACT Inc.) Academic discipline (10 items) -0.1170.501 Training commitment (10 items) 0.277-0.098 Academic self-confidence (12 items) 0.2070.195 Emotional stability (12 items) -0.225-0.375 Social support (10, Cutrona & Russell 1987) -0.195-0.360 Stress (4, Cohen et al. 1983) -0.271-0.413 Depression (9, CES-D) -0.0930.424 Career knowledge Career knowledge index (6 items, new) 0.1660.064

14. Estimates for Financial and Time Constraints 14 Domain, Covariates (Each Domain Added Separately to Base Model) Ever Enrolled (Logit) Months Enrolled Resources: financial constraints Income last year (vs. $30K+) $0-14K -0.265-0.545 $15-29K -0.442-0.497 Received WIC/SNAP -0.066-0.702 Received TANF/welfare -0.092-0.343 Financial hardship -0.2770.136 Resources: time constraints Current work hours (vs. <20) 20-34 0.2530.153 35+ 0.2680.118 Expected work hours (vs. <20) 20-34 -0.260-0.385 35+ -0.739-0.689 Expect to attend school part time -0.249-0.103

15. Estimates for Life Challenges 15 Domain, Covariates (Each Domain Added Separately to Base Model) Ever Enrolled (Logit) Months Enrolled Life challenges Interfered with school, work, family in last 12 months Child care arrangements -0.042-0.303 Transportation 0.046-0.290 Alcohol/drug use -0.3320.857 Illness/health condition -0.106-0.321 Arguments with family members 0.0090.049 Physical threats/violence from family 0.0130.385 Ever arrested -0.125-0.036

16. Preliminary Highlights (I)  Factors predicting both enrollment and persistence –Good high school grades (+), 1+ yr college (+), self- confidence (+), stress (-), low income (-), expected work hours (-)  Factors more predictive of enrollment –<1 year college/AA (+), training commitment (+), career knowledge (+), financial hardship (-), current work hours (-), substance abuse (-)  Factors more predictive of persistence –Parents attended college (+), <1 year college/vocational certificate (-), academic discipline (+), social support (-?), depression (+?), gov’t assistance (-), child care, transportation, illness (-), substance abuse (+?) 16

17. Preliminary Highlights (II)  Most findings in expected direction, a few unexpected  “Kitchen sink” models (not shown) leave site differences mostly unexplained –Unmeasured influences (local economy, training opportunities…)? –Specification and measurement error? 17

18. Summary  ISIS programs target diverse economically disadvantaged populations  Control groups vary substantially in college enrollment and persistence after random assignment—fairly high bar in several sites  Implications for targeting and program focus  More in paper (psychometrics, characteristics*site, college outcome detail)  Suggestions welcome! 18

19. For Further Information  David Fein (Abt), david_fein@abtassoc.comdavid_fein@abtassoc.com  Brendan Kelly (ACF), brendan.kelly@acf.hhs.govbrendan.kelly@acf.hhs.gov  Molly Irwin, (ACF), molly.irwin@acf.hhs.govmolly.irwin@acf.hhs.gov 19