1. Job Corps: Lessons from U.S. Programs for At-Risk, Disadvantaged Youth Peter Z. Schochet, Ph.D. Sciences Po, Paris April 15, 2016

2. 1 Need Ongoing Research to Support Evidence-Based Policies Need rigorous evidence to improve social programs and policies Need rigorous evidence to improve social programs and policies Need to know “what works and for whom” to allocate scare resources efficiently Need to know “what works and for whom” to allocate scare resources efficiently

3. 2 The Use of Impact Evaluations Has Increased in the U.S. Example: 925 evaluations of education interventions have been rated as rigorous Example: 925 evaluations of education interventions have been rated as rigorous Congressional funding of U.S. programs often mandates rigorous evaluations Congressional funding of U.S. programs often mandates rigorous evaluations The White House Budget Office requires federal agencies to: The White House Budget Office requires federal agencies to: “improve program performance … using experimentation and innovation to test new approaches to program delivery”

4. 3 Purpose of Presentation Discuss a rigorous impact evaluation of Job Corps, the largest education and training program in the U.S. for disadvantaged youth Discuss a rigorous impact evaluation of Job Corps, the largest education and training program in the U.S. for disadvantaged youth  What is Job Corps and how does it operate?  What was the study design?  What did we learn?  How did the results affect policy?

5. 4 Four Take-Away Points 1. A well-designed evaluation can teach us a lot 2. The Job Corps model has promise 3. Intensive interventions are needed for this population 4. Need longer-term impacts  Impacts could increase later in life  Occurred in some other programs (Head Start)

6. What Is Job Corps?

7. 6 Job Corps Was Established in 1964 “For the million young men and women who are out of school and who are out of work, this program will permit us to take them off the street, put them into work training programs, to prepare them for productive lives, not wasted lives.” - President Lyndon B. Johnson, 1964 - President Lyndon B. Johnson, 1964

8. 7 Key Features of Job Corps Serves disadvantaged youths ages 16 to 24 Serves disadvantaged youths ages 16 to 24 Primarily a residential program Primarily a residential program Provides training, education, and other services in centers Provides training, education, and other services in centers Administered by the U.S. Department of Labor (DOL) Administered by the U.S. Department of Labor (DOL)

9. 8 Job Corps Is Large and Expensive Serves 70,000 per year Serves 70,000 per year  More than 2 million since 1964 120 centers nationwide 120 centers nationwide Costs $1.5 billion per year Costs $1.5 billion per year  60 percent of all DOL funds spent on youth education and training  $20,000 per participant

10. 9 JOB CORPS CENTERS, BY REGION Indicates Job Corps Center

11. 10 Characteristics of Job Corps Applicants (Percentages) Gender Male 59 Female, no kids 29 Female with kids 12 Age 16-17 42 18-19 32 20-24 27 Race White 27 Black 48 Hispanic 18 No High School Credential 77 Avg. Highest Grade Completed 10 th Ever Arrested 27 Received Welfare 53

12. 11 Key Eligibility Criteria: Some Subjective Legal U.S. resident Legal U.S. resident Meets low income criteria Meets low income criteria Has a child care plan (if relevant) Has a child care plan (if relevant) Can benefit from additional training to complete school or find a job Can benefit from additional training to complete school or find a job Has no serious behavioral problems Has no serious behavioral problems Has no upcoming court appearances Has no upcoming court appearances Does not use illegal drugs Does not use illegal drugs

13. How Does Job Corps Operate?

14. 13 Services Delivered in Three Stages 1. Outreach and admissions  Recruit  Screen for eligibility  Assign youth to centers 2. Center services 3. Placement services

15. 14 Public-Private Partnership 100% federally funded by DOL 100% federally funded by DOL Mostly run by private contractors Mostly run by private contractors

16. 15 Center Characteristics Range in size (200 to 3,000 places) Range in size (200 to 3,000 places) Located in rural and urban areas Located in rural and urban areas

17. 16 87 Percent of Students Are Residents Participants typically enroll in centers in their home state Participants typically enroll in centers in their home state Students stay for 8 months on average, but it varies Students stay for 8 months on average, but it varies Zero tolerance policies Zero tolerance policies Nonresidents tend to be females with children in urban areas Nonresidents tend to be females with children in urban areas

18. 17 Vocational and Academic Center Services: Individualized and Self-Paced Vocational training: classroom and work experience Vocational training: classroom and work experience  100 trades with input from business and labor organizations  Emphasize specific competencies Academic education Academic education  Remedial education, GED, high-school diploma  Uniform, computer-based curriculum

19. 18 Other Center Services Counseling Counseling Social skills and parenting classes Social skills and parenting classes Health and dental care Health and dental care Student government Student government Recreation Recreation

20. 19 How Does Job Corps Differ from Other Education and Training Programs? Offers more comprehensive services Offers more comprehensive services More intensive and expensive: More intensive and expensive: $20,000 versus  $12,000 for high school  $10,000 for community colleges  $3,000 for other DOL youth programs Administered by DOL rather than state or local agencies Administered by DOL rather than state or local agencies  More uniform program structure and content

21. Design of Job Corps Evaluation

22. 21 Policy Context for Job Corps in the 1990s Only a few evaluations of training programs for disadvantaged youth had been conducted Only a few evaluations of training programs for disadvantaged youth had been conducted  These programs were nonresidential and less intensive than Job Corps  Results were disappointing (LaLonde, 2003) Job Corps under scrutiny from Congress Job Corps under scrutiny from Congress DOL issued a request for proposals to conduct a rigorous evaluation DOL issued a request for proposals to conduct a rigorous evaluation

23. 22 Job Corps Evaluation: Three Complementary Components 1. Impact Study: Does Job Corps make a difference?  Improve earnings?  Reduce criminal behavior?  Do effects differ for subgroups defined by age, gender, residential status? 2. Process Study: What is the Job Corps model? 3. Benefit-Cost Study: How do program benefits compare with its costs?

24. 23 DOL Design Regulations “Evaluations shall utilize sound statistical methods ….., including random assignment methodologies if feasible”

25. 24 What Is Random Assignment? Involves a lottery where: Involves a lottery where:  The “program group” is offered Job Corps services  The “control group” cannot enroll in Job Corps but can enroll in other programs in the community

26. 25 Why Random Assignment? Outcomes Are Not Evidence of Effectiveness  Suppose the earnings of Job Corps students increase after leaving the program  Does this mean that Job Corps worked?

27. 26 Control Group Provides a Benchmark Program group gets Job Corps services Control group can get other services or find jobs

28. 27 Random Assignment Ensures Balanced Treatment and Control Groups Gold standard for assessing whether an intervention causes the outcomes we observe Gold standard for assessing whether an intervention causes the outcomes we observe Allows for simple analysis and interpretation Allows for simple analysis and interpretation Other methods for obtaining comparison groups have been shown to be less reliable Other methods for obtaining comparison groups have been shown to be less reliable Common for studies funded by U.S. federal agencies for a range of populations Common for studies funded by U.S. federal agencies for a range of populations

29. 28 Is Random Assignment Ethical? Yes in the U.S. if: There are more potential participants than can be served by program funds There are more potential participants than can be served by program funds  Provides a fair way of allocating scarce program slots No individual is denied other services No individual is denied other services It helps build new evidence on program effects It helps build new evidence on program effects Participants are informed and provide consent Participants are informed and provide consent

30. 29 Random Assignment Is Used to Test New Drugs Shouldn’t we adopt a similar standard of rigor for testing and improving social programs and policies?

31. 30 The Job Corps Study Adopted a Random Assignment Design Methodologically the best Methodologically the best Supported by DOL and a strong national Job Corps director Supported by DOL and a strong national Job Corps director Determined feasible by program staff and advisory panel Determined feasible by program staff and advisory panel Rigorous design politically important Rigorous design politically important  Job Corps was under intense scrutiny and needed rigorous evidence of its effectiveness

32. 31 Which Locations Should Be Included? All Youth Nationwide All Youth Nationwide  Results generalizable to full program  Everybody shared in the burden  Few control group members in each location  Feasible due to strong national office Trained 1,500 intake staff in study procedures Trained 1,500 intake staff in study procedures

33. 32 One Year Rolling Intake Period Random assignment occurred nationwide in 1995 Random assignment occurred nationwide in 1995 81,000 eligible applicants randomly assigned across all recruiting offices 81,000 eligible applicants randomly assigned across all recruiting offices Randomization materials sent to our team Randomization materials sent to our team  Signed consent forms  Contact and baseline information forms

34. 33 Small Percentage Assigned to the Control Group Randomized groups: Randomized groups:  6,000 in control group: 7.4 percent of total  75,000 in program group: 9,400 followed  Sample sizes based on power considerations Job Corps given funds to increase recruitment to accommodate the control group Job Corps given funds to increase recruitment to accommodate the control group

35. 34 Random Assignment Worked 99.4 percent of Job Corps enrollees were randomized before enrolling 99.4 percent of Job Corps enrollees were randomized before enrolling 1.4 percent of controls enrolled 1.4 percent of controls enrolled Program and control groups are very similar on baseline measures Program and control groups are very similar on baseline measures

36. Data

37. 36 Data from Surveys and Tax Records Interviews at baseline, 12, 30, and 48 months after random assignment Interviews at baseline, 12, 30, and 48 months after random assignment  80 percent response rate at 48 months 81 percent for the program group 78 percent for the control group 78 percent for the control group Administrative annual earnings records covering years 1 to 9: Administrative annual earnings records covering years 1 to 9:  In Year 9, the sample was 25 to 33

38. 37 Outcomes from Survey Education and training Education and training Employment and earnings Employment and earnings Crime Crime Other outcomes: Not presented today Other outcomes: Not presented today  Welfare receipt  Fertility  Marriage  Drug and alcohol use  Health

39. 38 Impact Findings

40. 39 Job Corps Experiences of the Program Group

41. 40 The Program Group Participated in Job Corps for an Extended Period 73 percent enrolled 73 percent enrolled  Statistical methods were used to adjust for those who did not enroll Average length of stay was 8 months, but varied Average length of stay was 8 months, but varied  28% stayed less than 3 months  23% stayed more than 12 months

42. 41 Job Corps Provided Extensive Services to Participants 82 percent took academic classes 82 percent took academic classes 89 percent took vocational training 89 percent took vocational training Most participated in other core activities Most participated in other core activities

43. 42 Impacts on Education and Training: Survey Data: Years 1-4

44. 43 Study Counterfactual Is Substantial Educational Activity Measure Control Group Mean % Ever in Any Program 72 Average Hours 848

45. 44 Many Controls Attended Other Education and Training Programs *Difference is Significant at the 5% Level Percentage Attended Type of Program

46. 45 Large Impacts on Participation in Education and Training Programs Impact Per Impact Per Measure Program Control Eligible Participant % Ever in Any Program 93 72 21* 29* Average Hours 1,560 848 712* 989* * Significant at the 5% Level One year of school

47. 46 Impacts Were Large Early Percentage Ever in Education or Training in Quarter *Difference is Significant at the 5% Level In Program Period Post Program Period

48. 47 Large Impacts on the Receipt of GED and Vocational Certificates Percentage With Credential *Difference is Significant at the 5% Level ^For Those Without a HS Credential at Baseline

49. 48 Impacts on Crime: Survey Data: Years 1-4

50. 49 Job Corps Reduced Arrests, Convictions, and Incarcerations by 16 Percent *Difference is Significant at the 5% Level

51. 50 Reductions Found for Most Arrest Categories, But Mostly for Minor Charges % With ChargeProgram Control Impact Murder 0.4 0.5 -0.0 Assault 4.1 3.8 0.3 Robbery 2.1 2.2 -0.2 Burglary 2.7 3.0 -0.4 Larceny/Auto Theft 8.0 8.6 -0.6 Drug Law Violations 7.1 7.9 -0.7 Other Personal 5.2 5.4 -0.2 Other Misc. 16.6 19.5 -2.8* * Difference is Significant at the 5% Level

52. 51 Arrest Impacts Occurred Early, But Persisted Percentage Ever Arrested in Year *Difference is Significant at the 5% Level

53. 52 Impacts on Earnings

54. 53 Presentation of Findings Survey Survey  Years 1 to 4 Administrative earnings records Administrative earnings records  Years 1 to 4  Years 5 to 9

55. 54 Impacts on Earnings: Survey: Years 1 to 4

56. 55 12 Percent Earnings Gains After Two Years: Survey Data Average Earnings Per Week in Quarter (1995 $s) *Difference is Significant at the 5% Level $218 $199 In Program Period Post Program Period

57. 56 Positive Impacts on Employment Rates After Quarter 10: Survey Data Percentage Ever Employed in Quarter *Difference is Significant at the 5% Level 71.1% 68.7% In Program Period Post Program Period

58. 57 The Program Group Had Better Jobs In Quarter 16: Survey Data Earned $0.22 per hour more: Earned $0.22 per hour more:  $7.55 vs $7.33 More likely to have key fringe benefits More likely to have key fringe benefits  57% vs 54% had health insurance  48% vs 44% had pension benefits No differences in occupations No differences in occupations

59. 58 Earnings Impacts Are Similar Across Subgroups: Survey Data Positive impacts in years 3 and 4 by: Positive impacts in years 3 and 4 by:  Gender  Age  Race/ethnicity  Education level  Residential status  Center characteristics Notable exception: Notable exception:  Hispanics

60. 59 Impacts on Earnings: Administrative Records: Years 1 to 4

61. 60 Earnings Impacts Are Larger Using the Survey Data, But Patterns Are Similar Estimated Earnings Impacts Time Period Survey Tax Data 1996 (~Year 2) $ - 584* $ -177* 1997 (~Year 3) 292* 172* 1998 (~Year 4) 972* 220* *Difference is Significant at the 5% Level

62. 61 Impacts on Earnings: Administrative Records: Years 5 to 9 DID THE EARNINGS IMPACTS PERSIST?

63. 62 No, Except for the Oldest Students Administrative data show impacts declined sharply after year 4 Administrative data show impacts declined sharply after year 4  Overall  For most subgroups  Defied predictions by experts! Impacts did persist for older youths Impacts did persist for older youths

64. 63 Impacts Declined in 1999 to 2003 According to the Tax Data Impacts on Average Annual Earnings, by Data Source *Difference is Significant at the 5% Level Years 5-9 Years 1-2 Years 3-4

65. 64 Earnings Gains Persisted for Those 20 to 24 **Difference is Significant at the 5% Level * Difference is Significant at the 10% Level Annual SSA Earnings Impacts per Participant

66. 65 Why Did They Persist for Those 20 to 24? More motivated More motivated Better behaved Better behaved Fewer controls went back to school Fewer controls went back to school

67. 66 No Long-Term Earnings Gains for Residents or Nonresidents *Difference is Significant at the 5% Level Mean 2003 SSA Earnings, by Residential Status

68. Benefit-Cost Analysis

69. 68 Measured Benefits Increased output from the additional productivity of participants Increased output from the additional productivity of participants  Earnings plus the cost of fringe benefits Reduced use of programs and services Reduced use of programs and services  Other education and training programs  Public assistance (welfare, food stamps, etc.) Reduced crime Reduced crime

70. 69 Costs Program operating costs Program operating costs Capital costs Capital costs  Land, buildings, furniture, equipment used by Job Corps

71. 70 Benefits and Costs from Different Perspectives Society Society Taxpayers Taxpayers Program participants Program participants

72. 71 Costs to Society Exceed Benefits by $9,200 Per Participant Total Benefits $4,647 Increased Output $1,220 Increased Output $1,220 Other Programs and Services $2,186 Other Programs and Services $2,186 Reduced Crime $1,240 Reduced Crime $1,240 Total Costs -$13,844 Benefits – Costs -$9,197

73. 72 BUT….. Job Corps is cost-effective for those 20 to 24 at program application Job Corps is cost-effective for those 20 to 24 at program application  Benefits exceed costs by $17,000 Benefits exceed costs from the perspective of participants Benefits exceed costs from the perspective of participants

74. 73 What Have We Learned?

75. 74 The Job Corps Model Has Promise Provides an extra year of school Provides an extra year of school Increases the receipt of GED and vocational certificates Increases the receipt of GED and vocational certificates Reduces criminal activity by 16 percent Reduces criminal activity by 16 percent Only program shown to improve post- program earnings for this population Only program shown to improve post- program earnings for this population  Earnings gains sustained for older students Benefits exceed costs for participants Benefits exceed costs for participants

76. 75 But the Program Needs to Do More The earnings gains did not persist overall The earnings gains did not persist overall Benefits are less than program costs to society except for the 20- to 24-year olds Benefits are less than program costs to society except for the 20- to 24-year olds Challenge is to improve program services to sustain the earnings gains for the younger students Challenge is to improve program services to sustain the earnings gains for the younger students

77. 76 Need Longer-Term Follow-Up Earnings gains may be experienced later in life Earnings gains may be experienced later in life  Non-cognitive skills may be important as the sample matured  Intergenerational effects? Plan to conduct a longer-term study Plan to conduct a longer-term study  Sample is now 37 to 45

78. 77 How Did the Results Affect Policy?

79. 78 Job Corps Funding Has Increased First Report Released Second Report Released

80. 79 Policy Responses to Study Findings DOL sponsored a study to understand the lack of impacts for Hispanics DOL sponsored a study to understand the lack of impacts for Hispanics Job Corps remained a federal program rather than becoming a state program Job Corps remained a federal program rather than becoming a state program Efforts to improve placement services Efforts to improve placement services Efforts to improve services for those 16-17 Efforts to improve services for those 16-17  Some discussion about separate centers for them

81. 80 Thanks for Listening! pschochet@mathematica-mpr.com