1. 1 YOUTHBUILD EVALUATION Building evidence about the effect of YouthBuild on the young people it serves August 17, 2011 Washington, D.C.

2. 2 Overview Project overview and rationale for study Research questions and study design Roles and responsibilities of organizations participating in the study Timetable for study activities Questions 2

3. 3 What is the YouthBuild Evaluation and who is involved? Large scale, random assignment study evaluating the effects of program participation on youth enrolled in 77 DOL and CNCS funded programs nationwide Sponsored by the U.S. Department of Labor and the Corporation for National and Community Service Research team: – MDRC – Social Policy Research Associates (SPR) – Mathematica Policy Research (MPR) 3

4. 4 Why a study of YouthBuild? Reengaging young high school dropouts is an important national policy challenge Rigorous studies of second chance programs like YouthBuild can help shape their services and policies for youth Previous YouthBuild studies very useful, but not designed to measure program effects 4

5. 5 What are the study benefits? Distinguishes YouthBuild as a model youth program to policymakers and local program operators Provides evidence that supports national policy and public funding Opens up the possibility of enrollment for some applicants who might otherwise be screened out 5

6. 6 What questions will the study answer? What are the effects or impacts of YouthBuild participation? How are YouthBuild services delivered in different settings and what are the characteristics of the youth who participate? What is the cost effectiveness of YouthBuild programs? 6

7. 7 What will we learn about program effects? How do the outcomes for youth enrolled in YouthBuild programs compare to similar youth? – On educational attainment, employment, earnings, civic engagement, criminal justice involvement? – Do the effects change over time? – Do some youth benefit more? Data sources: public records data, enrollment forms, and surveys at 12, 30 and 48 months 7

8. 8 What will we learn about program implementation? What services are provided and to whom? – How are services delivered? – What kinds of organizations provide services? – What are the characteristics of the youth? – What challenges are encountered and how are they addressed? – What alternative programs are available? Data sources: visits to programs, focus groups with participants, and program service data 8

9. 9 What will we learn about cost effectiveness? What is the cost of serving the average YouthBuild participant? How does the cost compare to the impacts generated by the program? Data sources: analysis of programs costs and participation 9

10. 10 What is random assignment? Random assignment is a research method that determines the benefit a program has for the individuals it serves Uses a lottery-like process to place individuals into a program group and a comparison group Assures that groups have similar characteristics, so the differences over time between the groups is the impact of the program 10

11. 11 Why use random assignment? Most reliable way to measure program effects Widely used in social service settings Endorsed by OMB, DOL, DOE and other federal and private agencies Fair and equitable way to determine who receives program services 11

12. 12 80 70 0 20 40 60 80 100 Ready to Work Percent Placed In A Job I Want A Job Program Group Comparison Group 40 0 20 40 60 80 100 Percent Placed In A Job 20 General example of random assignment results 12

13. 13 Example from National Guard Youth ChalleNGe Program Educational Attainment Outcome (%)ChalleNGeComparisonDifference Has High School diploma or GED71.855.516.2*** HS Diploma30.326.63.7 GED56.934.522.4*** Earned any college credit34.918.816.1*** Employment in the Past 12 Months Employed88.484.53.9* Earnings ($)13,51511,2482,266*** Number of months employed8.17.20.9*** 13

14. 14 Summary of random assignment steps YouthBuild group invited to mental toughness interviews Random Assignment Consent & Data Youth watch informational video and give consent Complete forms to record enrollment data and contact information Recruitment Applicants program staff deem eligible and appropriate are invited to participate in study Comparison group referred to other services in the community Youth receive study welcome letter and $10 gift card 14

15. 15 What is the program’s role? Designate a contact person to liaise with study team on progress and challenges Participate in training on the research procedures Recruit a sufficient pool of eligible applicants to make random assignment possible Administer the research procedures – Ensure applicants are informed about the study and consent to participate – Record enrollment data into research database Participate in survey of grantees and host the study team during the process study 15

16. 16 What is the research team’s role? Schedule meetings and site visits Tailor random assignment to your program Train program staff on research procedures Monitor applicant pool to ensure that there are no crossovers between YB and Non-YB comparison groups Collect and analyze research data Provide assistance throughout research Report findings to funder Disseminate findings to practitioners, policy-makers and others 16

17. 17 What is the coach’s role? Provide technical assistance and clarification to programs as needed. Potential issues include: – Challenges regarding recruitment – Comparison group – Data entry requirements – DOL performance standards 17

18. 18 When will programs be involved? Select 77 sites from pool of DOL and CNCS funded YouthBuild programs (2011) Visit all sites and begin to develop study enrollment plan (2011) Enroll approx. 4,600 youth in the study (2011-2012) Administer the Grantee survey (2012) Study program implementation (2012- 2013) 18

19. 19 QUESTIONS? 19

20. 20 Need more information? Please contact Sharon Rowser sharon.rowser@mdrc.org Cynthia Miller cynthia.miller@mdrc.org or Eileen Pederson pederson.eileen@dol.gov or visit www.mdrc.org