1. 1/27 CRESST/UCLA The Long-Term Effects of After-School Programming on Educational Adjustment and Juvenile Crime: A Study of the LA’s BEST After-School Program Denise Huang, Pete Goldschmidt American Educational Research Association Annual Meeting New York – March 28, 2008 UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing

2. 2/27 CRESST/UCLA Purpose of the Study: There has been little long-term assessment of the effectiveness of ASP especially on juvenile crime The annual cost of juvenile crime is estimated to be $56.7 billion This study intends to fill this research gap Including a cost-benefit analysis of afterschool programs on the effects of juvenile crime. When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

3. 3/27 CRESST/UCLA Research Questions Is there a difference in the long-term educational outcomes of LA’s BEST participants in comparison with non-participants Is there a difference in the students’ rate of committing juvenile crime among LA’s BEST participants and non-participants What is the cost-effectiveness of LA’s BEST in terms of students’ long-term educational or juvenile crime hazard?

4. 4/27 CRESST/UCLA LA’s BEST LA’s BEST is a large urban-based ASP 180 sites within the LAUSD A student population of approximately 30,000 students Free program, serving students on a first-come, first-served basis Establishing sites in low-income, high crime, and low performance schools

5. 5/27 CRESST/UCLA Study Design  Quasi-experimental design  Longitudinal sampling of academic and juvenile crime data  Advanced multilevel propensity scores methods to establish study samples  Hierarchical growth modeling and survival analysis (multilevel discrete-time hazard)  Examined moderating factors such as gender, ethnicity, language proficiency, SES, and programmatic mediating factors  Students were followed from 1994-1995-2002- 2003 school years

6. 6/27 CRESST/UCLA Student Sample 2331 student participants from the LA’s BEST programs at 24 school sites 2331 non LA’s BEST participants from the same schools 1914 students who attended comparable schools (24 schools) with no LA’s BEST programs

7. 7/27 CRESST/UCLA Sampling Scheme by Cohort, Year, and Grade Sampling YearsFollow-up Years Cohort- Grade 91-92*92-93*93-9494-9595-9696-9797-98 98- 99** 99- 00** 00-0101-0202-03 I Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 II Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 III Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 IV Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 *There was no demographic information for this year. **There was no attendance information for this year.

8. 8/27 CRESST/UCLA Demographic Analysis We used 1990 and 2000 census data by zip code to compare schools’ demographic composition to the community Given strong correlations, we use census data as a proxy for resources available to students in a particular school We incorporated census based family income to set school economic context as a principle between school moderating variable.

9. 9/27 CRESST/UCLA Student Academic Samples Analysis Sample 1-comparing participating students to non-participating students in the 24 schools Analysis Sample 2- include all 48 sampled schools and compared participating students to non-participating students When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

10. 10/27 CRESST/UCLA Note of Caution During this 10 year span, the district used 3 un-equated assessments; the Comprehensive Test of Basic Skills, the Stanford Achievement Test v.9, and the California Achievement Test v.6 We used Normal Curve Equivalent scores Apply HLM analysis that capture both the fluctuations in student performances as well as the test change effect

11. 11/27 CRESST/UCLA Achievement Outcomes Inconclusive We examined exposure (number of years attending), intensity (daily attendance), engagement (weekly average), and fidelity (volunteers and professional development) Implied marginal, positive program effects on achievement Exposure demonstrated consistent positive effects for first post treatment year but the effect did not last When accounting for both exposure and intensity, intensity was a significant predictor of Reading achievement at the end of the treatment year The monthly number of volunteer hours was significant and positively related to Math and Reading linear achievement growth Positive diffusion effect

12. 12/27 CRESST/UCLA Sample for Juvenile Crime Number and Percentage of Juveniles in the Original and DOJ Data Sample sent to DOJ Office Sample with arrest info in DOJ % in DOJ Data LA's BEST23311847.9% Control I23311797.7% Control II1237967.8% Total58984597.8%

13. 13/27 CRESST/UCLA Distribution of the Samples by Gender and Ethnicity Original SampleArrested Sample Control IIControl ILA's BESTControl IIControl ILA's BEST %N%N%N%N%N% Female 64152%118051%115949.7%2021%5028%5228% Male 59648%115149%117250.3%7679% 129 72% 132 72% Hispanic 109689%188681%188981%8083% 118 66% 131 71% African American 12510%39017%38316%1314%5732%5027% Asian 71%221%311%00%0 11% Other 10%100%8 11%00%0 White 81%231%201%22%4 21% Total 1237100%2331100%2331100%96100% 179 100% 184 100%

14. 14/27 CRESST/UCLA General Findings Model 3 included student background characteristics Boys were 3 times as likely than girls to commit a crime Asians were predicted to commit crime at a significantly lower rate than White students Hispanics were also less likely to commit crime than their White classmates African American students were just as likely to commit crime as their white classmates Students with disabilities were estimated to commit crimes about 30% more often than their non-disabled classmates Students whose parents have less than a college education were about 25% more likely to commit crime than students of college educated parents

15. 15/27 CRESST/UCLA HLM Findings Similar to achievement models, a simple treatment indicator did not effectively capture the treatment effect Program quality, exposure, and engagement all needed to be considered in order to identify program effects Once engagement and exposure were properly parameterized, the results were extremely robust Results from the multilevel survival analyses indicated that LA’s BEST positively impacted juvenile crime survival probabilities Program effects remained consistent irrespectively of other concomitant student factors or school and neighborhood context factors that were included into the model The results of differential crime hazards was not found between LA’s BEST and non-LA’s BEST schools, but directly related to individual student participation in the program Note:-Engagement (low: 4-9 days, med:10-14 days, high: over 15 days per month )

16. 16/27 CRESST/UCLA Survival Probabilities for Treatment and Control groups

17. 17/27 CRESST/UCLA Effect of neighborhood poverty and low treatment engagement on survival probabilities

18. 18/27 CRESST/UCLA Summary of Juvenile Results LA’s BEST positively impacted juvenile crime probabilities The results is directly related to individual participation in the program Students who are actively and intensively engaged benefited the most While those who moderately engaged also benefited Those students who sporadically attended did not benefit from the program unless we considered mediating circumstances

19. 19/27 CRESST/UCLA Cost-Benefit Analysis Cost-actual program costs (include actual incurred costs as well as adult volunteer costs) using CPI to adjust to 1998 (end of treatment) $ value Benefits-derived from avoided costs that potentially occurred over the entire study period 3 sets of cost ratios: based on a single year of participation; based on average exposure; and based on each year of exposure separately Each scenario presents ratio for low, high, and lifetime crime estimates Note: Estimates of tangible costs for crimes based on the National Crime Victimization Survey

20. 20/27 CRESST/UCLA Cost-benefit Comparisons The benefits and costs of the intent to treat group (low engagement) were compared against the controls. As well as the effects of treatment on the treated (medium and high engagement) against the control and the low engagement group When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

21. 21/27 CRESST/UCLA Results of Cost-Benefit Analysis The analysis demonstrated LA’s BEST student and the larger society benefit from the program Cost-benefit ratios demonstrated extreme variability depending on assumptions The ratio ranged from-$40.76 to $68.81 Highlighted the importance of assumptions when deriving estimates It is important to focus on the most plausible

22. 22/27 CRESST/UCLA Benefit/Cost ratios Using sample average exposure and condition Cost Assumption Low High Life Sample Ave. Expected value vs. control (includes low, medium, and high engagement)-0.33-1.32-4.35-0.48 Expected value vs. control (includes medium, and high engagement)1.676.7322.162.46 Expected value vs. low engagement (includes medium, and high engagement)2.9111.6938.504.28

23. 23/27 CRESST/UCLA Implications of the Study More sensitive indicators of program implementation is needed to provide cleaner estimates of program effects and give sponsors a clearer picture of best practices Cost-benefit ratios are an important extension of traditional program evaluations, however, reasonable assumptions needs to be assessed Afterschool programs need to focus on engaging students, ensuring a minimum of 10 days of attendance per month Neighborhood poverty is at least as important as school context, programs should improve their outreach efforts in the most at-risk communities

24. 24/27 CRESST/UCLA Conclusion LA’s BEST generally present a positive relationship between achievement scores and intensity of attendance LA’s BEST demonstrated statistically positive effects on crime abatement, especially for students who attended over 10 days a month Each dollar spent on LA’s BEST returned a benefit of $2.50 to the society

25. 25/27 CRESST/UCLA To adjust the slide numbering, do the following: 1.Go to the VIEW menu, MASTER, and select SLIDE MASTER 2.In the lower right, change the number 27 to your number of slides 3.Do not change the character. It generates the auto-numbers.