1. Chapin Hall Center for Children Measuring Changes in Reclaiming Futures Communities: National Evaluation Results Annual Meeting of the Coalition for Juvenile Justice Washington, DC June 11, 2007 Jeffrey A. Butts Chapin Hall Center for Children University of Chicago Views expressed are those of the author and should not be attributed to the University of Chicago, Chapin Hall Center for Children, their trustees or funders..

2. Chapin Hall Center for Children Reclaiming Futures  An initiative of Robert Wood Johnson Foundation  Focused on substance abuse interventions in juvenile justice  Principal strategy was organizational change/system reform  Evaluation challenge: Tracking change in 10 completely different service systems with 10 different change strategies

3. Chapin Hall Center for Children Surveys of Reclaiming Futures Communities  National evaluation tracked change with bi-annual surveys  Surveys measured quality of systems as reported by 20 to 40 expert informants in each community  Six surveys conducted in: 2003 – December 2004 – June and December 2005 – June and December 2006 – June  Response rate averaged 70 percent and ranged from 63% (December 2003) to 73% (December 2004)

4. Chapin Hall Center for Children Surveys of Reclaiming Futures Communities  58 questions about quality and effectiveness  All questions were brief statements  Respondents indicated whether they strongly disagreed, disagreed, were neutral, agreed, or strongly agreed  Answers coded so higher scores indicate positive opinions  Responses scored: -10 strongly negative -5 somewhat negative 0 neutral +5 somewhat positive +10 strongly positive

5. Chapin Hall Center for Children Surveys of Reclaiming Futures Communities 3 General Areas13 Indices ADMINISTRATIONAccess to Services Data Sharing Systems Integration Resource Management COLLABORATIONClient Information Partner Involvement Agency Collaboration QUALITYAOD Assessments Treatment Effectiveness Targeted Treatment Cultural Integration Family Involvement Pro-social Activities

6. Chapin Hall Center for Children Surveys of Reclaiming Futures Communities 3 General Areas13 Indices ADMINISTRATIONAccess to Services Data Sharing Systems Integration Resource Management COLLABORATIONClient Information Partner Involvement Agency Collaboration QUALITYAOD Assessments Treatment Effectiveness Targeted Treatment Cultural Integration Family Involvement Pro-social Activities The overall effectiveness of treatment was measured with five statements: In the past three months, … 1. The substance abuse treatment needs of youth in my community were adequately met. 2. The mental health needs of youth in my community were adequately met. 3. Graduated sanctions were used effectively to support treatment goals for youth. 4. Youth-serving agencies in my community generally did a good job serving youth. 5. Youth-serving agencies in my community were usually able to provide youth with the range of services they needed.

7. Chapin Hall Center for Children Surveys of Reclaiming Futures Communities 3 General Areas13 Indices ADMINISTRATIONAccess to Services Data Sharing Systems Integration Resource Management COLLABORATIONClient Information Partner Involvement Agency Collaboration QUALITYAOD Assessments Treatment Effectiveness Targeted Treatment Cultural Integration Family Involvement Pro-social Activities

8. Chapin Hall Center for Children Survey Results for Each Index

9. Chapin Hall Center for Children Access to Services ADMINISTRATION INDICES

10. Chapin Hall Center for Children Data Sharing ADMINISTRATION INDICES

11. Chapin Hall Center for Children Systems Integration ADMINISTRATION INDICES

12. Chapin Hall Center for Children Resource Management ADMINISTRATION INDICES

13. Chapin Hall Center for Children Client Information COLLABORATION INDICES

14. Chapin Hall Center for Children Partner Involvement COLLABORATION INDICES

15. Chapin Hall Center for Children Agency Collaboration COLLABORATION INDICES

16. Chapin Hall Center for Children AOD Assessments QUALITY INDICES

17. Chapin Hall Center for Children Treatment Effectiveness QUALITY INDICES

18. Chapin Hall Center for Children Targeted Treatment QUALITY INDICES

19. Chapin Hall Center for Children Cultural Integration QUALITY INDICES

20. Chapin Hall Center for Children Family Involvement QUALITY INDICES

21. Chapin Hall Center for Children Pro-social Activities QUALITY INDICES

22. Chapin Hall Center for Children 1AOD Assessment28% Yes 2Treatment Effectiveness26% Yes 3Client Information24% Yes 4Family Involvement22% Yes 5Systems Integration19% Yes 6Pro-social Activities18% Yes 7Data Sharing18% Yes 8Resource Management16% Yes 9Agency Collaboration14% Yes 10Targeted Treatment12% Yes 11Cultural Integration11% Yes 12Access to Services9% Yes 13Partner Involvement-2% No Change in scores across all Reclaiming Futures communities between December 2003 and June 2006, ranked by percentage of possible improvement realized Survey Index Rank Percent of Possible Improvement Realized Statistically Significant? Note: The total improvement possible in any index is limited by the score in Survey 1. Indices with high values in Survey 1 have less room to improve, given that +10 is the maximum possible score. “Possible improvement” realized was calculated by taking the total difference in survey scores (Survey 6 – Survey 1), and dividing by the maximum possible improvement, or 10 minus the first survey score. Change Indices are rank ordered from largest to smallest change, based on percentage of possible improvement realized. A t-test was used to determine each if change was significantly different from zero.

23. Chapin Hall Center for Children Significant increases in survey indices by RF community, 2003-2006 Note: Significant increases occurred when the difference in mean index scores on the first survey (December 2003) and last survey (June 2006) was statistically significant ( p <.05). An increase was considered to be linear when at least 4 of 5 possible score changes between the six surveys were in a positive direction.

24. Chapin Hall Center for Children Community Significant and Linear Increases: 2003-2006Community Significant and Linear Increases: 2003-2006 Anchorage  Access to Services  AOD Assessment  Treatment Effectiveness  Cultural Integration New Hampshire  Access to Services  Data Sharing  AOD Assessment  Treatment Effectiveness  Targeted Treatment Santa Cruz None Dayton  Access to Services  Data Sharing  Resource Management  Client Information  AOD Assessment  Pro-social Activities Chicago  Data Sharing  Systems Integration  Resource Management  AOD Assessment  Pro-social Activities Portland  Data Sharing  Cultural Integration SE Kentucky  Data Sharing  Systems Integration  Client Information  Agency Collaboration  AOD Assessment  Family Involvement Rosebud  Data Sharing  Client Information Marquette  Data Sharing  Pro-social Activities Seattle  Treatment Effectiveness  Targeted Treatment None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.

25. Chapin Hall Center for Children What About Bias?  Answers to survey questions could be measures of self-interest or wishful thinking  Regression analyses were performed to check on this possibility  Analyses tested the influence of several factors on the trajectory of changes in individual survey scores  Main variable of interest: Did survey scores change more for RF “Insiders” -- the 4 or 5 people in each community who were involved in planning the RF initiative and who were closer to RWJF?

26. Chapin Hall Center for Children Most Indices Were Not Affected by Bias

27. Chapin Hall Center for Children Treatment Effectiveness The values differ, but the trajectories are positive for both groups, indicating growing confidence in the effectiveness of treatment.

28. Chapin Hall Center for Children Treatment Effectiveness

29. Chapin Hall Center for Children AOD Assessment

30. Chapin Hall Center for Children A Few Indices Were Affected by Bias

31. Chapin Hall Center for Children Cultural Integration

32. Chapin Hall Center for Children Partner Involvement

33. Chapin Hall Center for Children Community Significant and Linear Increases: 2003-2006Community Significant and Linear Increases: 2003-2006 Anchorage  Access to Services  AOD Assessment  Treatment Effectiveness  Cultural Integration New Hampshire  Access to Services  Data Sharing  AOD Assessment  Treatment Effectiveness  Targeted Treatment Santa Cruz None Dayton  Access to Services  Data Sharing  Resource Management  Client Information  AOD Assessment  Pro-social Activities Chicago  Data Sharing  Systems Integration  Resource Management  AOD Assessment  Pro-social Activities Portland  Data Sharing  Cultural Integration SE Kentucky  Data Sharing  Systems Integration  Client Information  Agency Collaboration  AOD Assessment  Family Involvement Rosebud  Data Sharing  Client Information Marquette  Data Sharing  Pro-social Activities Seattle  Treatment Effectiveness  Targeted Treatment None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve. If we cross out the indices that were possibly affected by bias…

34. Chapin Hall Center for Children Community Significant and Linear Increases: 2003-2006Community Significant and Linear Increases: 2003-2006 Anchorage  Access to Services  AOD Assessment  Treatment Effectiveness  Cultural Integration New Hampshire  Access to Services  Data Sharing  AOD Assessment  Treatment Effectiveness  Targeted Treatment Santa Cruz None Dayton  Access to Services  Data Sharing  Resource Management  Client Information  AOD Assessment  Pro-social Activities Chicago  Data Sharing  Systems Integration  Resource Management  AOD Assessment  Pro-social Activities Portland  Data Sharing  Cultural Integration SE Kentucky  Data Sharing  Systems Integration  Client Information  Agency Collaboration  AOD Assessment  Family Involvement Rosebud  Data Sharing  Client Information Marquette  Data Sharing  Pro-social Activities Seattle  Treatment Effectiveness  Targeted Treatment None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.

35. Chapin Hall Center for Children Community Significant and Linear Increases: 2003-2006Community Significant and Linear Increases: 2003-2006 Anchorage  Access to Services  AOD Assessment  Treatment Effectiveness  Cultural Integration New Hampshire  Access to Services  Data Sharing  AOD Assessment  Treatment Effectiveness  Targeted Treatment Santa Cruz None Dayton  Access to Services  Data Sharing  Resource Management  Client Information  AOD Assessment  Pro-social Activities Chicago  Data Sharing  Systems Integration  Resource Management  AOD Assessment  Pro-social Activities Portland  Data Sharing  Cultural Integration SE Kentucky  Data Sharing  Systems Integration  Client Information  Agency Collaboration  AOD Assessment  Family Involvement Rosebud  Data Sharing  Client Information Marquette  Data Sharing  Pro-social Activities Seattle  Treatment Effectiveness  Targeted Treatment None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.

36. Chapin Hall Center for Children Community Significant and Linear Increases: 2003-2006Community Significant and Linear Increases: 2003-2006 Anchorage  Access to Services  AOD Assessment  Treatment Effectiveness  Cultural Integration New Hampshire  Access to Services  Data Sharing  AOD Assessment  Treatment Effectiveness  Targeted Treatment Santa Cruz None Dayton  Access to Services  Data Sharing  Resource Management  Client Information  AOD Assessment  Pro-social Activities Chicago  Data Sharing  Systems Integration  Resource Management  AOD Assessment  Pro-social Activities Portland  Data Sharing  Cultural Integration SE Kentucky  Data Sharing  Systems Integration  Client Information  Agency Collaboration  AOD Assessment  Family Involvement Rosebud  Data Sharing  Client Information Marquette  Data Sharing  Pro-social Activities Seattle  Treatment Effectiveness  Targeted Treatment None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.

37. Chapin Hall Center for Children Community Significant and Linear Increases: 2003-2006Community Significant and Linear Increases: 2003-2006 Anchorage  Access to Services  AOD Assessment  Treatment Effectiveness  Cultural Integration New Hampshire  Access to Services  Data Sharing  AOD Assessment  Treatment Effectiveness  Targeted Treatment Santa Cruz None Dayton  Access to Services  Data Sharing  Resource Management  Client Information  AOD Assessment  Pro-social Activities Chicago  Data Sharing  Systems Integration  Resource Management  AOD Assessment  Pro-social Activities Portland  Data Sharing  Cultural Integration SE Kentucky  Data Sharing  Systems Integration  Client Information  Agency Collaboration  AOD Assessment  Family Involvement Rosebud  Data Sharing  Client Information Marquette  Data Sharing  Pro-social Activities Seattle  Treatment Effectiveness  Targeted Treatment None of the changes reported in Santa Cruz were significant and linear, but scores in Santa Cruz started out very high and thus had less room to improve.

38. Chapin Hall Center for Children Conclusions Respondent ratings of local juvenile justice systems improved significantly between 2003 and 2006. Increases were statistically significant in 12 of the 13 survey indices. Improvements were especially dramatic in the ratings for: - treatment effectiveness, - the use of client information in support of treatment, - the use of screening and assessment tools, and - overall systems integration. These findings suggests that the coordination and quality of substance abuse interventions for youthful offenders improved during the RWJF Reclaiming Futures initiative.

39. Chapin Hall Center for Children Jeffrey A. Butts Research Fellow Chapin Hall Center for Children University of Chicago 1313 East Sixtieth Street Chicago, IL 60637 (773) 256-5163 jabutts@uchicago.edu This presentation can be accessed at: www.jeffreybutts.net For more information: