Correlates of Perceived Effectiveness: Preliminary Results from the National Evaluation of the Safe Schools/Healthy Students Initiative Bruce Ellis, M.S., Ping Yu, Ph.D., Aaron Alford, Ph.D. Danyelle Mannix, Ph.D., Sharon Xiong, M.S. American Evaluation Association 2010 San Antonio, Texas November 8-13, 2010

Safe Schools/Healthy Students Initiative The Safe Schools/Healthy Students Initiative is a joint effort of the U.S. Departments of Education, Health and Human Services, and Justice Intent of the SS/HS Initiative is to help communities collaborate across systems to build safer, healthier environments for students in grades K-12 About $2 billion in grants awarded to 365 grantees since 1999 to support effective partnerships among schools and local mental health, law enforcement, and juvenile justice agencies

Program Theory Model

Data Sources Surveys Site visits Telephone interviews Service logs, archival data and grant applications

Six Outcome Measure s School-perceived impact on  Safety/Violence Prevention  Substance Use Prevention  Mental Health Services  Early Childhood Development  School Relationship with the Community, and  Overall Effectiveness

M etho ds Correlational studies used to eliminate pre-grant, grant operation, and near-term outcome variables that were unrelated to outcome measures Patterns of change and school/grantee variability were analyzed to develop best-fit unconditional growth model with a linear term for rate of growth and a non-linear term for change in rate of growth Evaluated a series of progressively more complex conditional growth models using significance of covariates and error terms, and model stability

Results of Unconditional Models Significant within-school and between-school differences in school-perceived impact for all six outcomes were found and justified the use of a multi-level growth model Significant improvement over time with significant slowing in rate of change for all six outcomes

Unconditioned and Conditioned Growth Trajectories by Outcome and by Year of Measurement

Results of Conditional Models: Pre-Grant Environment Poverty was found to be significantly related to 4 of the 6 outcome measures of school-perceived impact (overall, substance use prevention, mental health services and school-community relations) Funding per targeted capita was found to be significantly and positively related to initial school- perceived impact on mental health services but not to mean an accelerated changes in perceived impact

Grant Operations School resources were positive and significant for initial perceived impact and over time for safety and violence prevention, substance use prevention, mental health and early childhood development. School involvement results were mixed: change in school involvement was positively associated with school-community relationship, but negatively associated with the other outcome measures Partnership functioning was positively associated with change in perceived overall impact while partnership organization negatively affected change in impact on early childhood development

Near-term Outcomes Project-level comprehensive programs and activities aimed at early childhood development were significantly and positively related to initial school- perceived impact on safety/violence prevention and on mental health (although not to change over time), but negatively to perceived change overall Project-level programs and activities for mental health services were positively associated with initial school- perceived impact on early childhood development Coordination and service integration was significantly and positively related to initial perceived impact overall

Summary Program Theory model helped to frame a complex evaluation with many variables Multi-level growth model allowed us to evaluate nested school-within-grantee survey data and appropriately describe non-linear change over time Poverty, a pre-grant environment variable, was strongly and consistently predictive of the outcomes of interest School- and Project-level grant operations and near-term outcome variables showed inconsistent results – school resource predictors were generally positive, while school involvement predictors were mixed

Future Study Understanding the importance of school resources and school involvement Testing additional measures Exploring effects of school type – effects at elementary, middle and high schools Procedural refinements and better measurement of some time-dependent variables Additional cohorts: data are being collected for grantees awarded in , with more planned