Lessons learned from the Evaluation of Baltimore Expanded School Mental Health: Challenges and Solutions Georgetown University National Technical Assistance Center Webinar Series October 21, 2010 Making the Case for Evaluation of School Mental Health Programming: Strengthening Partnerships for Positive Outcomes Bruno J. Anthony, Ph.D. Georgetown University Center for Child and Human Development

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Collaborators Baltimore City Public Schools –Louise Fink –Ike Dibor –Shawn Hales Baltimore Mental Health Systems –Jane Paplinger –David Jones –Denise Wheatley-Rowe Georgetown Center for Child and Human Development –Bruno Anthony –Joyce Sebian –Courtney Holland –Clare Dunne

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Overview BESMH Baltimore ESMH a recognized state and national leader for more than 20 years. ESMH services have complemented and supplemented the services provided by Baltimore City Public Schools (social workers, psychologists, and guidance counselors) Currently, prevention and treatment services provided in 106 schools (approximately half of the schools in Baltimore City) through 12 provider agencies and 106 clinicians. Intended to address barriers to learning to enable students to make better use of educational programs in their schools and to provide an alternative to mental health services within structure of Special Education System Funding: Braided funding from school system, Department of Human Resources, Baltimore Substance Abuse System, Baltimore Mental Health System.

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Evaluation of BESMH Purpose: Assess the impact of a comprehensive ESMH program –Test assumptions that ESMH is beneficial in terms of academic outcomes –Inform programming –Inform ongoing evaluation and sustainability efforts. Funding –Planning: Internal funding from BESMH budget –Implementation: State of Maryland Mental Health Transformation State Incentive Grant (TSIG) Demand for academic accountability heightens the need for studies examining the specific impact of ESMH

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Challenges and Approaches: Developing a Meaningful and Feasible Plan  Key Informant Input –Focus Groups (administrators, program directors, clinicians/providers, caregivers, youth) –Consultation (advisory committee, monthly program directors meetings, leaders in the field) –School system partners (DREAA, Student Support Services, leaders)  Goals –Identifying important barriers and challenges –Identifying the appropriate and relevant outcomes –Incorporating content expertise in understanding ESMH in Baltimore –How to use the evaluation findings to improve the ESMH program and for advocacy. A participatory approach

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Challenges and Approaches: Examining Outcomes at Individual Level.  Employed the “zipper” for linking separate confidential data sets that contain personal identifying information while preserving anonymity. –Safe analyst to link and process data who is not part of either the research team, the agency with overall administrative oversight over the ESMH program or the school system. –Providers send service use data set with ID numbers sent to the “safe analyst,” who has no access to student names. –School system sends demographic and academic/school functioning with ID numbers to safe analyst. –Safe analyst uses ID numbers to merge the two data sets then encrypts ID numbers Risk of connecting private school and ESMH data

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Challenges and Approaches: Capturing Spectrum of ESMH Services  Electronic data collection –Developed in a participatory process Individual: For each date of contact, the type(s) of services and, at the end of each month, the focus of treatment and the type of the interventions. Group: attendance in prevention and treatment groups. Universal: four types of prevention activities. –Training: direct contact, manual and web-based Service logs were not capturing the range of promotion, prevention and treatment work carried out by clinicians.

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Extent and Type of Services  ESMH Clinicians touch an enormous number of students, caregivers and school staff, extending the range of clinical service that could not happen in more traditional settings.  Clinicians had direct contact with more 6254 students - 1/5 th of students attending the 106 schools that support ESMH  A large part of the clinicians work involves indirect, collateral services

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Percentage of Students by Grade

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Percentage of Service Types

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Percent of Direct and Indirect Services

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY

Service Focus Clinicians focus on issues that disrupt a student’s engagement with school Clinicians for the most part report using approaches with an evidence base Family/caregiver engagement is challenging

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Percentage of Focus Areas Coded

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY

Percentage of Intervention Types Coded

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Challenges and Approaches: Comparison Groups  Quasi-experimental strategies to support causal influences of ESMH.  Approaches: –Contact Groups –Matched Comparison Groups –Dosage Groups As is often the case with on-going, community-based programs, a full-fledged experimental (random-assignment) design was not practical, feasible or ethical.

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY E-E: Students receiving ESMH services in an ESMH school NoE-E: Students NOT receiving ESMH services in an ESMH school NoE-NoE: Students NOT receiving ESMH services in a school without ESMH Distribution of Students in Different Dosage Groups

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Matched Comparison Group Formation Propensity scores –Probability of a student in a school with an ESMH clinician receiving ESMH services, given a set of background variables for that individual. –Determined background risk factors that predicted participation in ESMH using Binary Logistic Regression ( attendance, suspensions, MSA Reading and Math scale scores). –Children receiving ESMH would have high propensity scores based on the set of background variables. Students with similar background characteristics but not receiving ESMH services should also have high propensity scores. –Propensity score of each ESMH student matched with a student drawn randomly from those students not enrolled in ESMH to form Comparison Group.

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Preliminary Overall Results A greater percentage of students attending a school with ESMH services and accessing these services showed –Improvement over the course of the year in Math and English benchmarks. –Decreases in suspensions

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Challenges and Approaches: Outcome Measurement Framework  Integrate data needs and system pressures of different stakeholders. –Use available data records, outcome measurements already in place, –Stakeholders recognized the imperative of demonstrating effectiveness in the school setting with selected mediating/moderating contextual variables. –The importance of looking at school outcomes (e.g., academic success, promotion, absences and suspensions) to ensure sustainability Devising an outcome measurement framework that satisfies different stakeholders without overburdening clinicians

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY

Evaluation Contributions to Sustaining ESMH  Building evaluation and research model – participatory process  Building infrastructure for ongoing evaluation and research – Service Trends Impact on programming decisions Impact on financial models Training –Leveraging other initiatives.  Utilizing evaluation data to make the case for ESMH program –Case for Ancillary and Supportive Services Teacher Consultation Care Coordination Family contacts –Case for School-related Benefits Improvement in academic measures Positive effects on school attendance and behavior

© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY Contacting Us BESMH Evaluation Team: Bruno Anthony, Ph.D. Professor and Director of Research and Evaluation, Joyce Sebian MS. Ed. Senior Policy Associate, Georgetown University Center for Child and Human Development Department of Pediatrics 3300 Whitehaven Street, NW, #3300 Washington, DC