Sel in ymca afterschool project results November 28, 2018 © 2017 YMCA of the USA

Table of contents Project goals & context Project learnings Lessons learned & next steps SEL in YMCA Afterschool Project Results

Project goals & context

SEL in YMCA Afterschool Project goal This project sought to positively impact adult practices by providing Y Afterschool Programs with tools, resources, training, and technical assistance to assess, plan, and improve their program's quality and the social-emotional competencies of the young people they serve. The goal was that these efforts would improve the social-emotional development of participants, and ultimately their educational, career, and life success. 2 min In a larger effort at Y-USA for a renewed focus on character development WHAT IS YOUR DEFINITION OF PRACTICES?

SEL in YMCA ASP sites

ASSESS PLAN IMPROVE SEL in YMCA As project implementation process Youth Social-Emotional Competencies Staff Social-Emotional Competencies Program Quality PLAN IMPROVE TRAINING & DATA COLLECTION PHASE (August-December) PLANNING WITH DATA PHASE (December-February) IMPROVEMENT PHASE (February- June) DATA & CONTINUOUS QUALITY IMPROVEMENT- Importance of linking SEL data and PQA data….how does one feed of other? Planning with Data session- happening early next year, will be the opportunity to do this. All of the SEL practices and the PQA practices that we’re using here are linked to one another.

SEL in YMCA As project implementation process ASSESS PLAN IMPROVE TRAINING & DATA COLLECTION PHASE (August-December) PLANNING WITH DATA PHASE (December-February) IMPROVEMENT PHASE (February- June) Program Level Assess the quality of programs and identify staff training needs Analyze data reports, identify and discuss emerging needs using the data, and select methods targeted for program improvement Execute the improvement plan to improve instructional program quality and meet staff needs Data Collection Tools/ Resources School-Age Program Quality Assessment (PQA) PQA Report Youth Work Methods Workshops Participant Level Observe social-emotional skills of participants Analyze reports, identify and discuss emerging needs of participants using the data, and select strategies targeted for program improvement Execute the improvement plan to improve participant’s social-emotional skills What does this mean to you? Can someone walk me through this? Data Collection Tools/ Resources DESSA-mini *Follow-up with full DESSA for participants who demonstrate need for instruction DESSA Report(s) SEL Strategies

Staff social-emotional competencies assessments YOUTH Social-emotional competencies Staff social-emotional competencies The SEL in YMCA Afterschool Project sites utilized the Devereux Student Strengths Assessment (DESSA)-mini version (8 items), with an option to complete a full DESSA (72 items) for participants who demonstrated need for instruction to further develop their social-emotional competencies, at the beginning and end of the 2016-17 program year. ​ The DESSA is a standardized, strength-based measure of the social and emotional competencies of children in kindergarten through 8th grade. The DESSA focuses on 8 competencies of social-emotional learning: self-awareness, self-management, goal-directed behavior, social awareness, relationship skills, personal responsibility, decision-making, and optimistic thinking. Following an assess-plan-improve sequence, out-of-school time staff assessed the social-emotional competencies of youth, utilized individual, group, or site-based data reports to create an improvement plan, and had access to SEL intervention strategies that help improve specific social-emotional competencies to carry out their improvement plans. Studies show that high-quality youth programs can help children develop social-emotional competencies, which is a key predictor of a successful adult life¹. The goal of this project was to provide afterschool program staff with tools that they can use to build and improve their own social and emotional competencies as well as effective strategies to promote the development of social and emotional competencies in youth. Developed by the American Institutes for Research, the Staff SEL Reflection tool focuses on five social and emotional competencies, including self-awareness, self-management/emotion regulation, social awareness, relationship/social skills, and responsible decision-making. The tool is designed to help afterschool program staff reflect upon their own social and emotional competencies and their ability to support young people's SEL through program practices. SOCIAL AWARENESS SELF-MANAGEMENT/ EMOTION REGULATION RESPONSIBLE DECISION-MAKING RELATIONSHIP/ SOCIAL SKILLS SOCIAL-EMOTIONAL COMPETENCIES SELF-AWARENESS RESPONSIBILITY AND CHOICE PROGRAM STAFF LANGUAGE WARMTH AND SUPPORT SOCIAL PRACTICES YOUTH-CENTERED PROBLEM-SOLVING SELF-REFLECTION AND SELF-ASSESSMENT GROUP DISCUSSIONS BALANCED PROGRAM PRACTICES PROGRAM IMPLEMENTATION PRACTICES COOPERATIVE/GROUP LEARNING ENCOURAGING GRIT AND PERSISTENCE COMPETENCE BUILDING SEL in YMCA Afterschool Project Results ¹ American Institutes for Research, 2015.

assessments Program quality Research has shown that afterschool programs that are of high quality make larger impacts than those without¹. In order to provide programs of high quality to our youth participants, it is important to us to measure program quality at each of our programs to define what effective practice looks like, evaluate a site against a set of quality standards, and allow programs to self-assess, reflect, and improve upon their programs. The School-Age Program Quality Assessment (S-PQA) is a validated tool that uses a continuous program quality improvement model and is designed to evaluate the quality of programs. It is based on observations of the program and contains items that focus on specific elements of best practice when working with children and youth. The pyramid on the left represents the four domains assessed by the S-PQA: Safe Environment, Supportive Environment, Interaction, Engagement. The S-PQA is part of a continuous improvement process, utilizing an assess-plan-improve approach. The process began with assessment in order to build on out-of-school time staff’s existing strengths and identify areas to address. These areas then went into an improvement plan as specific, measurable, attainable, results-focused, and time-bound goals and benchmarks for success. Finally, there were strong supports for youth leaders to manage improvement, and the high quality Youth Work Methods series of workshops for staff were made available to all sites. As a whole, the assess-plan-improve sequence establishes a supportive system for continuous improvement. Program quality ¹ Eccles, J.S. and J.A. Gootman (2002). Community programs to promote youth development. Washington, DC, National Academies Press. SEL in YMCA Afterschool Project Results

Ymca of the usa site Support SEL in YMCA Afterschool Site Association Leads Afterschool Online Community Email Reminders and Resources (NC) State Alliance Learning Community Calls Evaluation & Technical Training and Support SEL in YMCA Afterschool Project Results

Project learnings SEL in YMCA Afterschool Project Results

demographics The SEL in YMCA Afterschool Project was implemented in 30 afterschool sites across three states; California (5 associations in 5 sites), Minnesota (3 associations in 5 sites), and North Carolina (7 associations in 20 sites). Altogether, they completed DESSAs for a total of 1,200 participants in both Fall and Spring. Full DESSAs were completed for 140 participants. About 47% of participants were boys and 82% of participants were in grades kindergarten through 4th grade. State Mini-DESSA (matched) Full DESSA (matched) California 300 18 Minnesota 241 2 North Carolina 659 120 TOTAL 1,200 140 This slides shows matched data for all participants. Overall, there were 2,050 participants overall for which there is data (pre OR post) SEL in YMCA Afterschool Project Results

changes in social-emotional competencies 121 youth moved from need to typical or strength Something to note for pre-data 116/1200 (~10%) had “perfect” scores 217/1200 (~18%) had scores within 5 points of perfect NOT a normal distribution (see next slide) AG Last year: Need 15% to 9% Typical 66% both Strength 19% to 25% SEL in YMCA Afterschool Project Results

changes in social-emotional competencies by state California minnesota North carolina

Changes in social-emotional competencies california In the state of California, DESSA-minis were completed for 300 participants at the beginning and the end of the program year. There was a decrease in the number of participants in need for instruction*, from 21% to 14%, as well as an increase of participants in a typical range of social-emotional competencies, from 71% to 77%. Of the 62 participants who began the year in need for instruction, 37 (60%) moved to typical or strength ranges by the end of the program year. *T-scores on the DESSA are categorized as follows: T-Scores of 60 and above indicate that the child has social and emotional strengths, T-scores of 41-59 inclusive indicate that the child is showing a typical amount of positive behaviors related to social and emotional competence, and T-scores of 40 and below indicate that the child has a need for instruction to further develop social and emotional skills as compared to the national standardization sample. SEL in YMCA Afterschool Project Results

Changes in social-emotional competencies minnesota In the state of Minnesota, DESSA-minis were completed for 241 participants at the beginning and the end of the program year. There was a decrease in the number of participants in need for instruction*, from 8% to 6%, as well as an increase of participants in a typical range of social-emotional competencies, from 68% to 74%. Of the 18 participants who began the year in need for instruction, 16 (88%) moved to typical or strength ranges by the end of the program year. *T-scores on the DESSA are categorized as follows: T-Scores of 60 and above indicate that the child has social and emotional strengths, T-scores of 41-59 inclusive indicate that the child is showing a typical amount of positive behaviors related to social and emotional competence, and T-scores of 40 and below indicate that the child has a need for instruction to further develop social and emotional skills as compared to the national standardization sample. SEL in YMCA Afterschool Project Results

Changes in social-emotional competencies north carolina In the state of North Carolina, DESSA-minis were completed for 659 participants at the beginning and the end of the program year. There was a decrease in the number of participants in need for instruction*, from 13% to 9%, as well as an increase of participants in a typical range of social-emotional competencies, from 53% to 78%. Of the 86 participants who began the year in need for instruction, 70 (81%) moved to typical or strength ranges by the end of the program year. It should be noted that of the 34% of the participants that began in the strength range, 97 participants (43%) were given a perfect social-emotional competency score at the beginning of the program year, possibly accounting for the decrease in the strength range percentage from the beginning to the end of the program year. SEL in YMCA Afterschool Project Results *T-scores on the DESSA are categorized as follows: T-Scores of 60 and above indicate that the child has social and emotional strengths, T-scores of 41-59 inclusive indicate that the child is showing a typical amount of positive behaviors related to social and emotional competence, and T-scores of 40 and below indicate that the child has a need for instruction to further develop social and emotional skills as compared to the national standardization sample.

changes in social-emotional competencies Typically, if full DESSAs were completed, you should see all or most of the youth in the need category at the beginning of the program year, but there was one site that completed full DESSAs on all of their participants (didn’t do a mini at all). In the SEL in YMCA Afterschool Project, full DESSAs were completed for 140 participants at the beginning and the end of the program year. There was a decrease in the number of participants in need for instruction*, from 24% to 21%, as well as an increase of participants in the strength range of social-emotional competencies, from 11% to 33%. Of the 33 participants who began the year in need for instruction, 11 (33%) moved to typical or strength ranges by the end of the program year. SEL in YMCA Afterschool Project Results

Staff social-emotional competencies State Self-Reflection (Fall & Spring) California 2 Minnesota 8 North Carolina 10 TOTAL 20 Across the three states, 20 staff completed the Staff SEL Reflection tool at the beginning and end of the program year. The charts demonstrate the percentage of staff who felt competent in the areas of social-emotional competencies, staff practices, and program implementation practices at the beginning and end of the program year. SEL in YMCA Afterschool Project Results

Staff social-emotional competencies SEL in YMCA Afterschool Project Results

Program quality PQA Scores Improvement Youth Work Methods Plans Responsibility (2) Reflection (5) School-Age Planning (5) School-Age Choice (1) School-Age Leadership (5) Manage Feelings (1) Active Engagement (5) Skill Building (3) Encouragement (1) Emotional Safety (1) ASSESS PLAN IMPROVE PQA Scores (n=27) 3.54 /5 3.98 /5 3.89 /5 4.57 /5 One of the goals of the SEL in YMCA Afterschool Project was to implement the program quality improvement process during the program year. Of the 30 afterschool sites that participated, 27 completed the S-PQA, with higher scores on the lower levels of the pyramid and the lowest score in the Engagement area. Of the 27 sites that completed the S-PQA, 29 improvement plan goals across the four areas of the pyramid. The improvement plan goals focused heavily in the areas of Supportive Environment and Engagement. SEL in YMCA Afterschool Project Results

Lessons learned & next steps SEL in YMCA Afterschool Project Results

Lessons learned Bright Spots Learning community provided space for sites to share successes, challenges, and be accountable Growth in staff competencies may have been supported by A-P-I cycle State alliance model was key in continuing work in NC Positive experiences of using the data to inform practice Evaluation support was key for direct access to resources, tools, and learnings Challenges Staff turnover was a challenge at many sites Sites in Associations with similar work and those that took advantage of site “buddy” pairs felt more supported Time to complete assessments and process was a challenge at many sites SEL in YMCA Afterschool Project Results

Next steps All sites have access to assessment systems for the 2017-18 program year Sites in North Carolina will continue to check-in via the Learning Community calls and possibly seek additional funding Two Associations in North Carolina expanding work to additional sites SEL in YMCA Afterschool Project Results

Maria D. Guzmán-Rocha Research & Evaluation maria.guzman@ymca.net 312-419-8329 SEL in YMCA Afterschool Project Results