Boys & Girls Clubs of King County Preliminary Results of BGCKC’s Program Evaluation Survey

Program Evaluation Rubric We sent our the survey on Monday, the 28th. We have had 27 site leaders out of 35 we contacted for participation. About 75% of our responses back. Here is a list of 22 programs we set out to evaluate, this list was determined by site leaders who are on our program evaluation team. This chart shows how many sites run each of these programs. These responses show that Summer Brain Gain, Power Hour, Smart girls and Wise Guys are our most commonly run programs. Be Great Graduate, Career Launch, Club Tech, Money Matters and Smart Moves are run by the least number of sites at this time. I highlight those in lighter blue because it will be good to keep in mind that low response rate for those programs as we walk through the evaluation data. Scores and averages for those programs won’t be as meaningful because they are derived from only one or two responses—so just keep that in mind. Also, none of our sites run Sport SMART at this time, so that program is not evaluated in this presentation.

How many sites run each program? We sent our the survey on Monday, the 28th. We have had 27 site leaders out of 35 we contacted for participation. About 75% of our responses back. Here is a list of 22 programs we set out to evaluate, this list was determined by site leaders who are on our program evaluation team. This chart shows how many sites run each of these programs. These responses show that Summer Brain Gain, Power Hour, Smart girls and Wise Guys are our most commonly run programs. Be Great Graduate, Career Launch, Club Tech, Money Matters and Smart Moves are run by the least number of sites at this time. I highlight those in lighter blue because it will be good to keep in mind that low response rate for those programs as we walk through the evaluation data. Scores and averages for those programs won’t be as meaningful because they are derived from only one or two responses—so just keep that in mind. Also, none of our sites run Sport SMART at this time, so that program is not evaluated in this presentation.

Alignment with Mission Program Evaluation 9-Block Financial Sustainability Red quartile Negative financial contribution and low alignment with mission—potential distractions Lower right blue—negative financial contribution that are aligned to mission—investments Yellow is questions Upper left—high financial contributions, low alignment with mission. Upper right is our star programs, high financial contributions, high alignment Alignment with Mission

Questions?

Our Three Pillars: Academic Success 1 No Focus 2 Implied or Possible Focus 3 Moderate or Occasional Focus 4 Strong or Explicit Focus Here we are jumping in to one of the first scales on our rubric. On the survey the question was posed “Part of our mission is to promote academic sucess for youth. Do you feel that the SMART Girls program at your site is focused on academic sucess?” The response categories are listed in the legend on the left. These map directly onto the rubric we will eventually use to plot our programs. Our programs that have scored high in academic focus are BE Great Graduate, Career Launch, CIT, Summer Brain Gain, and Power Hour. Lower scoring programs are Games room tournaments, Teen late night, Triple play and drama matters

Our Three Pillars: Good Character and Citizenship 1 No Focus 2 Implied or Possible Focus 3 Moderate or Occasional Focus 4 Strong or Explicit Focus Similarly, these scores are in response to a question on the survey that was posed: Part of our mission is to promote good character and citizenship for youth. Do you feel that the SMART Girls program at your site is focused on good character and citizenship? Career Launch, Date Smart, Smart moves, Triple Play and Smart Girls are high scoring programs in this area.

Our Three Pillars: Healthy Lifestyles 1 No Focus 2 Implied or Possible Focus 3 Moderate or Occasional Focus 4 Strong or Explicit Focus Lastly, these responses came from a question on the survey asking about healthy lifestyles. Our programs with higher scores in this area look to be Career Launch, Date Smart, Smart Girls, CIT, Tour Club and Triple Play Lower scoring programs are Club Tech, CS First, Power Hour and Games Room Tournaments

Demonstrated Program Quality: YPQA 1 2 3 4 4+ for Safe Environment 4+ for Supportive Environment 4+ for Youth Interaction 4+for Youth Engagement Moving down the rubric, this next section is an evaluation of programs based on our youth program quality initiative. These four criteria used in our Youth Program Quality assessment are meant to build on each other—with the Youth Engagement category being the top of the pyramid. You will see the criteria organized this way in the legend in the top right corner. On the survey, site leaders responded to the question phrased “Keeping the YPQA tool in mind, do you feel that the SMART Girls program at your site demonstrates: A Safe Environment, A supportive environment, youth interaction and youth engagement.” The response options was a 5 point likert scale ranging from strongly disagree to strongly agree. I coded each of these response options with a five number scale, with strongly disagree equal to a score of 1 and strongly agree equal to 5. I then took the averages of these scores for each YPQA criteria by program, and was able to score each of the programs based on the scale defined in our rubric, and represented here in the legend. Using this method, we had many programs who had an average score of 4 points or more in all four YPQ criteria, including Chess, Keystone, Smart Moves, Visual and performing arts programs etc. Lower scoring programs include BE GREAT Graduate, Games Room Tournaments, Club Teach, CS First and Power Hour

Social And Emotional Learning 1 Promotes 2 or fewer SEL competencies 2 Promotes 3-5 SEL competencies 3 Promotes 5-7 SEL competencies 4 Promotes all SEL competencies This next cateorgy of program evaluation is based on social and emotional learning. These scores were dirived from repsonses to the question posed on the survey: Social and emotional skills refer to the management of emotions and the ability to establish positive and rewarding relationships with others. Thinking about the SMART Girls program at your site, which of the following social and emotional competencies do you feel this program promotes?” We had the eight competencies listed in the rubric as response options, and site leaders could check all that apply. In analysing this data, I worked through scores by program, and I counted any of the competencies that had been “checked off” by majority of site leaders in reference to that program. If a program was shown to promote two or fewer SEL competencies, they received a score of one, as you will see represented in the legend on the left hand site. Our programs that scored high in Social and Emotional learning included BE Great Graduate, CIT, Keystone, Summer Brain Gain and Wise Guys. Lower score programs were Club Tech and Teen Late night

Financial Sustainability: Staff to Member Ratio 1 2 3 4 Average 1:1 Average 1:2-5 Average 1:6-10 Average 1:11+ Now we move into the financial sustainability portion of our rubric. This chart shows the average staff to member ratio for each program, with higher ratios scoring higher. Programs with a ration of 1 staff to 11 or more members were programs like Wise Guys, Triple Play, Games Rooms Tournaments, Chess and Career Launch. Programs with lower staff to member ratios include Be Great Graduate, CIT, CS First, Lego Robotics and SMART Girls

Financial Sustainability: Staff Preparation Time 1 Prep Time > Delivery Time 2 Prep Time = Delivery Time 3 Delivery Time > Prep Time 4 Minimal Prep Time In this section of the survey, site leaders were asked to evaluate the preparation time required to run a given program, in relation to the actual program delivery time. Programs with less prep time score higher on our rubric, and programs where prep time is greater than actually delivery time scored the lowest. As shown in the chart, high scoring programs include Club Tech, Games Room Tournaments, Power Hour and Teen Late Night Lower scoring programs include Summer Brain Gain, Money Matters, Lego Robotics and Career Launch

Financial Sustainability: Reliance on Unique Passion or Skill Next, on the survey we asked site leads whether or not running a given program relies on the unique passion or special training of staff. As is illustrated in the chart—programs that site leaders agree require specialized training or passion include Club Tech, BE GREAT Graduate, Lego Robotics and SMART Moves. Programs that more site leaders feel do not rely on special training or passion include Games Room Tournaments and Power Hour

Financial Sustainability: Requires Specialized Training for Staff Next, on the survey we asked site leads whether or not running a given program relies on the unique passion or special training of staff. As is illustrated in the chart—programs that site leaders agree require specialized training or passion include Club Tech, BE GREAT Graduate, Lego Robotics and SMART Moves. Programs that more site leaders feel do not rely on special training or passion include Games Room Tournaments and Power Hour

Financial Sustainability: Additional Program Costs 1 Greater Than $1,000 Per Member 2 $500-1,00 Per Member 3 $0-500 Per Member 4 No Additional Costs Lastly, site leaders were asked to report the additional costs of running certain programs. Programs scored higher in this section if they have low or no additional costs associated. High scoring programs in this area are Club Tech, Drama Matters, Games Room Tournaments, Power Hour and Smart Moves. The lowest scoring program is Keystone, in the 500-1000$ per member category.

Additional Fee?