Statewide Afterschool Evaluation— What Do the Data Tell Us Jason Patrie Missouri Afterschool Network (MASN) Wayne Mayfield, PhD Office of Social and Economic Data Analysis (OSEDA) University of Missouri Missouri School Age Community Coalition Annual Meeting & Professional Development Institute November 16, 2017 Independence, MO

Agenda Examine the Logic Model Discuss the data sources for evaluation Surveys Discuss how data are used Present statewide results over time

Logic Model College and Career Readiness/Success Afterschool Program Quality   Youth Outcomes College and Career Readiness/Success AS Staff Skills AS Program Structure Training Coaching Self-assessment

Logic Model - Inputs College and Career Readiness/Success Afterschool Program Quality   Youth Outcomes College and Career Readiness/Success AS Staff Skills AS Program Structure Training Coaching Self-assessment Ways to measure Improvement Strategies Training: Documentation of clock hour training for staff Coaching: Working Alliance Inventory Self-assessment: Documentation of self-assessment activities Ways to measure AS Staff Skills and Program Structure AS Staff Skills: PQA, Core Competencies, Highest level of education AS Program Structure: PQA

Logic Model - Output College and Career Readiness/Success Afterschool Program Quality   Youth Outcomes College and Career Readiness/Success AS Staff Skills AS Program Structure Training Coaching Self-assessment Ways to measure the output Afterschool Program Quality Program Quality Assessment (PQA) external or self-assessment Dimensions of Success Missouri Afterschool Program Standards (MAPS)

Logic Model - Output College and Career Readiness/Success Afterschool Program Quality   Youth Outcomes College and Career Readiness/Success AS Staff Skills AS Program Structure Training Coaching Self-assessment Youth Outcome domains Academic Achievement Academic Self-efficacy Positive School Behaviors Personal and Social Skills Commitment to Learning Ways to measure Youth Outcomes Grades MAP scores Leading Indicator surveys School-day attendance AS attendance DESE Core Data on suspensions, referrals School-day teacher, AS staff, youth, and parent surveys

Logic Model - Output College and Career Readiness/Success Afterschool Program Quality   Youth Outcomes College and Career Readiness/Success AS Staff Skills AS Program Structure Training Coaching Self-assessment Potential ways to measure College and Career Readiness On-time high school graduation Attendance at college, other career/technical preparation Completion of college, career/technical preparation Viable employment (These would be most easily addressed via a longitudinal data system.)

Data Sources These data sources are used for both the local and statewide evaluation Program Quality Assessments (PQAs) Kids Care Center Data Surveys DESSA

Surveys What to expect this year: Survey window: Mid-February through March Paper surveys available for older youth You received a survey packet with samples, tips, and timelines to help you be better prepared Prep-survey sent mid-January

How Data Are Used Local Sites use PQA, Leading Indicator, and Missouri Afterschool Survey Results reports at Planning with Data sessions Will use data to target specific areas for improvement Use data to drive staff PD needs Include in local evaluations

How Data Are Used Statewide DESE reviews results to refine the grant requirements for the 21st CCLC and SAC grantees. MASN’s MOARC review results to refine the training and technical assistance provided to 21st CCLC and SAC grantees. MASN’s Quality Committee review results to plan for systems level changes or new development that will support afterschool sites. MASN’s Policy Committee use aggregate data to communicate the importance and impacts of afterschool.

Evaluation Results over Time

Data for Statewide Evaluation 2013-14 2014-15 2015-16 2016-17 Sites 185 207 201 181 Youth 9,015 11,569 12,936 12,567 Parents 2,880 3,569 3,652 4,330 Site coordinators 242 254 259 255 Staff 1,280 1,348 1,409 1,251 Community partners 330 364 362 384 School administrators 232 217 266 230

Comparing 21st CCLC and SAC Sites 21st CCLC sites accounted for about 85% of data; SAC, 15%. 21st CCLC grants are larger per site than SAC grants. 21st CCLC sites less likely to charge fees compared to SAC sites. 21st CCLC sites more likely to serve older youth (over 5th grade), minority youth, youth in poverty, and youth from households with less education.

Afterschool Goals Goal 1: Support or increase student achievement and sense of competence in the areas of reading/communication arts, mathematics, and science. Goal 2: Develop and maintain a quality program that includes a safe and supportive environment, positive interactions, and meaningful opportunities for engagement. Goal 3: Enhance youth’s college and career readiness skills and behaviors, including positive school behaviors, personal and social skills, and commitment to learning.

Goal 1: At least 50% of youth maintain/increase grades in reading, math, and science (21st CCLC only) Don’t have data for this year

Goal 1: At least 70% of youth report medium to high levels of efficacy in reading, math, and science For 21st CCLC , we are a bit concerned about the decrease in efficacy and STEM interest/engagement over time. We see a potentially similar patter for SAC, although Math efficacy may be on the rebound.

Goal 2: All sites will score at least 2.9 overall on PQA Don’t have data from this year. Doing pretty well overall.

Statewide PQA Results for 21st CCLC Sites Don’t have data from this year. 3.9 on the Overall Instructional Quality (top read line) is where Weikart research shows that students’ positive engagement leads to good outcomes. Below 2.9 results in student disengagement and potential for harmful outcomes. 21st CCLC is almost at 3.9!

Longitudinal PQA Results: 21st CCLC

Statewide PQA Results for SAC Sites Don’t have data from this year. 3.9 on the Overall Instructional Quality (top read line) is where Weikart research shows that students’ positive engagement leads to good outcomes. Below 2.9 results in student disengagement and potential for harmful outcomes. SAC is over 3.9!

Longitudinal PQA Results: SAC

Goal 2: At least 85% of sites will score an average 3 Goal 2: At least 85% of sites will score an average 3.0 on the Organizational Context Leading Indicators of Staffing Model and Continuous Improvement. We are improving!

Goal 2: At least 85% sites will score an average 3 Goal 2: At least 85% sites will score an average 3.0 on the Instructional Context Leading Indicators of Academic Press and Engaging Instruction Hit 100%!

Goal 2: At least 85% of sites will score an average 3 Goal 2: At least 85% of sites will score an average 3.0 on the External Relationships Leading Indicators of Family Communication and School Alignment Don’t’ have data for this year. We are improving!

Goal 3: At least 50% of total youth enrolled in the afterschool program per site will have at least 60 days of attendance in the afterschool program. We are improving!

Attendance Benchmarks Based on Age Group Served, 2016-17

Goal 3: At least 85% of sites will score medium to high levels of personal/social skills Holding steady for the most part.

Goal 3: At least 70% of youth per site will indicate medium to high levels of commitment to learning Improvement over time.

Family Survey Results: School Day Linkages, Benefits of Afterschool, and Strengthening Families For 21st CCLC, School Day Linkages and Benefits of Afterschool have remained consistent and high. Strengthening Families has increased steadily over time. For SAC, there has been a steady increase in the Benefits of Afterschool and an overall positive trend for Strengthening Families (although down from last year).

21st CCLC: Is program quality related to youth and parent outcomes? Calculated correlations at the program level of Total PQA with youth and parent scales for 21st CCLC programs. The following measures were significantly correlated (p < .05). Science grade maintenance/increase (.21) Youth Engagement and Belonging (.26) Technology Efficacy (.20) This analysis used last year’s data. Program quality was related to a number of important outcomes.

What factors are associated with maintaining/increasing grades? Calculated ANCOVAs examining maintenance/increase of grades in three subject areas. Youth demographic variables and scales, as well as Time 1 grades, used as independent variables. Significant predictors shown below. Reading (n = 3311) Math (n = 3308) Science (n = 3342) Time 1 reading grade (lower grade more likely) Time 1 math grade (lower grade more likely) Time 1 science grade (lower grade more likely) SiteOrgID (site related to outcome) Race (nonminority more likely) Sex (girls more likely) Work Habits scale (higher scores more likely) Math Efficacy scale (higher scores more likely) Grade level (younger children more likely) Reading Efficacy scale (higher scores more likely)

How are we doing? Quality (based on PQA) is improving. Met Organizational Context and Instructional Context benchmarks. Need additional work on External Relationships. Progress on increasing afterschool attendance. Met Personal/Social Skills and Commitment to Learning benchmarks. Parents report more support from programs (Strengthening Families). Significant correlations between 21st CCLC program quality and science grade maintenance/increase, youth engagement and belonging, and youth technology efficacy. Youth work habits and reading/math efficacy are significantly related to grade maintenance/increase.

DESSA-mini Pilot School-day teachers rated youth using DESSA-mini (n = 1216) DESSA-mini is a screener for social-emotional functioning. 21% rated as Strength 62% as Typical 17% as Need for Instruction School-day teachers also rated youth on six items that address academic functioning (teacher survey). Both the DESSA-mini and the teacher survey were significantly related to maintenance/increase of grades. For 2017-18, afterschool staff will rate youth at two time points using the DESSA-mini.

Contact Information Jason Patrie (573) 884-7590 patriej@missouri.edu Wayne Mayfield (573) 882-5428 mayfieldw@missouri.edu