1. Statewide Afterschool Evaluation— What Do the Data Tell Us
Jason Patrie
Missouri Afterschool Network (MASN)
Wayne Mayfield, PhD
Office of Social and Economic Data Analysis (OSEDA)
University of Missouri
Missouri School Age Community Coalition Annual Meeting & Professional Development Institute
November 16, 2017
Independence, MO

2. Agenda Examine the Logic Model Discuss the data sources for evaluation
Surveys
Discuss how data are used
Present statewide results over time

3. Logic Model College and Career Readiness/Success
Afterschool Program Quality
Youth Outcomes
College and Career Readiness/Success
AS Staff Skills
AS Program Structure
Training
Coaching
Self-assessment

4. Logic Model - Inputs College and Career Readiness/Success
Afterschool Program Quality
Youth Outcomes
College and Career Readiness/Success
AS Staff Skills
AS Program Structure
Training
Coaching
Self-assessment
Ways to measure Improvement Strategies
Training: Documentation of clock hour training for staff
Coaching: Working Alliance Inventory
Self-assessment: Documentation of self-assessment activities
Ways to measure AS Staff Skills and Program Structure
AS Staff Skills: PQA, Core Competencies, Highest level of education
AS Program Structure: PQA

5. Logic Model - Output College and Career Readiness/Success
Afterschool Program Quality
Youth Outcomes
College and Career Readiness/Success
AS Staff Skills
AS Program Structure
Training
Coaching
Self-assessment
Ways to measure the output Afterschool Program Quality
Program Quality Assessment (PQA) external or self-assessment
Dimensions of Success
Missouri Afterschool Program Standards (MAPS)

6. Logic Model - Output College and Career Readiness/Success
Afterschool Program Quality
Youth Outcomes
College and Career Readiness/Success
AS Staff Skills
AS Program Structure
Training
Coaching
Self-assessment
Youth Outcome domains
Academic Achievement
Academic Self-efficacy
Positive School Behaviors
Personal and Social Skills
Commitment to Learning
Ways to measure Youth Outcomes
Grades
MAP scores
Leading Indicator surveys
School-day attendance
AS attendance
DESE Core Data on suspensions, referrals
School-day teacher, AS staff, youth, and parent surveys

7. Logic Model - Output College and Career Readiness/Success
Afterschool Program Quality
Youth Outcomes
College and Career Readiness/Success
AS Staff Skills
AS Program Structure
Training
Coaching
Self-assessment
Potential ways to measure College and Career Readiness
On-time high school graduation
Attendance at college, other career/technical preparation
Completion of college, career/technical preparation
Viable employment
(These would be most easily addressed via a longitudinal data system.)

8. Data Sources
These data sources are used for both the local and statewide evaluation
Program Quality Assessments (PQAs)
Kids Care Center Data
Surveys
DESSA

9. Surveys What to expect this year:
Survey window: Mid-February through March
Paper surveys available for older youth
You received a survey packet with samples, tips, and timelines to help you be better prepared
Prep-survey sent mid-January

10. How Data Are Used
Local
Sites use PQA, Leading Indicator, and Missouri Afterschool Survey Results reports at Planning with Data sessions
Will use data to target specific areas for improvement
Use data to drive staff PD needs
Include in local evaluations

11. How Data Are Used Statewide
DESE reviews results to refine the grant requirements for the 21st CCLC and SAC grantees.
MASN’s MOARC review results to refine the training and technical assistance provided to 21st CCLC and SAC grantees.
MASN’s Quality Committee review results to plan for systems level changes or new development that will support afterschool sites.
MASN’s Policy Committee use aggregate data to communicate the importance and impacts of afterschool.

12. Evaluation Results over Time

13. Data for Statewide Evaluation
Sites
185
207
201
181
Youth
9,015
11,569
12,936
12,567
Parents
2,880
3,569
3,652
4,330
Site coordinators
242
254
259
255
Staff
1,280
1,348
1,409
1,251
Community partners
330
364
362
384
School administrators
232
217
266
230

14. Comparing 21st CCLC and SAC Sites
21st CCLC sites accounted for about 85% of data; SAC, 15%.
21st CCLC grants are larger per site than SAC grants.
21st CCLC sites less likely to charge fees compared to SAC sites.
21st CCLC sites more likely to serve older youth (over 5th grade), minority youth, youth in poverty, and youth from households with less education.

15. Afterschool Goals
Goal 1: Support or increase student achievement and sense of competence in the areas of reading/communication arts, mathematics, and science.
Goal 2: Develop and maintain a quality program that includes a safe and supportive environment, positive interactions, and meaningful opportunities for engagement.
Goal 3: Enhance youth’s college and career readiness skills and behaviors, including positive school behaviors, personal and social skills, and commitment to learning.

16. Goal 1: At least 50% of youth maintain/increase grades in reading, math, and science (21st CCLC only)
Don’t have data for this year

17. Goal 1: At least 70% of youth report medium to high levels of efficacy in reading, math, and science
For 21st CCLC , we are a bit concerned about the decrease in efficacy and STEM interest/engagement over time. We see a potentially similar patter for SAC, although Math efficacy may be on the rebound.

18. Goal 2: All sites will score at least 2.9 overall on PQA
Don’t have data from this year. Doing pretty well overall.

19. Statewide PQA Results for 21st CCLC Sites
Don’t have data from this year. 3.9 on the Overall Instructional Quality (top read line) is where Weikart research shows that students’ positive engagement leads to good outcomes. Below 2.9 results in student disengagement and potential for harmful outcomes. 21st CCLC is almost at 3.9!

20. Longitudinal PQA Results: 21st CCLC

21. Statewide PQA Results for SAC Sites
Don’t have data from this year. 3.9 on the Overall Instructional Quality (top read line) is where Weikart research shows that students’ positive engagement leads to good outcomes. Below 2.9 results in student disengagement and potential for harmful outcomes. SAC is over 3.9!

22. Longitudinal PQA Results: SAC

23. Goal 2: At least 85% of sites will score an average 3
Goal 2: At least 85% of sites will score an average 3.0 on the Organizational Context Leading Indicators of Staffing Model and Continuous Improvement.
We are improving!

24. Goal 2: At least 85% sites will score an average 3
Goal 2: At least 85% sites will score an average 3.0 on the Instructional Context Leading Indicators of Academic Press and Engaging Instruction
Hit 100%!

25. Goal 2: At least 85% of sites will score an average 3
Goal 2: At least 85% of sites will score an average 3.0 on the External Relationships Leading Indicators of Family Communication and School Alignment
Don’t’ have data for this year. We are improving!

26. Goal 3: At least 50% of total youth enrolled in the afterschool program per site will have at least 60 days of attendance in the afterschool program.
We are improving!

27. Attendance Benchmarks Based on Age Group Served, 2016-17

28. Goal 3: At least 85% of sites will score medium to high levels of personal/social skills
Holding steady for the most part.

29. Goal 3: At least 70% of youth per site will indicate medium to high levels of commitment to learning
Improvement over time.

30. Family Survey Results: School Day Linkages, Benefits of Afterschool, and Strengthening Families
For 21st CCLC, School Day Linkages and Benefits of Afterschool have remained consistent and high. Strengthening Families has increased steadily over time.
For SAC, there has been a steady increase in the Benefits of Afterschool and an overall positive trend for Strengthening Families (although down from last year).

31. 21st CCLC: Is program quality related to youth and parent outcomes?
Calculated correlations at the program level of Total PQA with youth and parent scales for 21st CCLC programs.
The following measures were significantly correlated (p < .05).
Science grade maintenance/increase (.21)
Youth Engagement and Belonging (.26)
Technology Efficacy (.20)
This analysis used last year’s data. Program quality was related to a number of important outcomes.

32. What factors are associated with maintaining/increasing grades?
Calculated ANCOVAs examining maintenance/increase of grades in three subject areas.
Youth demographic variables and scales, as well as Time 1 grades, used as independent variables.
Significant predictors shown below.
Reading (n = 3311)
Math (n = 3308)
Science (n = 3342)
Time 1 reading grade (lower grade more likely)
Time 1 math grade (lower grade more likely)
Time 1 science grade (lower grade more likely)
SiteOrgID (site related to outcome)
Race (nonminority more likely)
Sex (girls more likely)
Work Habits scale (higher scores more likely)
Math Efficacy scale (higher scores more likely)
Grade level (younger children more likely)
Reading Efficacy scale (higher scores more likely)

33. How are we doing? Quality (based on PQA) is improving.
Met Organizational Context and Instructional Context benchmarks.
Need additional work on External Relationships.
Progress on increasing afterschool attendance.
Met Personal/Social Skills and Commitment to Learning benchmarks.
Parents report more support from programs (Strengthening Families).
Significant correlations between 21st CCLC program quality and science grade maintenance/increase, youth engagement and belonging, and youth technology efficacy.
Youth work habits and reading/math efficacy are significantly related to grade maintenance/increase.

34. DESSA-mini Pilot
School-day teachers rated youth using DESSA-mini (n = 1216)
DESSA-mini is a screener for social-emotional functioning.
21% rated as Strength
62% as Typical
17% as Need for Instruction
School-day teachers also rated youth on six items that address academic functioning (teacher survey).
Both the DESSA-mini and the teacher survey were significantly related to maintenance/increase of grades.
For , afterschool staff will rate youth at two time points using the DESSA-mini.

35. Contact Information
Jason Patrie (573) Wayne Mayfield (573)