1. Statewide Afterschool Evaluation— What Do the Data Tell Us
Jason Patrie
Missouri Afterschool Network (MASN)
Wayne Mayfield, PhD
Office of Social and Economic Data Analysis (OSEDA)
Missouri School Age Community Coalition Annual Meeting & Professional Development Institute
November 2, 2018
Lodge of the Four Seasons, Lake Ozark, MO

2. Today’s Agenda Overview of the statewide evaluation system
Data Collection
Surveys
DESSA
Evaluation Results

3. Evaluation Work MASN coordinates the statewide and local Evaluations
Subcontracts with Weikart Center and OSEDA for survey development, data analysis, and report writing
Provides training/certification of external evaluators
Manages the data collection process
Surveys
Contract with Aperture

4. Guiding Principles
Grantees and sites should see and be able to use the data collected
Expectations (goals) should be consistent across all levels (site, grant, state)
Programs need support with interpreting and using their data
Same data used for quality improvement efforts and evaluation efforts

5. Comprehensive Data Collection and Improvement Framework

6. Apples-to-Apples Evaluation System
Additional Analysis
Statewide Evaluation
State
Evaluation Design
MASN
OSEDA
DESE
Data Collection
Grantees
Data Report Generation
Weikart
Guided Reflection Document
Local Evaluator
Grantee Level Action Plans
Coaches
OSEDA
Weikart
OSEDA
Weikart
Grantee
Planning with Data
Site Level Action Plans
Site
Sites
Coaches
Sites
Coaches

7. Site Level Cycle of Improvement
Training and Resources
PQA
KCC
Surveys
Action Plan
Implement Plan
Site Visits
Revisit the action plan
Plan
Assess

8. Grantee Level Cycle of Improvement
Guided Reflection
Local Context
Data Review
Written responses
Includes a review of previous year’s goal
Program Level Action Plan
Developed by Program Director and ARE
Based on External Evaluators recommendation
Submitted to Brad for data entry
Copy provided to External Evaluator
Implement Plan
Training, resources, etc.
Revisit the Action Plan at 2nd visit and at other contact points
Improve
Assess
Plan

9. Use of Evaluation Results to Develop and Refine the Afterschool Program
Quality Action Plan
FY17 Guided Reflection Recommendations
FY18 Guided Reflection Review
DOE is looking at how States are holding grantees accountable for meeting their goals

10. Data Collection

11. KCC and PQAs Program attendance Entered monthly
Minimum of 5 days to be included in attendance calculations
Grades
Use the first and last entered for each student
Fall pre/post
Spring pre/post
June 15 data check

12. Surveys Youth Survey Family Survey Coordinator/Director Survey
Younger youth - paper
Older youth – paper
Older youth - online
Family Survey
Paper – English
Paper – Spanish
Online
Coordinator/Director Survey
Online – link sent directly
Site Staff Survey
Online – link sent to site director for distribution
School Administrator Survey
Community Partner Survey

13. Survey Process Pre-survey in January February 15th – Window opens
How many paper surveys
Where send surveys
s of site coordinators/facilitators
February 15th – Window opens
links open
Site Coordinators and School Administrators sent links directly
Reminder s and progress checks
March 31st – Window closes
Clean paper surveys
Contact programs

14. What is the DESSA-mini The DESSA-mini is NOT:
An 8-item screener; a brief review of SEL skills
Provides a simple and effective way to track changes in social and emotional competence over time
The DESSA-mini is NOT:
An assessment
A way to label a child

15. DESSA-mini Data Collection
Year two underway
High School Edition available now!
Expanded use of tool within programs

16. Statewide Evaluation Results over Time

17. Data for Statewide Evaluation
Sites
185
207
201
181
187
Youth
9,015
11,569
12,936
12,567
12,788
Parents
2,880
3,569
3,652
4,330
4,312
Site directors
242
254
259
255
299
Staff
1,280
1,348
1,409
1,251
1,467
Community partners
330
364
362
384 NA
School administrators
232
217
266
230
262

18. Comparing 21st CCLC and SAC Sites
21st CCLC sites accounted for about 85% of data; SAC, 15%.
21st CCLC grants are larger per site than SAC grants.
21st CCLC sites less likely to charge fees compared to SAC sites.
21st CCLC sites more likely to serve older youth (over 5th grade), minority youth, youth in poverty, and youth from households with less education.

19. Goals
Goal 1: Support or increase student achievement and sense of competence in the areas of reading/communication arts, mathematics, and science.
Goal 2: Develop and maintain a quality program that includes a safe and supportive environment, positive interactions, and meaningful opportunities for engagement.
Goal 3: Enhance youth’s college and career readiness skills and behaviors, including positive school behaviors, personal and social skills, and commitment to learning.
Positive School Behaviors
School day attendance
Afterschool attendance
Fewer discipline referrals
Fewer in-school and out-of-school suspensions
Commitment to Learning
Homework completion
Work and study skills
Initiative
Positive school engagement
Personal and Social Skills
Planning skills
Time management
Critical thinking/problem solving
Communication skills
Team work
Relationships with peers and adults
Accountability and personal responsibility

20. Goal 1: Support or increase student achievement and sense of competence in the areas of reading/communication arts, mathematics, and science.
Objective 1.1: At least 50% of youth per site will maintain and/or increase their grades in reading/communication arts during the school year as measured by pre-/post-grades entered into Kids Care Center.
Objective 1.2: At least 50% of youth per site will maintain and/or increase their grades in math during the school year as measured by pre-/post-grades entered into Kids Care Center.
Objective 1.3: At least 50% of youth per site will maintain and/or increase their grades in science during the school year as measured by pre-/post- grades entered into Kids Care Center.
Objective 1.4: At least 70% of youth per site will report a medium to high level of reading efficacy as measured by items on the youth survey (average score of 3.5 or higher).
Objective 1.5: At least 70% of youth per site will report a medium to high level of math efficacy as measured by items on the youth survey (average score of 3.5 or higher).
Objective 1.6: At least 70% of youth per site will report a medium to high level of interest and engagement in STEM as measured by items on the youth survey (total score of 3.0 or higher).

21. Goal 2: Develop and maintain a quality program that includes a safe and supportive environment, positive interactions, and meaningful opportunities for engagement.
Objective 2.1: All sites will score at least an average 2.9 on the Program Quality Assessment tool. Objective 2.2: All sites will score at least an average 3.0 on the Organizational Context Leading Indicators of Staffing Model and Continuous Improvement. Objective 2.3: All sites will score at least an average 3.0 on the Instructional Context Leading Indicators of Academic Press and Engaging Instruction. Objective 2.4: All sites will score at least an average 3.0 on the External Relationships Leading Indicators of Family Communication and School Alignment.

22. Goal 3: Enhance youth’s college and career readiness skills and behaviors, including positive school behaviors, personal and social skills, and commitment to learning.
Objective 3.1: At least 50% of youth per site will meet or exceed the school district’s average rate of school-day attendance. Objective 3.2: At least 50% of total youth enrolled in the afterschool program per site will have at least 60 days of attendance in the afterschool program. Objective 3.3: At least 50% of youth per site will have no in-building or out-of- school suspensions. Objective 3.4: At least 70% of youth per site will indicate a medium to high level of personal and social skills as measured by items on the youth survey (average score of 3.5 or higher). Objective 3.5: At least 70% of youth per site will indicate a medium to high level of commitment to learning as measured by items on the youth survey (average score of 3.5 or higher).

23. Goal 1: At least 50% of youth maintain/increase grades in reading, math, and science (21st CCLC only)
Don’t have data for this year

24. Goal 1: At least 70% of youth report medium to high levels of efficacy in reading, math, and science
For 21st CCLC , we are a bit concerned about the decrease in efficacy and STEM interest/engagement over time. We see a potentially similar patter for SAC, although Math efficacy may be on the rebound.

25. Goal 2: All sites will score at least 2.9 overall on PQA
Don’t have data from this year. Doing pretty well overall.

26. Statewide PQA Results for 21st CCLC Sites
Don’t have data from this year. 3.9 on the Overall Instructional Quality (top read line) is where Weikart research shows that students’ positive engagement leads to good outcomes. Below 2.9 results in student disengagement and potential for harmful outcomes. 21st CCLC is almost at 3.9!

27. Statewide PQA Results for SAC Sites
Don’t have data from this year. 3.9 on the Overall Instructional Quality (top read line) is where Weikart research shows that students’ positive engagement leads to good outcomes. Below 2.9 results in student disengagement and potential for harmful outcomes. SAC is over 3.9!

28. Goal 2: At least 85% of sites will score an average 3
Goal 2: At least 85% of sites will score an average 3.0 on the Organizational Context Leading Indicators of Staffing Model and Continuous Improvement.
We are improving!

29. Goal 2: At least 85% sites will score an average 3
Goal 2: At least 85% sites will score an average 3.0 on the Instructional Context Leading Indicators of Academic Press and Engaging Instruction
Hit 100%!

30. Goal 2: At least 85% of sites will score an average 3
Goal 2: At least 85% of sites will score an average 3.0 on the External Relationships Leading Indicators of Family Communication and School Alignment
Don’t’ have data for this year. We are improving!

31. Goal 3: At least 50% of total youth enrolled in the afterschool program per site will have at least 60 days of attendance in the afterschool program.
We are improving!

32. Goal 3: At least 70% of youth per site will score medium to high levels of personal/social skills
Holding steady for the most part.

33. Goal 3: At least 70% of youth per site will indicate medium to high levels of commitment to learning
Improvement over time.

34. Is program quality related to youth and parent outcomes?
Calculated correlations at the program level of Total PQA with youth and parent scales for 21st CCLC and SAC programs.
The following measures were significantly correlated (p < .05).
***p < **p < .01 *p < .05.
21st CCLC sites
Partial r
SAC sites
% maintaining/increasing science grade
.19*
Reading efficacy
.44*
Science interest & engagement
.43*
Personal and social skills
.38*
Commitment to learning
Youth engagement and belonging
.53**
School day linkages
.61***
Benefits of afterschool
.36*
Strengthening families (parent)
Family quality
.49**
Family youth outcomes
.58***
This analysis used last year’s data. Program quality was related to a number of important outcomes.

35. What factors are associated with maintaining/increasing grades?
Calculated ANCOVAs examining maintenance/increase of grades in three subject areas.
Youth demographic variables and scales used as independent variables.
Significant predictors shown below.
Reading (n = 4387)
Math (n = 4336)
Science (n = 3972)
Time 1 reading grade (lower grade more likely)
Time 1 math grade (lower grade more likely)
Time 1 science grade (lower grade more likely)
SiteOrgID (site related to outcome)
Race (nonminority more likely)
Sex (girls more likely)
Work Habits scale (higher scores more likely)
Age (younger youth more likely)
Math Efficacy scale (higher scores more likely)
Work Habits scale
(higher scores more likely)
Reading Efficacy scale
Math Efficacy scale
Commitment to Learning

36. DESSA-mini Results,
Afterschool staff rated youth using DESSA-mini in fall and spring (n = 7,396 with same rater at both time points).
Category
Fall 2017
Spring 2018
Difference
Need
15%
13%
-2%
Typical
60%
56%
-4%
Strength
25%
31%
+6%

37. DESSA-mini Results, 2017-18 (cont.)
Category
Declined
16%
Maintained
61%
Improved
23%
Category
Strength to Strength
16%
Strength to Typical
10%
Strength to Need 1%
Typical to Strength
15%
Typical to Typical
39%
Typical to Need 6%
Need to Strength
Need to Typical 7%
Need to Need

38. DESSA-mini Results, 2017-18 (cont.)
PQA total < PQA total 3.9+

39. DESSA-mini Results, 2017-18 (cont.)
School-day teachers also rated youth using DESSA- mini in Spring 2018 (n = 8,192).
Note similarities in ratings from afterschool staff in Spring 2018.
Category
School-day teachers
Afterschool staff (spring)
Need
12%
13%
Typical
52%
56%
Strength
36%
31%

40. How are we doing? The Bad News
For 21st CCLC, percentage of sites meeting reading, math efficacy and science interest and engagement benchmarks is decreasing.
SAC sites saw decreases in reading efficacy and science interest and engagement.
For 21st CCLC, percentage of sites meeting Personal/Social Skills and Commitment to Learning benchmarks is in decline.
SAC sites saw decrease in Commitment to Learning
For SAC, PQA means are decreasing slightly.
For SAC, percentage of sites meeting Instructional Context benchmark decreased.

41. How are we doing? The Good News
Grades maintenance/increase is back on an upward trend.
For 21st CCLC, quality (based on PQA) is improving.
Nearly all sites met Organizational Context and Instructional Context benchmarks.
The number of sites meeting External Relationships benchmark is improving.
Afterschool attendance is increasing.
For 21st CCLC, program quality is related to maintenance/increase of science grades
For SAC sites, program quality is related to a number of youth and parent outcomes.
Youth work habits and math/reading efficacy are significantly related to grade maintenance/increase.
DESSA-mini results indicate that nearly one-quarter of youth are improving social- emotional functioning over time.

42. Contact Information
Jason Patrie (573) Wayne Mayfield (573)