1. Supplemental Educational Services (SES)
Virtual Academy: Data and Reporting Series
Statewide and Provider Evaluation Results
Federal Programs Unit
Data, Program Evaluation and Reporting (DPER)
By Tina Negley and Nazanin Mohajeri-Nelson
January 7, 2016

2. Background

3. 4/6/2019
Program Reach in
120 schools in 48 districts required to offer SES
Turnaround and Priority Improvement schools operating a Title I program for two consecutive years
88 schools (73%) in 28 districts served students with SES
3,967 students participated in SES
Below proficiency on state assessments (TCAP, ACCESS, READ Act)
Other body of evidence (ex. local assessments)
New ESEA waiver implemented in the year also requires Title I Focus schools to provide SES and Choice 3

4. Students Served Longitudinal Reach of Program
4/6/2019
Students Served Longitudinal Reach of Program
Decrease in the number of students served over the past 3 years
Majority of students participated in SES for the first time, but more than ¼ (26%) of students received services in at least one prior year 4

5. Demographics of Students Served
4/6/2019
Demographics of Students Served
A larger proportion of disaggregated groups served than the state percentage for that group
80% of students served were Hispanic or Latino, compared to 33% in state
Notes: Minority category includes all race/ethnicity categories excluding White students; EL category includes all non-English proficient (NEP), limited English proficient (LEP), fluent English proficient (FEP; monitor years 1 and 2, and exited), and former English language learner (FELL) students. 5

6. Grades of Students Served
Highest percentage of students served (85%) were elementary school students (K-5) 6

7. Hours of Services Received
Majority of students (55%) received more than 25 hours of services 7

8. Location and Format of Services Received
Majority of students received services at their school (87%) and in a group format (83%)
Format 8

9. Statewide Effectiveness

10. Methods ~ Data Used READ Act TCAP and ACCESS
4/6/2019
Methods ~ Data Used
READ Act
Students with valid DRA2 data for 2013 and 2014
TCAP and ACCESS
Students with valid scores for 2013 and 2014
Must have corresponding growth percentile for 2014
Students with multiple assessment records (more than one test score) were removed
Students must have progressed one grade from 2012 to 2013 (i.e. students held back or students who skipped a grade were excluded)
DRA2 was selected for students in K-3 because it was the primary READ Act assessment taken by the students receiving services 10

11. Methods ~ Inclusion Rules
4/6/2019
Methods ~ Inclusion Rules
Students served
Students finished at least 50% of their contracted hours and at least 75% of the 20 hours minimum required by the state prior to assessment cut-point
Comparison group
Enrolled in participating SES schools
Equal proportions for each grade / proficiency category
Random samples drawn to ensure similar demographics
Effectiveness analyses
Improvement calculations only include students starting below grade-level (DRA2), Unsatisfactory or Partially Proficient (TCAP), or Levels 1-4 (ACCESS)
MGP calculations include all students 11

12. Reading Statewide Effectiveness of Students Grades K-3
4/6/2019
Reading Statewide Effectiveness of Students Grades K-3
In total, 1,796 students in grades K-3 received reading services
Majority of students started below grade-level target (67%)
19% of SES students improved proficiency, compared to 15% of comparison group 12

13. Reading Statewide Effectiveness of Students Grades 4-10
4/6/2019
Reading Statewide Effectiveness of Students Grades 4-10
Majority of students started Unsatisfactory or Partially Proficient (81%)
A higher percent of SES students increased in PL than comparison
MGP of SES students was significantly higher 13

14. Math Statewide Effectiveness of Students Grades 4-10
4/6/2019
Math Statewide Effectiveness of Students Grades 4-10
Majority of students started Unsatisfactory or Partially Proficient (77%)
A higher percent of SES students increased in PL than comparison
MGP of SES students was significantly higher 14

15. Writing Statewide Effectiveness of Students Grades 4-10
4/6/2019
Writing Statewide Effectiveness of Students Grades 4-10
Majority of students started Unsatisfactory or Partially Proficient (85%)
MGP of SES students was significantly higher 15

16. ELD Statewide Effectiveness
4/6/2019
ELD Statewide Effectiveness
ELs receiving SES services demonstrated higher growth on ACCESS
ELs receiving services were also more likely to improve at least one proficiency level on ACCESS 16

17. Statewide Effectiveness: Hours Received

18. Reading Statewide Effectiveness by Hours Received
4/6/2019
Effectiveness of at least 25 hours of services
Grades K-3
Reading Statewide Effectiveness by Hours Received
A higher percent of students improved PL if they received at least 25 hours, compared to students receiving less than 25 hours
MGP for students 4-10 was also higher if they received at least 25 hours
Grades 4-10 18

19. Math Statewide Effectiveness by Hours Received
4/6/2019
Effectiveness of at least 25 hours of services
Math Statewide Effectiveness by Hours Received
Grades 4-10
Students receiving at least 25 hours had higher MGP than students receiving fewer services 19

20. Writing Statewide Effectiveness by Hours Received
4/6/2019
Effectiveness of at least 25 hours of services
Writing Statewide Effectiveness by Hours Received
Grades 4-10
Students receiving at least 25 hours had higher MGP than students receiving fewer services
Note: Asterisks (*) indicate one or more cells contain a count less than 16, and have therefore been suppressed. Ranges have been provided for the percent of students improving at least one proficiency level to allow for comparisons between groups. 20

21. Statewide Effectiveness by Completion Rate
4/6/2019
Improved Proficiency for students starting below grade-level (DRA2), Unsatisfactory or Partially Proficient (TCAP), or Levels 1-4 (ACCESS)
Statewide Effectiveness by Completion Rate
TCAP MGP was higher in all content areas for students completing all of their contracted hours
Growth for all students included in evaluation 21

22. Statewide Effectiveness: Format of Services

23. Statewide Effectiveness by Format
4/6/2019
Individual vs. Group (In-person only)
Reading Grades K-3
Statewide Effectiveness by Format
Reading Grades 4-10
Students receiving reading services in a group format improved PL more and demonstrated higher growth
Students receiving math services one-to-one were more likely to improve and demonstrated higher growth
Math Grades 4-10
Note: Asterisks (*) indicate one or more cells contain a count less than 16, and have therefore been suppressed. Ranges have been provided for the percent of students improving at least one proficiency level to allow for comparisons between groups. 23

24. Reading and Math Statewide Effectiveness by Format
4/6/2019
In-person vs. Online
Reading Grades 4-10
Reading and Math Statewide Effectiveness by Format
Greater success for in person formats
Math Grades 4-10
Note: Asterisks (*) indicate one or more cells contain a count less than 16, and have therefore been suppressed. Ranges have been provided for the percent of students improving at least one proficiency level to allow for comparisons between groups. 24

25. Provider Effectiveness

26. K-3 Reading Providers 4/6/2019
Note: Table only includes providers with at least 16 students starting below grade-level target. 26

27. Grades 4-10 Reading Providers
4/6/2019
Grades 4-10 Reading Providers
Note: Table only includes providers with at least 16 students starting unsatisfactory or partially proficient (% improved) and/or at least 20 students with valid growth data (MGP). 27

28. Grades 4-10 Math Providers
4/6/2019
Grades 4-10 Math Providers
Note: Table only includes providers with at least 16 students starting unsatisfactory or partially proficient (% improved) and/or at least 20 students with valid growth data (MGP). 28

29. Grades 4-10 Writing Providers
4/6/2019
Grades 4-10 Writing Providers
Note: Table only includes providers with at least 16 students starting unsatisfactory or partially proficient (% improved) and/or at least 20 students with valid growth data (MGP). 29

30. 4/6/2019
ELD Providers
Note: Table only includes providers with at least 16 students starting unsatisfactory or partially proficient (% improved) and/or at least 20 students with valid growth data (MGP). 30

31. Effective Providers
8 providers were considered effective in both Math and Reading, grades 4-10:
Above & Beyond Learning (Also reading K-3 and writing)
Advanced Brain Gym Plus (Also writing and ELD)
Alternatives Unlimited
Aurora Public Schools RWaM (Also writing)
Club Z (Also reading K-3, writing, and ELD)
Imagine Learning (Also reading K-3)
Orion’s Mind (Also reading K-3)
Sylvan Learning Center 31

32. Additional Information

33. Where to Find More Information…
Evaluation reports are posted on the Data, Program Evaluation and Reporting Office webpage
Currently, evaluation reports for through are available
evaluation reports will be posted soon 33

34. Contact Information
Questions regarding the program evaluation, reporting, and data use:
Nazanin Mohajeri-Nelson
Tina Negley
Questions regarding SES data collection and submission:
Alexandra Rechlin
Questions regarding SES program:
Stacy Goodman 34