Supplemental Educational Services (SES) Virtual Academy: Data and Reporting Series 2013-2014 Statewide and Provider Evaluation Results Federal Programs Unit Data, Program Evaluation and Reporting (DPER) By Tina Negley and Nazanin Mohajeri-Nelson January 7, 2016

Background

4/6/2019 Program Reach in 2013-2014 120 schools in 48 districts required to offer SES Turnaround and Priority Improvement schools operating a Title I program for two consecutive years 88 schools (73%) in 28 districts served students with SES 3,967 students participated in SES Below proficiency on state assessments (TCAP, ACCESS, READ Act) Other body of evidence (ex. local assessments) New ESEA waiver implemented in the 2015-16 year also requires Title I Focus schools to provide SES and Choice 3

Students Served Longitudinal Reach of Program 4/6/2019 Students Served Longitudinal Reach of Program Decrease in the number of students served over the past 3 years Majority of students participated in SES for the first time, but more than ¼ (26%) of students received services in at least one prior year 4

Demographics of Students Served 4/6/2019 Demographics of Students Served A larger proportion of disaggregated groups served than the state percentage for that group 80% of students served were Hispanic or Latino, compared to 33% in state Notes: Minority category includes all race/ethnicity categories excluding White students; EL category includes all non-English proficient (NEP), limited English proficient (LEP), fluent English proficient (FEP; monitor years 1 and 2, and exited), and former English language learner (FELL) students. 5

Grades of Students Served Highest percentage of students served (85%) were elementary school students (K-5) 6

Hours of Services Received Majority of students (55%) received more than 25 hours of services 7

Location and Format of Services Received Majority of students received services at their school (87%) and in a group format (83%) Format 8

Statewide Effectiveness

Methods ~ Data Used READ Act TCAP and ACCESS 4/6/2019 Methods ~ Data Used READ Act Students with valid DRA2 data for 2013 and 2014 TCAP and ACCESS Students with valid scores for 2013 and 2014 Must have corresponding growth percentile for 2014 Students with multiple assessment records (more than one test score) were removed Students must have progressed one grade from 2012 to 2013 (i.e. students held back or students who skipped a grade were excluded) DRA2 was selected for students in K-3 because it was the primary READ Act assessment taken by the students receiving services 10

Methods ~ Inclusion Rules 4/6/2019 Methods ~ Inclusion Rules Students served Students finished at least 50% of their contracted hours and at least 75% of the 20 hours minimum required by the state prior to assessment cut-point Comparison group Enrolled in participating SES schools Equal proportions for each grade / proficiency category Random samples drawn to ensure similar demographics Effectiveness analyses Improvement calculations only include students starting below grade-level (DRA2), Unsatisfactory or Partially Proficient (TCAP), or Levels 1-4 (ACCESS) MGP calculations include all students 11

Reading Statewide Effectiveness of Students Grades K-3 4/6/2019 Reading Statewide Effectiveness of Students Grades K-3 In total, 1,796 students in grades K-3 received reading services Majority of students started below grade-level target (67%) 19% of SES students improved proficiency, compared to 15% of comparison group 12

Reading Statewide Effectiveness of Students Grades 4-10 4/6/2019 Reading Statewide Effectiveness of Students Grades 4-10 Majority of students started Unsatisfactory or Partially Proficient (81%) A higher percent of SES students increased in PL than comparison MGP of SES students was significantly higher 13

Math Statewide Effectiveness of Students Grades 4-10 4/6/2019 Math Statewide Effectiveness of Students Grades 4-10 Majority of students started Unsatisfactory or Partially Proficient (77%) A higher percent of SES students increased in PL than comparison MGP of SES students was significantly higher 14

Writing Statewide Effectiveness of Students Grades 4-10 4/6/2019 Writing Statewide Effectiveness of Students Grades 4-10 Majority of students started Unsatisfactory or Partially Proficient (85%) MGP of SES students was significantly higher 15

ELD Statewide Effectiveness 4/6/2019 ELD Statewide Effectiveness ELs receiving SES services demonstrated higher growth on ACCESS ELs receiving services were also more likely to improve at least one proficiency level on ACCESS 16

Statewide Effectiveness: Hours Received

Reading Statewide Effectiveness by Hours Received 4/6/2019 Effectiveness of at least 25 hours of services Grades K-3 Reading Statewide Effectiveness by Hours Received A higher percent of students improved PL if they received at least 25 hours, compared to students receiving less than 25 hours MGP for students 4-10 was also higher if they received at least 25 hours Grades 4-10 18

Math Statewide Effectiveness by Hours Received 4/6/2019 Effectiveness of at least 25 hours of services Math Statewide Effectiveness by Hours Received Grades 4-10 Students receiving at least 25 hours had higher MGP than students receiving fewer services 19

Writing Statewide Effectiveness by Hours Received 4/6/2019 Effectiveness of at least 25 hours of services Writing Statewide Effectiveness by Hours Received Grades 4-10 Students receiving at least 25 hours had higher MGP than students receiving fewer services Note: Asterisks (*) indicate one or more cells contain a count less than 16, and have therefore been suppressed. Ranges have been provided for the percent of students improving at least one proficiency level to allow for comparisons between groups. 20

Statewide Effectiveness by Completion Rate 4/6/2019 Improved Proficiency for students starting below grade-level (DRA2), Unsatisfactory or Partially Proficient (TCAP), or Levels 1-4 (ACCESS) Statewide Effectiveness by Completion Rate TCAP MGP was higher in all content areas for students completing all of their contracted hours Growth for all students included in evaluation 21

Statewide Effectiveness: Format of Services

Statewide Effectiveness by Format 4/6/2019 Individual vs. Group (In-person only) Reading Grades K-3 Statewide Effectiveness by Format Reading Grades 4-10 Students receiving reading services in a group format improved PL more and demonstrated higher growth Students receiving math services one-to-one were more likely to improve and demonstrated higher growth Math Grades 4-10 Note: Asterisks (*) indicate one or more cells contain a count less than 16, and have therefore been suppressed. Ranges have been provided for the percent of students improving at least one proficiency level to allow for comparisons between groups. 23

Reading and Math Statewide Effectiveness by Format 4/6/2019 In-person vs. Online Reading Grades 4-10 Reading and Math Statewide Effectiveness by Format Greater success for in person formats Math Grades 4-10 Note: Asterisks (*) indicate one or more cells contain a count less than 16, and have therefore been suppressed. Ranges have been provided for the percent of students improving at least one proficiency level to allow for comparisons between groups. 24

Provider Effectiveness

K-3 Reading Providers 4/6/2019 Note: Table only includes providers with at least 16 students starting below grade-level target. 26

Grades 4-10 Reading Providers 4/6/2019 Grades 4-10 Reading Providers Note: Table only includes providers with at least 16 students starting unsatisfactory or partially proficient (% improved) and/or at least 20 students with valid growth data (MGP). 27

Grades 4-10 Math Providers 4/6/2019 Grades 4-10 Math Providers Note: Table only includes providers with at least 16 students starting unsatisfactory or partially proficient (% improved) and/or at least 20 students with valid growth data (MGP). 28

Grades 4-10 Writing Providers 4/6/2019 Grades 4-10 Writing Providers Note: Table only includes providers with at least 16 students starting unsatisfactory or partially proficient (% improved) and/or at least 20 students with valid growth data (MGP). 29

4/6/2019 ELD Providers Note: Table only includes providers with at least 16 students starting unsatisfactory or partially proficient (% improved) and/or at least 20 students with valid growth data (MGP). 30

Effective Providers 8 providers were considered effective in both Math and Reading, grades 4-10: Above & Beyond Learning (Also reading K-3 and writing) Advanced Brain Gym Plus (Also writing and ELD) Alternatives Unlimited Aurora Public Schools RWaM (Also writing) Club Z (Also reading K-3, writing, and ELD) Imagine Learning (Also reading K-3) Orion’s Mind (Also reading K-3) Sylvan Learning Center 31

Additional Information

Where to Find More Information… Evaluation reports are posted on the Data, Program Evaluation and Reporting Office webpage http://www.cde.state.co.us/fedprograms/dper/evalrpts#tiases: Currently, evaluation reports for 2007-08 through 2012-13 are available 2013-14 evaluation reports will be posted soon 33

Contact Information Questions regarding the program evaluation, reporting, and data use: Nazanin Mohajeri-Nelson mohajeri-nelson_n@cde.state.co.us 303-866-6205 Tina Negley negley_t@cde.state.co.us 303-866-5243 Questions regarding SES data collection and submission: Alexandra Rechlin rechlin_a@cde.state.co.us 303-866-4571 Questions regarding SES program: Stacy Goodman goodman_s@cde.state.co.us 303-866-6298 34