External Evaluation of the 2011 – 2014 Demonstration Project Presented at October 2013 Replication Forum

External Evaluators The Meadows Center for Preventing Educational Risk PI: Dr. Saro Mohammed Researchers: Myriam Lopez, Deborah Van Kummer Concordia University Site visits: Students in the Educational Administration Masters program

Logic Model

Theory of Change – Regional Level [Funders, partners, Region 13 ESC, and participating schools] [train and support PDers and coaches] to [change the number of regional Pders, conferences, and schools] leading to [regional collaboration between PDers, districts, and schools] and eventually [embedding, awareness, and use of SIM regionally]

Theory of Change – School Level [Teachers of struggling students – highly mobile, economically disadvantaged, with limited English proficiency, experiencing achievement gaps in reading] [receive training, feedback, support, and implement SIM] to [change the number of classes, and students using SIM] leading to [teacher collaboration, student engagement, academic achievement, and accurate SLD referrals] and eventually [multidisciplinary student use of SIM, and positive student behaviors, and high school outcomes]

Based on detailed evaluation logic model Schools evaluated on: Outputs: Process metrics (Teachers trained, reviews, etc.) Implementation fidelity (practices observed in walk- throughs, student feedback, etc.) Outcomes: Change over year for struggling learners TAKS/STAAR scale score comparison for all students, and raw scores for struggling students, versus comparison schools matched on size, demographic make-up, previous results Evaluation Methods

Data Sources All data collected by program staff EXCEPT Site visits: classroom walkthroughs, device checklists, LLT meeting observations State Assessments: TAKS 2011, STAAR 2012, STAAR 2013

Outputs – Fidelity of Implementation (School Level) Implementation has improved over 2 years

Outputs – Fidelity of Implementation Implementation is widespread

Outcomes Populations (defined in Fall 2011) Struggling learners (project schools only) Gates standard score of 85 or less Pre and Post test scores (typically beginning and end of year) All students Took regular TAKS & STAAR (not modified versions of tests)

Findings – Reading (Struggling students, Gates)

Findings – Reading (Struggling students, Gates, Year 1) For students identified as struggling in year 1, percentile changes in Gates from pre-test to post-test were notable: 6 th grade growth=3 rd to 6 th percentile; n=138 7 th grade growth=4 th to 10 th percentile; n=124 8 th grade growth=5 th to 9 th percentile; n=104

Findings – Reading (Struggling students, Gates, Year 2) Also, for students identified as struggling in year 2, percentile changes in Gates from pre- test to post-test were notable: 6 th grade growth=6 th to 12 th percentile; n=118 7 th grade growth=2 nd to 7 th percentile; n=209 8 th grade growth=5 th to 6 th percentile; n=154

Comparison Schools Created a focal, local, comparison group Schools were matched on (in order): number of students, Eco Dis percentage, bilingual/LEP percentage, mobility percentage, ethnic makeup of student population, historical TAKS Match schools kept within district where possible All match schools were within Region 13

Findings – Reading (All students) No significant effects on reading yet on schools overall Posttest (STAAR 2012 & STAAR 2013) means adjusted for pretest (TAKS 2011 & STAAR 2012 respectively) for 7 th and 8 th graders Pooled standard deviations and posttest adjusted means were used where available Within-grade effect sizes ranged from to 0.05

Evaluation Summary Implementation process: project schools are being trained/supported in their implementation as intended Output metrics (PD goals, practice usage) generally being achieved and acceptably consistent across schools For struggling students, trends are positive and noteworthy On proximal measures of reading, students who continue to struggle from year to year outpace expected annual growth (as determined by national norms) In first and second years of implementation, as expected, no statistical difference in distal outcomes (state assessments) between project and match schools Student academic growth much greater in most RAISEup schools versus comparison schools