What Was Learned from a Second Year of Implementation IES Research Conference Washington, DC June 8, 2009 William Corrin, Senior Research Associate MDRC

Presentation Overview Project Background Do teachers need more than a year to master new practices? -Year 2 Implementation Findings (and Year 1 Comparison) -Impact findings -Exploratory findings o Two-Year Teachers and Replacement Teachers o Two-Year Teachers: Year 1 vs. Year 2 Summary

Project Background

The ERO Study The ERO study is an impact evaluation of two supplemental literacy programs targeted at 9 th grade students with limited reading skills. Key research question: -What are the impacts of the two interventions, together and separately, on ninth-grade students’ reading achievement and reading behaviors?

The Interventions Two literacy programs targeted at 9 th grade students with limited literacy skills (2-5 years below grade level) -Reading Apprenticeship Academic Literacy (RAAL) -Xtreme Reading (XR) The programs are supplemental reading classes -Designed as full-year courses that replace a 9 th grade elective class (rather than a core academic class) The two programs share common goals and instructional principles

Delivery of the Interventions Each high school implemented one of the two programs for ninth graders for two school years (Cohort 1) and (Cohort 2) One ERO teacher per high school -Experienced full-time ELA or social studies teacher was trained to teach the ERO reading course -Responsible for 4 sections of students each Training and technical assistance were provided to teachers by the developers -Summer training institutes, school-year off-site booster training sessions and on-site coaching

Implementation Data Collection To answer questions about how well the programs were put into place Allows us to answer questions about how implementation compared across program years: Do teachers do better with more experience teaching these programs? That is, is implementation stronger in the second year compared to the first? Allows us to investigate associations between implementation and impacts

Measuring Implementation Fidelity Two site visits in Year 2 (one per semester; only one site visit in Year 1) Implementation fidelity was assessed on two dimensions: -Classroom learning environment -Comprehension instruction Classroom observation ratings were obtained for 6 constructs common to both programs, and 7 program-specific constructs -Ratings are based on a 3-point scale

Measuring Implementation Fidelity (cont.) Composite ratings for each of the two dimensions were calculated, and then used to classify sites as: -“well aligned” (average rating > 2) -“moderately aligned” (average rating = 1.5 – 1.9) -“poorly aligned” (average rating < 1.5)

Implementation Findings

ERO teachers in Year 2 -Of the 34 ERO teachers in Year 2: o 25 had taught the entire first year of the study (13 RAAL, 12 XR) o 2 had taught part of the first year (both XR) o 7 were new to the programs (4 RAAL, 3 XR)  All Year 2 teachers taught the entire year

Implementation Findings (cont.) Implementation fidelity in Year 2 -In terms of classroom learning environment, 1 school was rated as poorly aligned with program models (vs. 4 schools in Year 1) -In terms of reading comprehension instruction, 1 school was rated as poorly aligned with program models (vs. 9 schools in Year 1) -23 schools were rated as well aligned on both dimensions (vs. 16 schools in Year 1) -Similar pattern of implementation findings at RAAL and XR schools

Implementation Findings (cont.) Implementation fidelity rated higher in Year 2 -In terms of classroom learning environment, the average rating at the spring site visit was 2.5 in Year 2 compared to 2.2 in Year 1 -In terms of reading comprehension instruction, the average rating at the spring site visit was 2.3 in Year 2 compared to 1.9

Learning Environment By Site Visit

Comprehension Instruction by Site Visit

Impact Findings

Random Assignment and Analysis Random assignment: -School-Level o 34 schools randomly assigned within 10 districts to RAAL or Xtreme Reading (17 RAAL sites and 17 Xtreme Reading sites) -Student-Level o In each high school, eligible students were randomly assigned to either:  Enroll in an ERO class (ERO group) or  Take one of the regular elective classes (non-ERO group) Analysis: Impacts are estimated by comparing the outcomes of students in the ERO and non-ERO group

Impact Findings: Reading Achievement Reading achievement was measured using the GRADE, a standardized reading assessment. Overall positive impact on reading comprehension SD for Cohort SD for Cohort 2 No impact on vocabulary -Same result for both cohorts

Impact Findings: Reading Behaviors Cohort 1 -Estimated impacts on three reading behaviors were not statistically significant (frequency of school- related reading, freq. of non-school-related reading, use of reading strategies) Cohort 2 -Estimated impact on students’ use of the reading strategies taught by the programs is 0.09 SD and statistically significant

Exploratory Analyses

Implementation Fidelity by Teacher Experience and Year Implementation ratings by year and by teacher experience:

Implementation Fidelity by Teacher Experience and Year Overall implementation fidelity: -Of the 25 sites with two-year teachers, 19 had higher overall implementation ratings in the second year compared to the first year -Of the 9 sites with replacement teachers, 8 sites had higher overall implementation ratings with the replacement teacher compared to the original teacher

Composite Fidelity Scores by Site Visit

Variation in Impacts Impacts on Student Outcomes by Teacher Experience Teaching ERO: Year 2

Variation in Impacts (cont.) Impacts on Student Outcomes by Implementation Year: Schools w/ Two-Year Teachers (N=25)

Summary Supplemental literacy courses for 9 th grade students can have a positive impact on students’ reading comprehension. These programs can be implemented with reasonable fidelity in a diverse array of high schools, and implementation fidelity can improve over time. On average, implementation fidelity was stronger in the second year. There is suggestive, but not convincing, evidence that teachers gained greater mastery of the programs having a second year to teach it. However, impacts on students reading comprehension remained the same. (Impacts on students’ use of reading strategies were greater.)

Further questions? William Corrin -(212) Marie-Andree Somers -(212) Paul Strasberg -(202)