1. Formative Assessment and Student Achievement: Two Years of Implementation of the Keeping Learning on Track® Program Courtney Bell (ETS) Jonathan Steinberg (ETS) Dylan Wiliam (Institute of Education, University of London) Caroline Wylie (ETS)

2. Keeping Learning on Track ® Program Developed from research-basis around formative assessment (Crooks, 1997; Natriello, 1998; Kluger & DeNisi; 1996; Black & Wiliam, 1998c; Nyquist, 2003; Brookhart, 2005) and the power of teacher learning communities (Grossman, Wineburg, & Woolworth 2001; Thompson, & Goe, 2006) Five research-based strategies (Leahy et al., 2005) Engineering effective classroom discussions, questions, and learning tasks that elicit evidence of student learning; Clarifying and sharing learning intentions and criteria for success; Providing feedback that moves learners forward; Activating students as the owners of their own learning; Activating students as instructional resources for one another.

3. KLT Components 2-day introductory workshop 2-day TLC leader workshop Monthly school-based TLC meetings Structured and supported by TLC Leader modules Sufficient content for two years of monthly meetings

4. Working in Cleveland 2005-06 Work began with 10 schools (Cohort A) 2006-07 Nine of the original schools (1 closed) continued Additional 5 new schools (Cohort B) All Schools All in third or fourth year of failing to make Adequate Yearly Progress

5. Details of sample Numbers of students in each grade in CMSD, 2005-2006 and 2006-2007

6. Data Used 2005-06 and 2006-07 Ohio Achievement Tests for Grades 3-8 Reliability Math: 0.85 to 0.90 Reading: 0.87 to 0.89 CMSD scores well below state mean 0.65 to 0.83 sd below state mean in math 0.60 to 0.71 sd below state mean in reading High student mobility 4400 of 16500 students changed school from 05-06 to 06-07 400 students were retained in grade or were promoted by more than one grade from 05-06 to 06-07 Analysis excluded these students No data available on: Teacher assignment Fidelity of implementation

7. Data analysis Univariate HLM Dependent variable: 2006-2007 test scores Students nested in schools, schools nested in treatment 2005-2006 scores used as a covariate No significant treatment effect p-values typically well over 0.5 Suggests weak effects rather than low power

8. Exploratory analyses Cleveland school data from the OH DOE website to identify matched schools Number of students Proportion of LEP students % of longevity (measure of teacher turnover) % of African American students Accountability designation in 06-07 (defined by state accountability system) Year of improvement status (defined by state accountability system) % of core courses not taught by a highly qualified teacher If a tie-breaker was needed, the final category was % of special education students

9. Exploratory results: math KLTNon-KLT Grade1 year2 yearAll1 year2 yearp-value 41.450.260.030.590.630.89 51.311.450.070.140.070.60 60.230.160.84 0.060.84 70.520.140.190.090.900.89 80.870.240.631.740.000.84 Model coefficients for 2006-2007 math scores, in standard deviations (positive/negative)

10. Exploratory results: reading KLTNon-KLT Grade1 year2 yearAll1 year2 yearp-value 40.770.800.541.470.870.52 51.111.490.870.660.590.10 60.370.690.200.530.370.82 70.590.630.040.59 0.75 80.790.291.291.490.290.08 Model coefficients for 2006-2007 reading scores, in standard deviations (positive/negative)

11. Comparison of matched schools Grade 1 year2 year 40.700.07 50.452.08 60.901.06 70.000.04 82.280.00 KLT schools - matched non-KLT schools (Coefficients for reading scores in standard deviations)

12. Discussion No statistically significant effect of KLT Exploratory analyses Math: no evidence of effect Reading: some evidence in earlier grades Limitations Fidelity of implementation Small effects Low statistical power Next steps Research designs based on “theory of action” of KLT Impact of training on teacher behaviors Impact of changes in teacher behaviors on students:  Attitudes  Outcomes

13. For More Information To view, download and print the papers: http://www.ets.org/research/aerancme08.html Conference files will be posted on the web for one month