The Impact of Literacy Coaching on Teachers’ Value- Added to Student Learning in Literacy Collaborative Gina Biancarosa, University of Oregon Anthony S. Bryk, The Carnegie Foundation for the Advancement of Teaching Allison Atteberry, Stanford University Heather Hough, Stanford University Institute of Education Sciences Annual Conference June 2010

Key Features of Literacy Collaborative Comprehensive school reform program designed to improve elementary children’s reading, writing, and language skills primarily through school-based coaching Used in over 700 elementary schools in 200 districts across 26 states Intensive professional development of coaches (selected from school faculty) – Trained over one year (Lesley University and the Ohio State University) – Ongoing support from local and national network Coach’s role and duties – Half-time teaching, half-time coaching – In-school professional development courses – One-on-one coaching sessions

Main Research Questions Does Literacy Collaborative improve the value- added to student literacy learning? Can Literacy Collaborative effects be attributed to coaching, either directly or indirectly?

Student Data Value-added analyses focused on grades exposed to LC professional development (K-2) Sample: 8576 children, 341 teachers, and 17 coaches in 17 public schools across 8 states in the Eastern U.S. Children tested in fall and spring for 4 years to measure change over time in students’ literacy learning using: – Dynamic Indicators of Basic Early Literacy Skills (DIBELS) – Terra Nova in spring Low Income46.0% Race/Ethnicity African-American Latino Other White 15.5% 5.8% 7.2% 70.6% Limited English Proficiency4.0%

Accelerated Longitudinal Cohort Design 6 cohorts studied over 4 years Year of Study First YearSecond YearThird YearFourth Year FallSpringFallSpringFallSpringFallSpring K CCDDEEFF 1 BBCCDDEE 2 AABBCCDD Grade Training year Year 1 of implementation Year 2 of implementation Year 3 of implementation

Our early literacy scale Equal differences on scale imply equal differences on the trait measured at any level Reported in logits (which describe the probability of a student with a given ability level getting a particular item right or wrong) But what do they mean given the particular assessments used? Mean at K entry Names about 30 letters in a minute Very low phonemic awareness (PA) Mean at K end & 1 st grade entry Accurate and fast letter recognition Good initial sound PA Little evidence of decoding Mean at 1 st grade end & 2 nd grade entry Accurate (not fast) PA Reads wpm Answers 1/3 of 1 st grade comprehension questions correctly Mean at 2 nd grade end Mastery of component skills Reads 90 wpm Answers 2/3 of 1 st grade comprehension questions correctly, 1/3 of 2 nd grade questions correctly

Value-added Hierarchical Cross- classified Effects Modeling Four Levels – time : (students x teachers) : school – Repeated measures on students (level 1) – Students (level 2) who cross Teachers (level 3) over time – All nested within Schools (level 4) The analysis model can be conceptualized as a joining of 2 separate multi-level models – One two-level model for individual growth in achievement over time, and – A second two-level model which represents the value-added that each teacher in a school contributes to student learning in that school in a particular year.

Value-added effects by year (prior to adding coaching as predictor) Year 1Year 2Year 3 Average value-added (overall) Performance improvement 16%28%32% Effect size School 95% plausible value-added range ±.23±.28±.37 Teacher 95% plausible value-added range ±.51±.71±.91 Average student learning growth in an academic year (1.02 logits)

Explaining variability in value-added effects Tested models with cumulative number of coaching sessions per year (derived from coach logs) – Per teacher – Averaged across teachers at school-level Also tested a variety of controls thought to influence teachers’ openness to, participation in, and selection for coaching – Prior use of reform literacy practices – Role conception – School commitment – New to school

Summary of findings Coaching at the teacher level significant Coaching at the school level not significant Teacher expertise of implementation not significant Only one teacher characteristic significant (role conception), but only in one year

Conditional value-added effects Year 1Year 2Year 3 Average value-added for teacher receiving NO coaching 0.26 *** 0.17 * 0.14 ns Role conception -.01 ns.04 *.01 ns Teacher expertise 0.02 ns ns 0.03 ns Value-added per coaching session (cumulative) *.012 *

Comparing Coaching Value-added to Unconditional Mean Value-Added Year 1Year 2Year 3 Value-added per coaching session (cumulative) Effect size per session Mean cumulative coaching sessions Mean coaching value-added Unconditional mean value-added Proportion accounted for by coaching NA

Across Seventeen Schools, Over Time 17 Variability in Coaching between Schools

Variability of Coaching between and within Schools: A Tale of Two Schools School 10: Riverside Staff size = 14 Value-added: Y1  Y2  Y  0.22  0.25 Starts below average and improves Variability between teachers decreases from Y1 to Y3 School 16: Tyson William Staff size = 14 Value-added: Y1  Y2  Y  0.13  0.01 Starts at average and deteriorates Variability between teachers increases from Y1 to Y3 How can we make sense of what happened in these two schools?

School 10: Coaching Sessions Accumulated per Teacher

School 16: Coaching Sessions Accumulated per Teacher

Network Analysis Conducted by Allison Atteberry & Tony Bryk Pre- and post-teacher surveys asked who talked to about instruction and student problems in literacy (up to 7) and how often Shape color = grade level Shape size = PD “dosage” Lines and arrows = reported ties Arrow weight = reported frequency of consultation Black box = coach

School 10 Riverside 2005

School 16 Tyson William Elem 2005

School 10 Riverside 2008

School 16 Tyson William Elem 2008

Summary of findings Evidence that the mechanism for improved value- added shifts from over time – Year 1: Coaching has no value-added – Year 2: Coaching begins to add to value-added for student learning – Year 3: Coaching becomes the primary mechanism for value-added to student learning Cumulative coaching explains differences in teacher value-added effects, but not school effects

Implications Coaching largely mediates teachers’ value- added to student learning Not in Year 1, but in Year 2 and especially 3 – Selection effects? – Dosage effects? – Developmental/expertise effect for teachers? – Developmental/expertise effect for coaches?

Future Steps Exploration of the role of coach – Expertise and its development – Coach decision-making, especially what influences whom coaches target Exploration of the influence of school context – Teacher influence in school decisions – Principal leadership and supportiveness – Trust

Limitations Limited sample, especially at school level, limits ability to explore contextual mechanisms Professional development for coaches is more intense than in most other models Coaching embedded in a school-wide reform model that included – Professional development courses – Detailed literacy instruction framework

Thank you!

Variability in school value-added, year 1 Average student gain per academic year No effect Year 1 mean effect (.16) High value-added schools Low value-added schools

Variability in school value-added, year 2 Average student gain per academic year No effect Year 1 mean effect (.16) Year 2 mean effect (.28)

Variability in school value-added, year 3 Average student gain per academic year No effect Year 1 mean effect (.16) Year 2 mean effect (.28) Year 3 mean effect (.33)

Variability in teacher value-added within 2 schools Average student gain per academic year No effect Year 1 mean effect (.16) Year 2 mean effect (.28) Year 3 mean effect (.33)