Coaching in Literacy Collaborative and Its Effects on Teachers and Students Gina Biancarosa, University of Oregon Anthony S. Bryk, Carnegie Foundation for the Advancement of Teaching Allison Atteberry, Stanford University Heather Hough, Stanford University Annual Meeting of the Society for Research on Educational Effectiveness March 2010

Key Features of Literacy Collaborative Comprehensive school reform program designed to improve elementary children’s reading, writing, and language skills primarily through school-based coaching Used in over 700 elementary schools in 200 districts across 26 states Intensive professional development – Coaches trained over one year (Lesley University and the Ohio State University) – Ongoing support from local and national network Coaches – In-school professional development courses – One-on-one coaching: viewed as the high leverage activity

Anatomy of a coaching session – Pre-briefing – Observation – Modeling – Debriefing Elements of literacy instruction – Interactive read aloud – Shared reading – Guided reading – Interactive writing – Writing workshop – Word study Key Features of Literacy Collaborative

Main Research Questions Does Literacy Collaborative improve the value-added to student literacy learning? Can any effects of Literacy Collaborative be indirectly attributed to coaching via teachers’ changing expertise implementing the instructional practices? Can any effects of Literacy Collaborative be directly attributed to coaching? – Does overall coaching activity in a school predict value- added to student literacy learning? – Does individual teacher participation in coaching predict value-added to student literacy learning?

Student Data Value-added analyses focused on grades exposed to LC professional development (K-2) Sample: 8576 children, 341 teachers, and 17 coaches in 17 public schools across 8 states in the Eastern U.S. Children tested in fall and spring for 4 years to measure change over time in students’ literacy learning using: – Dynamic Indicators of Basic Early Literacy Skills (DIBELS) – Terra Nova in spring Low Income46.0% Race/Ethnicity African-American Latino Other White 15.5% 5.8% 7.2% 70.6% Limited English Proficiency4.0%

Accelerated Longitudinal Cohort Design 6 cohorts studied over 4 years Year of Study First YearSecond YearThird YearFourth Year FallSpringFallSpringFallSpringFallSpring K CCDDEEFF 1 BBCCDDEE 2 AABBCCDD Grade Training year Year 1 of implementation Year 2 of implementation Year 3 of implementation

Our early literacy scale Equal differences on scale imply equal differences on the trait measured at any level Reported in logits (which describe the probability of a student with a given ability level getting a particular item right or wrong) But what do they mean given the particular assessments used? Mean at K entry Names about 30 letters in a minute Very low phonemic awareness (PA) Mean at K end & 1 st grade entry Accurate and fast letter recognition Good initial sound PA Little evidence of decoding Mean at 1 st grade end & 2 nd grade entry Accurate (not fast) PA Reads wpm Answers 1/3 of 1 st grade comprehension questions correctly Mean at 2 nd grade end Mastery of component skills Reads 90 wpm Answers 2/3 of 1 st grade comprehension questions correctly, 1/3 of 2 nd grade questions correctly

Additional Measures Year of implementation ParticipantConstructInstrument123 TeachersBackground characteristicsSurvey ●● Coaching participationCoach logs ●●● Frequency of implementation Coach report ●● Expertise of implementation Coach report ●●● CoachesBackground characteristicsSurvey ●● SchoolsContextual characteristicsSurvey ●● Social-professional network Survey ●●

Prior Findings: Coaching sessions per month (n=249) Coefficient (se)Standardized effect size Teacher-level moderators Role conception0.048 (0.021) *0.026 School commitment0.092 (0.0242) *** years prior teaching experience (0.046) * School-level moderators K-2 Staff Size (0.110) *** Perceived support for LC0.152 (0.060) *0.081 Coach’s prior training0.148 (0.057) *0.079 Teacher influence in school decisions (0.064) *0.054 * p <.05; ** p <.01; *** p <.001

Prior Findings: Frequency and expertise of implementation (n=249) Expertise Coefficient (se)Standardized effect size Initial status, Teacher-level moderators School commitment0.142 (0.073) **0.252 Prior professional development0.134 (0.078) *0.238 ≤ 3 years prior teaching experience (0.199) *** Growth, Teacher-level moderators Coaching participation0.130 (0.049) ***0.231 * p <.05; ** p <.01; *** p <.001

Value-added Hierarchical Cross- classified Effects Modeling Four Levels – time : (students x teachers) : school – Repeated measures on students (level 1) – Students (level 2) who cross Teachers (level 3) over time – All nested within Schools (level 4) The analysis model can be conceptualized as a joining of 2 separate multi-level models – One two-level model for individual growth in achievement over time, and – A second two-level model which represents the value-added that each teacher in a school contributes to student learning in that school in a particular year.

Hierarchical Crossed Value-added Effects Model Individual growth parameters overall value- added effects teacher-level school-level value-added effects

Value-added effects by year (prior to adding coaching as predictor) Year 1Year 2Year 3 Average value-added (overall) Performance improvement 16%28%32% Effect size Ave. student learning growth is 1.02 per academic year

School 95% plausible value- added range ±.23 Variability in school value-added, year 1 Average student gain per academic year No effect Year 1 mean effect (.16) High value-added schools Low value-added schools

School 95% plausible value- added range ±.23±.28 Variability in school value-added, year 2 Average student gain per academic year No effect Year 1 mean effect (.16) Year 2 mean effect (.28)

School 95% plausible value- added range ±.23±.28±.37 Variability in school value-added, year 3 Average student gain per academic year No effect Year 1 mean effect (.16) Year 2 mean effect (.28) Year 3 mean effect (.33)

Teacher 95% plausible value- added range ±.51 Variability in teacher value-added within schools, year 1 Average student gain per academic year No effect

Teacher 95% plausible value- added range ±.51±.71 Variability in teacher value-added within schools, year 2 Average student gain per academic year No effect

Teacher 95% plausible value- added range ±.51±.71±.91 Variability in teacher value-added within schools, year 3 Average student gain per academic year No effect

Explaining variability in value-added effects Tested models with cumulative number of coaching sessions per year (derived from coach logs) – Per teacher – Averaged across teachers at school-level Also tested a variety of controls thought to influence teachers’ openness to, participation in, and selection for coaching – Prior use of reform literacy practices – Role conception – School commitment – New to school

Hierarchical Crossed Value-added Effects Model Individual growth parameters overall value- added effects teacher-level school-level value-added effects Predictors added to baseline and LC value-added effects

Summary of findings Only one teacher characteristic significant Teacher expertise of implementation not significant Coaching at the school level not significant Coaching at the teacher level significant

Teachers’ role conception High scorers: Teachers who take an active stance in their professional role in terms of initiating contact and offering help to colleagues Higher value-added to student literacy learning in their schools in baseline and Y2 BaselineYear 1Year 2Year ** ns.042 *.009 ns

Average Value-added of Coaching by year Year 1Year 2Year 3 Average value-added for teacher receiving NO coaching 0.26 *** 0.17 * 0.14 ns Role conception -.01 ns.04 *.01 ns Teacher expertise 0.02 ns ns 0.03 ns Value-added per coaching session (cumulative) *.012 *

Average Value-added of Coaching by year Year 1Year 2Year 3 Value-added per coaching session (cumulative) *.012 * Mean cumulative coaching sessions Mean coaching value- added Unconditional average value-added (overall) Proportion accounted for by coaching NA Cumulative coaching sessions min-max

Across Seventeen Schools, Over Time 17

Value-added by coaching, year 1 No coaching effect

Value-added by coaching, year 2 No coaching effect

Value-added by coaching, year 3 No coaching effect

Summary of findings Evidence that the mechanism for improved value- added shifts from over time – Year 1: Coaching has no value-added – Year 2: Coaching begins to add to value-added for student learning – Year 3: Coaching becomes the primary mechanism for value-added to student learning Cumulative coaching explains differences in teacher value-added effects, but not school effects

Implications Coaching explains differences in teachers’ value- added to student learning Shift in coaching effects from negative in Year 1 to positive in Years 2 and 3 raises interesting hypotheses but offer no answers – A selection effect (on the part of coach or teacher) – A dosage effect – A change in coaching expertise effect – Unexplored school/coach effects Direct positive effects of coaching on students appear to take time to emerge

Limitations Limited sample, especially at school level, limits ability to explore contextual mechanisms Coaching was embedded in a school-wide reform model Professional development for coaches is more intense than in most other models

Future Steps Continued analyses of current data – Length of coaching session – Focus of coaching session – Observation vs. modeling Development and piloting of the Performance-based Assessment of Literacy Coaching (PALC)

Thank you!