Striving to Link Teacher and Student Outcomes: Results from an Analysis of Whole-school Interventions Kelly Feighan, Elena Kirtcheva, and Eric Kucharik Research for Better Schools, Philadelphia, PA American Evaluation Association Annual Meeting, November 12, 2009 in Orlando, Florida 1

Study Purpose  Investigate which variables best explain student reading outcomes following teacher professional development  Explore the contextual reasons that help explain why no intervention “impact” was detected  Inform educational policy and improve rigor of educational research 2

Project Background  Federal Striving Readers program aimed at improving pedagogy and student achievement  Schools were matched in pairs and then randomly assigned to the treatment or control condition  Professional Development: four-semester course, onsite literacy coaching, leadership seminar, and curricular material  Developer’s hypothesis: integrating literacy strategies in content areas will yield student gains 3

Factors Affecting Student Learning  Student - level: SES, socio-demographic variables, family background, early development (Barton & Coley, 2009)  Teacher/classroom-level: expectations, preparation, experience, class size (Cohen, McCabe, Mitchelli, and Pickeral, 2009)  School-level: school climate - safety, student- adult and peer relationships, curriculum rigor (Cohen, McCabe, Mitchelli, and Pickeral, 2009) 4

Study Participants 30 ELA teachers taught at eight schools –16 taught at intervention schools –14 taught at comparison schools 2,114 students linked to these teachers –state assessment reading scores (N = 2,064) –ITBS scale reading scores (N = 1741) 5

Methodology  Quantitative data sources:  RBS teacher survey  School district school climate survey  Department of Education teacher HQT statistics and student discipline data  Students’ scores on state assessment and ITBS 6

Methodology  Qualitative data sources:  Observations  56 classrooms (Year 1)  48 classrooms (fall of Year 2)  10 paired observations (spring of Year 2)  Interviews  8 principals and 19 school improvement team members in Years 1 and 2  Focus groups: seven groups with 62 teachers 7

Research Hypotheses  Exposure to professional development participants will yield gains in reading achievement  Including contextual variables in impact analysis will increase explanatory power of results 8

Quantitative Analysis  Used Hierarchical Linear Modeling (HLM) to predict student performance based on student-, teacher-, and school-level characteristics  Fully unconditional model represents how variation in an outcome measure is allocated across the three different levels 9

Variables Included in the HLM 10 Student Gender Race ELL State pre-test ITBS pre-test Grade level Attendance Teacher Had >8 hours of PD in past year Education level Years teaching Completed intervention Level of preparedness & frequency of using literacy strategies School % of classes taught by HQT Principal climate score % suspensions % perceiving they are college bound School safety score Staff stability

Outcome Variables: Reading Scores Random EffectVariance Component dfChi- square P ValueVariance Decomposition (% by level) State Test Students Teachers Schools ITBS Students Teachers Schools

Student-Level Variation  Across multiple model specifications, the only predictors with statistical significance were the student’s  Pre-test score  Gender  ELL status  Modeling teacher-level factors produced no significant results 12

Classroom Observation Results  No baseline differences in levels of engagement & cognitive demand, or in instructional strategies  Cognitive demand level of lessons was low in Year 2, irrespective of research condition  Intervention teachers tended to use more literacy strategies than comparison teachers in Year 2  38.5% of intervention teachers used multiple literacy strategies vs 18.2% of comparison teachers 13

Why We May Not Find Impact  Low c ognitive demand of lessons  Counterfactual situations may “water down” the treatment’s effect  Low implementation fidelity  Limitations in outcomes measures (just say measurement error) 14

Implications for Further Research  Better understanding of  the relationship between a school-level intervention and its potential to affect student achievement  Correlates of student achievement  Why an intervention that did not show impact may nevertheless be of value 15