1. Effects of the My Teaching Partner Intervention in Secondary School Classrooms Joseph P. Allen Robert C. Pianta University of Virginia Co-Collaborators: Amori Mikami Anne Gregory Project Team: Chris Hafen Sharon Deal Judith Wasserman Rachel Boren Janetta Lun

2. Context Number of Secondary School Students in U.S.: 24 million Number of Secondary School Classes being taught each week 6 million % of 9 th graders who won’t finish High school by the end of 12 th grade 25% Number of programs in ‘What Works’ Clearinghouse with demonstrated efficacy improving teaching quality enough to improve student achievement in these classrooms 0

3. Key Questions Can we identify teacher-student interaction qualities that predict student engagement and achievement? Can we change these qualities? Will changes lead to sustainable student achievement gains? What are the mechanisms of change?

4. Key Questions Can we identify teacher-student interaction qualities that predict student engagement and achievement? Can we change these qualities? Will changes lead to sustainable student achievement gains? What are the mechanisms of change?

5. Classroom Learning Assessment & Scoring System- Secondary (CLASS-S) Emotional Support Positive Climate Teacher Sensitivity Regard for Adolescent Perspectives Negative Climate Instructional Support Instructional Learning Format Content Understanding Analysis & Problem Solving Quality of Feedback Classroom Organization Behavior Management Productivity Student Outcomes Student Engagement

6. Evaluation Design 43 teachers within 8 schools (640 students) (The control condition in an RCT). 1 focal classroom selected per teacher Predicting Future Achievement after Covarying Baseline Achievement Test Scores Teacher Demographics: –64% female –83% White, 8% African-American; 6% Mixed-Ethnicity; 3% Other –54 middle school, 34 high school –35% BA degree; 65% at least a year of course work beyond BA –Average 8 years of teaching experience

7. Classroom Characteristics School type: 39% High school; 61% Middle School Subject: 52% Language/Social Studies; 48% Math/Science Average class size: 23 students Gender: 47% girls 53% boys Ethnicity: 23% African American 2% Asian 4% Hispanic 70% European-American

8. Observational Assessment of Classroom Environment Videotaped observations of a classroom – spread throughout course of year Two 20-minute segments per class session/tape –Each tape rated by 2 raters Coded Using CLASS-S System High inter-rater reliabilities; ICC’s range from –.73 -.82 for overarching domains –.50 -.78 for specific dimensions (all but one dimension >.64)

9. Student Academic Success Score on State “Standards of Learning” End of Year Subject Test The measure by which schools/students are judged for accreditation/graduation. Extensive seven-year validation/standardization process.

10. Analytic Approach Multi-level modelling All models covary: –Student factors: Grade level Gender Family poverty status –Classroom factors: Classroom size –Teacher Factors Teacher experience Teacher education Teacher gender and race Moderating effects of covariates were also examined.

11. Predicting Student Achievement Achievement EmotionalPositive Climate.22** Support:Negative Climate -.04 Teacher Sensitivity.16* Regard for Adol. Perspectives.21** ClassroomBehavior Management.06 Organization:Productivity.15 InstructionalContent Understanding.12 Support:Analysis & Prob. Solving.18* Instructional Learning Formats.22** Quality of Feedback.09 Composite of Significant Dimensions Above.32***

12. Key Questions ✔ Can we identify teacher-student interaction qualities that predict student engagement and achievement? Can we change these qualities? Will changes lead to sustainable student achievement gains? Why?

13. MyTeachingPartner Overview Consultant and teacher work together using the CLASS-S in cultivating: –Observation –Reflection –Development of knowledge and expertise Classroom Observation Teaching Practice Knowledge Expertise Support

14. The Steps of the Consultancy

15. MTPS Website www.mtpsecondary.net

16. Detailed Video Examples www.mtpsecondary.net

17. Key Questions ✔ Can we identify teacher-student interaction qualities that predict student engagement and achievement? Can we change these qualities? Will changes lead to sustainable student achievement gains? What are the mechanisms of change?

18. Evaluation Design 88 classrooms 45 Tx. 43 Control (Classrooms Randomized within school) 2237 Students Assessed Across 2 Years

19. Evaluation Design Treatment group: Year 1: –Introductory Workshop (late summer) –Ongoing consultancy –~ 2 days total in-service time Year 2: –Booster Workshop (late summer) only + Web site access Control group: Usual in-service practice.

20. Intervention Effect on Change in Classroom Qualities Intervention Target JanuaryMarch April/ May Overall Teacher-Student Interaction Composite ns Student Engagementns

21. Intervention Effect on Change in Classroom Qualities Intervention Target JanuaryMarch April/ May Overall Teacher-Student Interaction Composite ns Student Engagementns

22. Intervention Effect on Change in Classroom Qualities Intervention Target JanuaryMarch April/ May Overall Teacher-Student Interaction Composite ns.19* Student Engagementns.34*

23. Year 1 Change in Overall Teacher-Student Interactions Standardized Effects: Baseline =.45*** Intervention =.19* MTPS participation predicts higher quality teacher-student interactions

24. Key Questions ✔ Can we identify teacher-student interaction qualities that predict student engagement and achievement? ✔ Can we change these qualities? Will changes lead to sustainable student achievement gains? What are the mechanisms of change?

25. Year 1 Intervention Effects on Achievement No relation of intervention to either baseline or exit achievement test scores in Year 1 (all p’s >.35). Why? –No evidence we changed the classroom until the very end of the year when most teaching was past.

26. Year 2 Change in Achievement Standardized Effects: Pre-test =.54*** Intervention =.22* MTPS is predicting increases in End of Course Achievement Tests

27. Year 2 Intervention Effects on Achievement Real-world effect size =.22 SD increment in Achievement Test scores Average ‘Bump’ of students in MTP from 50 th to 59 th percentile in achievement If effect applies equally at all parts of achievement spectrum (as appears to be the case): a.22 SD boost would reduce failure rates from: 14% without the intervention to 10% with it Reducing the number of failing students each year by 29% *** This occurs in the year AFTER the intervention year (i.e., sustainability), across diverse subject matter/content areas.

28. Key Questions ✔ Can we identify teacher-student interaction qualities that predict student engagement and achievement? ✔ Can we change these qualities? ✔ Will changes lead to sustainable student achievement gains? What are the mechanisms of change?

29. A Preliminary Mediational Analysis “My Teaching Partner” Intervention Intervention Observed Change in Student Achievement * Environmental Outcome

30. A Preliminary Mediational Analysis “My Teaching Partner” Intervention Intervention Target Observed Change in Student Achievement Environmental Outcome ??

31. Mediational Analyses Assessed via Multi-level Structural Equation Modelling, followed up via parametric bootstrapping analysis (Preacher et al., 2010) Focus on target of intervention (Teacher- student interactions assessed via CLASS-S) Using Centered/Standardized data for ease of interpretation.

32. “My Teaching Partner” Intervention Intervention Target Student Outcome.37**.16** Change in Student Achievement MTP-S Effect as Mediated via Observed Interactions Initial Model (Simple Direct Effects).12* Observed Teacher-Student Interactions

33. “My Teaching Partner” Intervention Intervention Target.06* * Student Outcome.37**.16** Change in Student Achievement MTP-S Effect as Mediated via Observed Interactions.06 ns Initial Model (Simple Direct Effects) Final Model (Including Mediated Effect).12* Observed Teacher-Student Interactions

34. Limitations Design only supports causal interpretations for outcomes, not for mediating processes with analyses thus far. Some Attrition Took Place (though it was unrelated to the intervention in every possible way we could test). Modest statistical significance with small sample

35. Conclusions We CAN identify elements of the classroom environment that predict student achievement. We CAN change these environmental factors. If we do, student achievement will change as well, eventually. Changes can be sustained over time and in new classrooms, post- intervention. We can identify potential mechanisms of change linked to the intervention. Which has implications for cost effectiveness…

36. *BOE = Back of Envelope Potential Significance – Costs vs. Benefits (BOE* Calculation) Resources per classroomEstimated Cost 20 Teacher hoursNo additional cost to system (in lieu of Regular In-service) 1 Teacher-consultant per 20 teachersMaximum of $3,500 per teacher including benefits (Potentially offset by ongoing teacher supervision personnel costs) Video equipment$200 per teacher TOTALS:Maximum of $3,700/23 children = $160/child (i.e., < 2% of annual per pupil expenditures) ** Benefits Average ‘Bump’ in achievement of ALL students from 50 th to 59 th percentile Reduction of 1 course failure per classroom (**Results may apply to multiple classrooms taught by a teacher)

37. MyTeachingPartner Secondary Replication is ongoing with the support of IES Further information available at: www.myteachingpartner.net