School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration Independent Evaluator December 2012

Overarching Research Question: Does engagement in PD 360 and Observation 360, tools within the Educator Effectiveness System, significantly affect student success and school-wide metrics? Does engagement in PD 360 and Observation 360, tools within the Educator Effectiveness System, significantly affect student success and school-wide metrics?

Sample Description High Video Utilizers – 39 States – 211 Districts – 734 Schools Metrics: Educator Engagement Student Success School Impacts

32 data elements collected or computed through PD 360 and Observation 360 Contrasted higher engagement schools versus lower engagement schools – Improvement in percentages of students who tested advanced or proficient in math and reading – Classified into four quartiles – Analyses of highest and lowest quartiles only Study of Educator Engagement

Metrics for Differentiating Advantages for Higher Engagement Organizations: Focus Objectives Set Up Observations Performed Percent Registered Users Percent of Users in Communities Minutes Viewed Forums Viewed Programs Viewed Segments Viewed Links Viewed Follow-up Questions Answered Reflection Questions Answered Focus Objectives Set Up Forums Posted Downloaded Files Uploaded Files Participation in Communities Leadership, Implementation and Accountability Educator Participation Educator Engagement These are the 15 metrics for which higher engagement schools were significantly higher than their lower engagement counterparts

Sample of Differentiating Metrics of Utilization and Engagement Links Viewed Minutes Viewed Follow-up Questions Answered Observations Performed Passive participation (e.g. video viewing alone) is LESS influential than Active engagement 63.8% advantage (p<.001) 39.0% advantage (p<.001) 70.3% advantage (p<.001) 4.3% advantage (p<.001)

Passive participation (e.g. video viewing alone) is LESS influential than Active engagement Uploaded Files Downloaded Files Forums Viewed Forums Posted Sample of Differentiating Metrics of Utilization versus Engagement 47.3% advantage (p<.001) 30.5% advantage (p<.001) 79.5% advantage (p<.001) 68.6% advantage (p<.001)

Who Cares? Who cares if educators used it more? Did it make a difference for kids and schools?

Student Success: Performance on standardized tests – Percent either proficient or advanced in the following subjects: Reading Math Study of Student Success

Improved Student Performance 4.9% gain for lower engagement schools (p<.01)

Improved Student Performance 4.9% gain for lower engagement schools (p<.01) 18.0% gain for higher engagement schools (p<.001) Closed the Gap: 267% advantage in gains for higher engagement schools (p<.001) Nearly 4 times the impact Closed the Gap: 267% advantage in gains for higher engagement schools (p<.001) Nearly 4 times the impact

Improved Student Performance 0.5% gain for lower engagement schools (p=ns) Actually Important Gains: For every 200 students, 1 more performed at proficient or advanced level than in the previous year Actually Important Gains: For every 200 students, 1 more performed at proficient or advanced level than in the previous year

Improved Student Performance 0.5% gain for lower engagement schools (p=ns) 18.9% gain for higher engagement schools (p<.001) Surpassed the Gap: 3,520% advantage in gains for higher engagement schools (p<.001) 36 times greater impact Surpassed the Gap: 3,520% advantage in gains for higher engagement schools (p<.001) 36 times greater impact

Performance on key indicators from Internet (when publicly available) and structured telephone interviews: – Dropout Rates – Student Discipline Rates – Teacher Retention Rates – College-Bound Rates Metrics of School Impact

Improved Dropout Rates 4.9% improvement for lower engagement schools (p<.01) For every 100 students, 5 fewer dropped out than in the previous year Figures reflect rounding, projections reflect correct math.

Improved Dropout Rates 20.0% improvement for higher engagement schools (p<.001) 309.1% advantage in gains for higher engagement schools (p<.001) Executive Summary Higher engagement schools began statistically equal, then significantly outperformed their counterparts (p<.01) Executive Summary Higher engagement schools began statistically equal, then significantly outperformed their counterparts (p<.01) 4.9% improvement for lower engagement schools (p<.01) For every 100 students, 20 fewer dropped out than in the previous year.

Improved Student Discipline Rates Figures reflect rounding, projections reflect correct math. Y-Axis is inverted to reflect improvement as intuitively upward trend. 7.4% fewer disciplinary incidents for lower engagement schools (p<.01)

33.2% fewer disciplinary incidents for higher engagement schools (p<.001) Improved Student Discipline Rates Executive Summary Higher engagement schools significantly outperformed their counterparts (p<.01) Executive Summary Higher engagement schools significantly outperformed their counterparts (p<.01) 7.4% fewer disciplinary incidents for lower engagement schools (p<.01) 351% advantage in gains for higher engagement schools (p<.001) 4 ½ times the impact 351% advantage in gains for higher engagement schools (p<.001) 4 ½ times the impact For every 100 students, 33 fewer problem students than in the previous year.

Figures reflect rounding, projections reflect correct math. Improved Teacher Retention Rates 1.7% more teachers stayed for lower engagement schools (p<.01)

Improved Teacher Retention Rates Executive Summary Higher engagement schools significantly outperformed their counterparts (p<.01) Executive Summary Higher engagement schools significantly outperformed their counterparts (p<.01) 65.9% advantage in gains for higher engagement schools (p<.001) Nearly twice the impact 65.9% advantage in gains for higher engagement schools (p<.001) Nearly twice the impact 1.7% more teachers stayed for lower engagement schools (p<.01) 2.8% more teachers stayed for higher engagement schools (p<.01) For every 100 teachers, nearly 3 fewer left than in the previous year. Figures reflect rounding, projections reflect correct math.

Improved College-Bound Rates No decrease or gain for lower engagement schools (ns) Figures reflect rounding, projections reflect correct math. Percentage of students schools report as being college-bound.

Improved College-Bound Rates Executive Summary Higher engagement schools began statistically equal, then significantly outperformed their counterparts (p<.01) Executive Summary Higher engagement schools began statistically equal, then significantly outperformed their counterparts (p<.01) Incalculable advantage in gains for higher engagement schools (p<.001) 9.6% improvement for higher engagement schools (p<.001) No decrease or gain for lower engagement schools (ns) For every 100 students, 10 more were college-bound than in the previous year. Figures reflect rounding, projections reflect correct math.

Dropout Rates – Approx. 15 fewer dropouts per 100 students than lower engagement school counterparts Student Discipline Rates – Approx. 33 fewer students “in the office” per 100 students than for lower engagement school counterparts Teacher Retention Rates – Approx. 3 fewer teachers leaving per 100 teachers, which is 1 fewer than for lower engagement school counterparts College-Bound Rates – Approx. 10 more college-bound students per 100 students than for lower engagement school counterparts Summary of School Impacts

Student Success School Impacts Leadership, Implementation and Accountability Educator Participation Educator Engagement A Model for Educational Success