User vs Nonuser : A Multi-State, Multi-District Study of the Impact of Participation in PD 360 on Student Performance Prepared by Steven H. Shaha, PhD, DBA March

Overarching Research Question: Does teacher engagement in PD 360 and Observation 360, tools within the Educator Effectiveness System, significantly affect student success? Does teacher engagement in PD 360 and Observation 360, tools within the Educator Effectiveness System, significantly affect student success? 2

Methods Design: Quasi-experimental, retrospective, pre-post, normalized treatment-control / participation vs. non-participation ( , ) Goal: Multi-State, large n with comparable student populations (matched, controlled) Student Change: * Metric was percent students classified as Proficient or Advanced in respective States. 3

Sample Participation – Systematic sample of 176 schools, in 59 districts, in 25 States N determined by a priori Power analysis – Schools eligible for inclusion in the sample as participating Schools met the following criteria: More than 10 teachers total 80% or more of teachers viewed materials Minimum average of 90.0 minutes of viewing per teacher for the school – Districts included were only those for which eligible schools were included Normalizing for difference in socio-economic and demographic factors between participating Schools and their Districts cumulatively as the statistical comparison group Data – Participation data were extracted from the Internet-based professional development application as surveilled – Student performance data were captured from publically available, Internet-accessed sources (school as unit of measure, percent Proficient or Advanced as metric) 4

Demographics of the Sample 5

PD 360 Impact Assessment Interpretation: Schools that leverage PD 360 experience: – Significant and meaningful gains in student performance for Math and Reading – Bring their district performance averages up The biggest predictor of gains in student performance are: – Minutes per User – Teacher time using PD 360 – Programs viewed and re-viewed – Teacher study within PD * Statistical significance establishes genuine differences between groups and verifies that impacts were “real” and not merely due to chance and, in this case, due to any pre- existing biases in group differences. The appropriate p-values are included with all differences explained herein.

Math 7

Gains in Proficiency Rates 8 NOTE: Statistical significance is affected by sample size, wherein significance is achieved with smaller increases given larger samples or numbers of students. Consistent pattern: Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts. Consistent pattern: Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts.

Gains in Advanced Rates 9 NOTE: Statistical significance is affected by sample size, wherein significance is achieved with smaller increases given larger samples or numbers of students. Consistent pattern: Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts. Consistent pattern: Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts.

10 NOTE: Factors or variables listed are those isolated through Regression Analyses. These are those statistically most predictive or correlated with the increases identified.. Factors/Variables Most Affecting Increases in Math Achievement Rates

11 NOTE: Factors or variables listed are those isolated through Regression Analyses. These are those statistically most predictive or correlated with the increases identified.. Impact is predicted primarily by Teacher use and depth of personal investment

Greatest Predictors of Improvement in Math Schools: Average Minutes per User Numbers of Pro Viewed Districts: Percent of Users Registered Percent of Users with Community Forums Posted 12

Reading 13 Note to Reader: To better dramatize the magnitude of the consistently favorable impact of PD 360, graphics included hereafter represent a variety of perspectives and a sampling of different interpretive insights, and not an exhaustive nor uniformly arrayed set of results.

Gains in Proficiency Rates 14 NOTE: Statistical significance is affected by sample size, wherein significance is achieved with smaller increases given larger samples or numbers of students. Consistent pattern: Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts. Consistent pattern: Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts.

Gains in Advanced Rates 15 NOTE: Statistical significance is affected by sample size, wherein significance is achieved with smaller increases given larger samples or numbers of students. Consistent pattern: Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts. Consistent pattern: Despite schools beginning at a disadvantage, by year 2 they not only close the gap, but surpass their districts.

Factors/Variables Most Affecting Increases in Reading Achievement Rates 16 NOTE: Factors or variables listed are those isolated through Regression Analyses. These are those statistically most predictive or correlated with the increases identified..

17 NOTE: Factors or variables listed are those isolated through Regression Analyses. These are those statistically most predictive or correlated with the increases identified.. Factors/Variables Most Affecting Increases in Reading Achievement Rates AND by Teacher use and depth of personal investment Impact is predicted by Teachers engaged and depth of engagement

Greatest Predictors of Improvement in Reading Schools: Percent of Users Registered Links Uploaded Link Viewed Districts: Average Minutes per User Percent of Users Registered Links Viewed Pro Viewed 18

PD 360 Impact Assessment Executive Summary Statistically significant* advantages were verified favoring schools with PD 360 versus District Benchmarks. Math (p<.001) Reading (p<.001) Average Gains: Schools: – Math Proficient Rate: 17.1% (p<.001) – Math Advanced Rate: 21.4% (p<.001) – Reading Proficiency Rate: 19.5% (p<.001) – Reading Advanced Rate: 19.0% (p<.001) Districts as Ripple Effect: – Math Proficient Rate: 3.1% (p<.001) – Math Advanced Rate: 2.3% (p<.001) – Reading Proficiency Rate: 6.7% (p<.001) – Reading Advanced Rate: -3.8% (unknown how much this was boosted by schools) 19 * Statistical significance establishes genuine differences between groups and verifies that impacts were “real” and not merely due to chance and, in this case, due to any pre- existing biases in group differences. The appropriate p-values are included with all differences explained herein.

Greatest Predictors of Improvement The most significant predictors of success were identified, which maximizes the generalizability of successes documented Average Minutes per User Percent of Users Registered Links and Pro Viewed Percent of Users with Community 20