Program Evaluation and Impact Assessment: Internet-delivered, On-Demand Professional Development Participating Schools versus Their Respective Districts Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration Independent Evaluator July

Overarching Research Question: Does teacher engagement in PD 360 and Observation 360, tools within the Educator Effectiveness System, significantly affect student success? Does teacher engagement in PD 360 and Observation 360, tools within the Educator Effectiveness System, significantly affect student success? 2

Methods Design: Quasi-experimental, retrospective, pre-post, normalized treatment-control / participation vs. non-participation ( , ) Goal: Multi-State, large n with comparable student populations (matched, controlled) Student Change: * Metric was percent students classified as Proficient or Advanced in respective States. 3

Sample Participation – Systematic sample of 169 elementary schools, in 73 districts, in 19 States N determined by a priori Power analysis – Schools eligible for inclusion in the sample as participating Schools met the following criteria: More than 10 teachers total 80% or more of teachers viewed materials Minimum average of 90.0 minutes of viewing per teacher for the school – Districts included were only those for which eligible schools were included Normalizing for difference in socio-economic and demographic factors between participating Schools and their Districts cumulatively as the statistical comparison group Data – Participation data were extracted from the Internet-based professional development application as surveilled – Student performance data were captured from publically available, Internet-accessed sources (school as unit of measure, percent Proficient or Advanced as metric) 4

Impacts on Math 5

6

7 Districts improved by 2.6 net percentage points (p<.01). Improvement is percent change: [(Year2-Year1)/Year1] Comparative change: [School change/District change] 7

Impacts on Math 8 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline.

Impacts on Math 9 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001).

Impacts on Math 10 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline.

Impacts on Math 11 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 8.4 points more than Districts (p<.001).

Impacts on Math 12 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines. Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines.

Impacts on Math 13 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools experienced 4.2 TIMES greater improvement or Effect Size (p<.001). Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines. Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines.

Impacts on Math 14 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools experienced 4.2 TIMES greater improvement or Effect Size (p<.001). That’s a 4.5 Effect Size versus baselines (p<.001) Participating Schools experienced 4.2 TIMES greater improvement or Effect Size (p<.001). That’s a 4.5 Effect Size versus baselines (p<.001) Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines. Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines.

Impacts on Math Comparative Growth for Participating Schools In percentage of students Proficient or Advanced: 11.1 net improvement for Schools (p<.001) – 8.4 more than their respective Districts (p<.001) – 18.9% better than their Yr. 1 baseline (p<.001) – 14.7% better than Districts vs. baselines (p<.001) Effect Sizes: – 4.2 times greater improvement for net growth vs. Districts (p<.001) – 4.5 times greater improvement for growth from baselines vs. Districts (p<.001) 15

Impacts on Reading

Districts improved by 1.6 net percentage points (p<.01). Impacts on Reading

Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Impacts on Reading

Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). Impacts on Reading

Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 15.3% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 15.3% better than baseline. Impacts on Reading

Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 15.3% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 15.3% better than baseline. Participating Schools improved by 8.7 points more than Districts (p<.001). Impacts on Reading

Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 15.3% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 15.3% better than baseline. Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 12.8% more versus baselines. Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 12.8% more versus baselines. Impacts on Reading

24 Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 15.3% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 15.3% better than baseline. Participating Schools experienced 6.5 TIMES greater improvement or Effect Size (p<.001). Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 12.8% more versus baselines. Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 12.8% more versus baselines. Impacts on Reading

25 Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 15.3% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 15.3% better than baseline. Participating Schools experienced 6.5 TIMES greater improvement or Effect Size (p<.001). That’s a 6.1 Effect Size versus baselines (p<.001) Participating Schools experienced 6.5 TIMES greater improvement or Effect Size (p<.001). That’s a 6.1 Effect Size versus baselines (p<.001) Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 12.8% more versus baselines. Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 12.8% more versus baselines. Impacts on Reading

Comparative Growth for Participating Schools In percentage of students Proficient or Advanced: 10.3 net improvement for Schools (p<.001) – 8.7 more than their respective Districts (p<.001) – 15.3% better than their Yr. 1 baseline (p<.001) – 12.8% better than Districts vs. baselines (p<.001) Effect Sizes: – 6.5 times greater improvement for net growth vs. Districts (p<.001) – 6.1 times greater improvement for growth from baselines vs. Districts (p<.001) 26