1. Program Evaluation and Impact Assessment: Internet-Delivered, On-Demand Professional Development Participating Schools versus Their Respective Districts Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration Independent Evaluator July 2013 1 NOTE: Findings that follow are published as: Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013 (in press)

2. Overarching Research Question: Does teacher engagement in PD 360 and Observation 360, tools within the Educator Effectiveness System, significantly affect student success? Does teacher engagement in PD 360 and Observation 360, tools within the Educator Effectiveness System, significantly affect student success? 2

3. Methods Design: Quasi-experimental, retrospective, pre-post, normalized treatment-control / participation vs. non-participation (2009-10, 2010-11) Goal: Multi-State, large n with comparable student populations (matched, controlled) Student Change: * Metric was percent students classified as Proficient or Advanced in respective States. 3 Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

4. Sample Participation – Systematic sample of 169 elementary schools, in 73 districts, in 19 States N determined by a priori Power analysis – Schools eligible for inclusion in the sample as participating Schools met the following criteria: More than 10 teachers total 80% or more of teachers viewed materials Minimum average of 90.0 minutes of viewing per teacher for the school – Districts included were only those for which eligible schools were included Normalizing for difference in socio-economic and demographic factors between participating Schools and their Districts cumulatively as the statistical comparison group Data – Participation data were extracted from the Internet-based professional development application as surveilled – Student performance data were captured from publically available, Internet-accessed sources (school as unit of measure, percent Proficient or Advanced as metric) 4 Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

5. Impacts on Math 5 Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

6. Impacts on Math 6 Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

7. Impacts on Math 7 Districts improved by 2.6 net percentage points (p<.01). Improvement is percent change: [(Year2-Year1)/Year1] Comparative change: [School change/District change] 7 Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

8. Impacts on Math 8 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

9. Impacts on Math 9 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

10. Impacts on Math 10 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

11. Impacts on Math 11 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 8.4 points more than Districts (p<.001). Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

12. Impacts on Math 12 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines. Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

13. Impacts on Math 13 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools experienced 4.2 TIMES greater improvement or Effect Size (p<.001). Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines. Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

14. Impacts on Math 14 Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Participating Schools experienced 4.2 TIMES greater improvement or Effect Size (p<.001). That’s a 4.5 Effect Size versus baselines (p<.001) Participating Schools experienced 4.2 TIMES greater improvement or Effect Size (p<.001). That’s a 4.5 Effect Size versus baselines (p<.001) Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines. Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

15. Impacts on Math Comparative Growth for Participating Schools In percentage of students Proficient or Advanced: 11.1 net improvement for Schools (p<.001) – 8.4 more than their respective Districts (p<.001) – 18.9% better than their Yr. 1 baseline (p<.001) – 14.7% better than Districts vs. baselines (p<.001) Effect Sizes: – 4.2 times greater improvement for net growth vs. Districts (p<.001) – 4.5 times greater improvement for growth from baselines vs. Districts (p<.001) 15 Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

16. Impacts on Reading Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

17. Impacts on Reading Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013. Reading: Pct. of Students Proficient OR Advanced Year 1Year 2 Net Change Statistical Significance Pct. Change from Yr 1 Statistical Significance SchoolsMean56.967.210.3 p<.001 6.518.1% p<.001 7.2 StDev22.422.1Effect Size DistrictsMean61.262.71.6 p<.01 2.5% p<.01 StDev21.120.7 Net Difference8.7 p<.001 15.6% p<.001 Level of Significance p=0.079p=0.063p<.001

18. Reading: Pct. of Students Proficient OR Advanced Year 1Year 2 Net Change Statistical Significance Pct. Change from Yr 1 Statistical Significance SchoolsMean56.967.210.3 p<.001 6.518.1% p<.001 7.2 StDev22.422.1Effect Size DistrictsMean61.262.71.6 p<.01 2.5% p<.01 StDev21.120.7 Net Difference8.7 p<.001 15.6% p<.001 Level of Significance p=0.079p=0.063p<.001 Districts improved by 1.6 net percentage points (p<.01). Impacts on Reading Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

19. Reading: Pct. of Students Proficient OR Advanced Year 1Year 2 Net Change Statistical Significance Pct. Change from Yr 1 Statistical Significance SchoolsMean56.967.210.3 p<.001 6.518.1% p<.001 7.2 StDev22.422.1Effect Size DistrictsMean61.262.71.6 p<.01 2.5% p<.01 StDev21.120.7 Net Difference8.7 p<.001 15.6% p<.001 Level of Significance p=0.079p=0.063p<.001 Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Impacts on Reading Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

20. Reading: Pct. of Students Proficient OR Advanced Year 1Year 2 Net Change Statistical Significance Pct. Change from Yr 1 Statistical Significance SchoolsMean56.967.210.3 p<.001 6.518.1% p<.001 7.2 StDev22.422.1Effect Size DistrictsMean61.262.71.6 p<.01 2.5% p<.01 StDev21.120.7 Net Difference8.7 p<.001 15.6% p<.001 Level of Significance p=0.079p=0.063p<.001 Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). Impacts on Reading Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

21. Reading: Pct. of Students Proficient OR Advanced Year 1Year 2 Net Change Statistical Significance Pct. Change from Yr 1 Statistical Significance SchoolsMean56.967.210.3 p<.001 6.518.1% p<.001 7.2 StDev22.422.1Effect Size DistrictsMean61.262.71.6 p<.01 2.5% p<.01 StDev21.120.7 Net Difference8.7 p<.001 15.6% p<.001 Level of Significance p=0.079p=0.063p<.001 Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Impacts on Reading Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

22. Reading: Pct. of Students Proficient OR Advanced Year 1Year 2 Net Change Statistical Significance Pct. Change from Yr 1 Statistical Significance SchoolsMean56.967.210.3 p<.001 6.518.1% p<.001 7.2 StDev22.422.1Effect Size DistrictsMean61.262.71.6 p<.01 2.5% p<.01 StDev21.120.7 Net Difference8.7 p<.001 15.6% p<.001 Level of Significance p=0.079p=0.063p<.001 Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Participating Schools improved by 8.7 points more than Districts (p<.001). Impacts on Reading Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

23. Reading: Pct. of Students Proficient OR Advanced Year 1Year 2 Net Change Statistical Significance Pct. Change from Yr 1 Statistical Significance SchoolsMean56.967.210.3 p<.001 6.518.1% p<.001 7.2 StDev22.422.1Effect Size DistrictsMean61.262.71.6 p<.01 2.5% p<.01 StDev21.120.7 Net Difference8.7 p<.001 15.6% p<.001 Level of Significance p=0.079p=0.063p<.001 Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 15.6% more versus baselines. Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 15.6% more versus baselines. Impacts on Reading Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

24. Reading: Pct. of Students Proficient OR Advanced Year 1Year 2 Net Change Statistical Significance Pct. Change from Yr 1 Statistical Significance SchoolsMean56.967.210.3 p<.001 6.518.1% p<.001 7.2 StDev22.422.1Effect Size DistrictsMean61.262.71.6 p<.01 2.5% p<.01 StDev21.120.7 Net Difference8.7 p<.001 15.6% p<.001 Level of Significance p=0.079p=0.063p<.001 24 Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Participating Schools experienced 6.5 TIMES greater improvement or Effect Size (p<.001). Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 15.6% more versus baselines. Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 15.6% more versus baselines. Impacts on Reading Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

25. Reading: Pct. of Students Proficient OR Advanced Year 1Year 2 Net Change Statistical Significance Pct. Change from Yr 1 Statistical Significance SchoolsMean56.967.210.3 p<.001 6.518.1% p<.001 7.2 StDev22.422.1Effect Size DistrictsMean61.262.71.6 p<.01 2.5% p<.01 StDev21.120.7 Net Difference8.7 p<.001 15.6% p<.001 Level of Significance p=0.079p=0.063p<.001 25 Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Participating Schools experienced 6.5 TIMES greater improvement or Effect Size (p<.001). That’s a 7.2 Effect Size versus baselines (p<.001) Participating Schools experienced 6.5 TIMES greater improvement or Effect Size (p<.001). That’s a 7.2 Effect Size versus baselines (p<.001) Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 15.6% more versus baselines. Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 15.6% more versus baselines. Impacts on Reading Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

26. Impacts on Reading Comparative Growth for Participating Schools In percentage of students Proficient or Advanced: 10.3 net improvement for Schools (p<.001) – 8.7 more than their respective Districts (p<.001) – 18.1% better than their Yr. 1 baseline (p<.001) – 15.6% better than Districts vs. baselines (p<.001) Effect Sizes: – 6.5 times greater improvement for net growth vs. Districts (p<.001) – 7.2 times greater improvement for growth from baselines vs. Districts (p<.001) 26 Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013. *The original study of 15.3% cited has an errata underway to become 18.1%.