1. June 25, 20161 Regional Educational Laboratory - Southwest Review of Evidence on the Effects of Teacher Professional Development on Student Achievement: Finding & Suggestions for Future Evaluation Designs Presented by: Kwang Suk Yoon (AIR) Teresa Duncan (AIR) Sylvia Lee (Nat’l Taiwan U) Kathy Shapley (Edvance Research, Inc.) American Evaluation Association Washington, DC November 8, 2007

2. June 25, 20162 Objective To conduct a systematic review of research-based evidence on the effects of teacher professional development (PD) on student achievement

3. June 25, 20163 Overview of Methodology Systematic review  Using explicit & transparent methods  Following a set of standards  Being accountable, replicable, and updatable Review protocol  Aligned with What Works Clearinghouse (WWC) standards  Study selection criteria  Rigorous evidence standards  Multi-coder, multi-stage review process Screening, coding & reconciliation Evidence Review Tool (ERT)

4. June 25, 20164 Overview of the coding process

5. June 25, 20165 Electronic searches by keywords Literature search

6. June 25, 20166 Study selection criteria Relevance of study by:  Topic  Population  Subject  Study design Randomized controlled trial (RCT) Quasi-experimental design (QED) with matched comparison group  Student achievement outcome  Measures and their psychometric properties  Time  Country

7. June 25, 20167 Prescreening stage  N = 1,343 “potentially relevant” studies Reasons for failing selection criteria (1)

8. June 25, 20168 Stage-1 coding  N = 132 relevant studies Reasons for failing selection criteria (2)

9. June 25, 20169 Stage-2 coding  N = 27 relevant studies eligible for quality ratings  18 failed to meet WWC evidence standards RCT – randomization, attrition, disruption, etc. QED – baseline equivalence, attrition, disruption, etc. Inter-rater reliability  4 met evidence standards without reservations  5 met evidence standards with reservations Reasons for failing evidence standards

10. June 25, 201610 Stage-3 coding  Effect size  Characteristics of PD Form, duration, contact hours Content Provider Participants (volunteers?) Information about the implementation of PD Replicability issue Documentation of studies

11. June 25, 201611 Results (1) Paucity of rigorous studies  Only 9 studies met evidence standards Distribution of 9 studies  By study design 5 RCT 4 QED  By content area Concentrated in reading  By grade level All focused on elementary school level

12. June 25, 201612 Results (2) Overall effect size  Average of 20 effect sizes drawn from 9 studies =.54 Of 20 effect sizes, 12 were not statistically significant. Nine of those 12, however, are substantively important according to WWC conventions. Effects by subject area  Fairly consistent across three subject areas Effects by form, duration, and intensity of PD  Some evidence of effect of intensive PD Effects by content of PD  No consistent pattern  Failed to replicate Kennedy’s (1998) finding

13. June 25, 201613 Suggestions (1) Matching units of assignment and analysis Increasing statistical power to detect effects Considering potential confounding effects of PD and other important instructional factors (e.g., curriculum)

14. June 25, 201614 Suggestions (2) Adequate documentation of PD & study  PD implementation  Sample and cluster, if any  group assignment  baseline equivalence  effect size (ES) calculating and reporting effect sizes (weighting ES, adjusting ES for multiple comparisons, and correcting ES for clusters) Use of structured abstract to facilitate research synthesis structured abstract (Mosteller et al, 2004) claim-based structured abstract (Kelly & Yin, 2007)

15. June 25, 201615 Contact info & links Kwang Suk Yoon  kyoon@air.org (202-403-5358) Link to our RELSW report on the IES website  http://ies.ed.gov/ncee/edlabs/projects/project.asp? id=70 Link to our presentation material  http://www.pdal.net/reports.asp Thank you!