1. Professional Development Activity Log: Comparing Teacher Log and Survey Approaches to Evaluating Professional Development AERA Annual Meeting Montreal, April 11, 2004 Symposium - Evaluating the Quality of Professional Development: Implications for Districts and States Kwang Suk Yoon Reuben Jacobson American Institutes for Research

2. Overview Comparisons between teacher logs with survey Teachers’ experience with logs Lessons learned Next steps

3. Professional Development Activity Log (PDAL) The PDAL is a web-based, self-administered, longitudinal data collection tool for teachers to record their professional development experiences in detail with the assistance of a series of structured prompts Teachers log on to their password-protected web account and fill out their PDAL at regular intervals Visit www.PDAL.net for more informationwww.PDAL.net

4. PDAL Entries Name of activity Number of hours spent on each activity and its duration Whether the activity is a one-time or continuous event (e.g., recurring over a number of months) Type of activity (e.g., workshop, summer institute, study group) Purpose of activity (e.g., strengthening subject matter knowledge) PD quality features (e.g., active learning, coherence, collective participation) Content focus (e.g., algebraic concepts: absolute values, use of variables, etc.) Instructional practice – instructional topics covered in each activity (e.g., use of calculators, computers, or other educational technology)

5. Why PDAL? Collects disaggregate information about specific PD activities – Increases the level of specificity of PD data and reduces bias introduced by gross data aggregation Gathers accurate, reliable, and time-sensitive information – Minimizes recall problem with retrospective reports Tailors technical assistance to teachers based on their response patterns Allows teachers to review their own logs – Teachers can reflect on their own PD experiences Generates context sensitive questions

6. “Black Box” of Survey Data Hypothetical Data

7. Fine-grain log-level data on contact hours : Disaggregated by teacher by activity by time Hypothetical Data

8. Validation of Teacher Logs as an Alternative Data Collection Method Comparing PDAL and exit survey results –Measurement properties –Correlations –Mean levels Assessing the relative efficacy of teacher logs and survey

9. Method Instruments –PDAL Conduced over 15 months –Exit Survey Follow-up questions about PD activities over the same period of 15 months Questions about teachers’ experiences with PDAL Sample –4 Math-Science Partnership Program projects –Sample 476 math and science teachers mostly in middle or high schools –Participants 326 teachers completed at least for a month in PDAL 165 teachers participated in the Exit Survey

10. Measurement Properties of PDAL

11. Agreement between Log & Survey Methods: Correlations between methods

12. Agreement between Log & Survey Methods: Mean Levels

13. Teachers’ Experience with or Opinions about PDAL: Results from the PDAL Exit Survey

14. Unique Benefits of PDAL Rich, in-depth data with a high level of specificity –Differences in features between PD sponsored by MSP vs. other PD –Topic intensity (i.e., amount of contact hours per topic) Time-dependent measures –Percent of months with PD –Average contact hours by month –Average span of activity Can be used for on-going formative evaluation to continuously improve PD –Episode-specific comments and feedback

15. Average Contact Hours by Month

16. Lessons Learned & Implications Complementary uses of logs & surveys for different purposes –Globally estimating the mean level of PD activities –Investigating the variability of specific PD elements and relating it to other outcomes –Improving PD design Cost and benefits of different data collection methods Improving survey method: Increase the level of specificity

17. Next Steps Final phase of data analysis –Using both PDAL and survey data to assess change in teaching practice and assess their relative predictive validity PDAL users focus group (May 2005) PDAL usability study Need for follow-up studies –Such as a new CCSSO-AIR study on improving evaluation of professional development in math and science at state and local levels

18. Contact Information Kwang Suk Yoon (202) 403-5358 ksyoon@air.org Reuben Jacobson (202) 403-6925 rjacobson@air.org Visit us www.PDAL.net www.air.org