1. Empirical Study of the Effects of Professional Development on Improving Mathematics and Science Instruction MSP Goal 3 NSF/RETA Study Website: http://www.ccsso.org/msppdstudy PI, Rolf K. Blank, rolfb@ccsso.org Council of Chief State School Officers Co-PI, Kwang Suk Yoon, Bea Birman, Mike Garet ksyoon@air.org ; American Institutes for Research Co-PI, John Smithson, johns@education.wisc.edu Wisconsin Center for Education Research http://www.ccsso.org/msppdstudyrolfb@ccsso.org ksyoon@air.orgjohns@education.wisc.edu

2. Demonstration Project to show: how multiple PD programs can be evaluated using a common set of research-based measures of quality and: how a survey methodology can be used to evaluate the effects of PD in improving math & science instruction

3. An evaluation study to determine: 1) whether PD activities supported by MSP programs are consistent with research-based definitions of quality PD 2) The effects of PD on mathematics/ science instructional practices and content 3) How MSP programs use study findings to improve PD effectiveness

4. Quality of PD Content Focus Collective Participation; Active Learning; Coherence; Sustained Effort Pre-Instruction: Alignment of Instruction with Content Standards; Instructional Practice Teacher Characteristics: Background Variables, Prior PD Experiences Target Class Students: Diversity School Culture: Trust Post-Instruction: Alignment of Instruction with Content Standards; Instructional Practice Year 1Year 2Year 3Year 0 PDAL SEC1SEC2 A Logic Model for Assessing the Effect of Professional Development on Instructional Practice and Content Alignment

5. Study Sample MSP Sites (N= 5: Cleveland, New Jersey, Brockport-SUNY, El Paso, Corpus Christi, TX) –Voluntary participation, Data feedback to sites School Districts (N= 23) – Urban/Rural, Size varies Schools (N= 79) –Treatment vs. Control (based on MSP PD cohort) Teachers (N= 570) –Middle Grades Math and Science Target Class – Data analysis one class/teacher

6. Instructional Content Topics by Cognitive Demand Instructional Practices & Teacher Characteristics Multi- purpose indicators of curriculum practice and professional development opportunities.

7. Instructional Content Topics by Cognitive Demand Instructional Practices & Teacher Characteristics

8. School & Class DescriptionUse of HomeworkClassroom PracticesStudent ActivitiesUse & Availability of Ed. Tech. Assessment UseInstructional Influences Instructional Preparation Teacher OpinionsProfessional Development Teacher CharacteristicsInstructional Content

9. School & Class DescriptionUse of HomeworkStudent ActivitiesUse & Availability of Ed. Tech.Assessment UseInstructional InfluencesInstructional PreparationTeacher OpinionsProfessional DevelopmentTeacher CharacteristicsInstructional Content Classroom Practices

10. School & Class DescriptionUse of HomeworkClassroom PracticesStudent ActivitiesUse & Availability of Ed. Tech.Assessment UseInstructional InfluencesInstructional PreparationTeacher OpinionsProfessional DevelopmentTeacher CharacteristicsInstructional Content

11. School & Class DescriptionUse of HomeworkClassroom PracticesStudent ActivitiesUse & Availability of Ed. Tech.Assessment UseInstructional InfluencesInstructional PreparationTeacher OpinionsProfessional DevelopmentTeacher CharacteristicsInstructional Content

12. School & Class DescriptionUse of HomeworkClassroom PracticesStudent ActivitiesUse & Availability of Ed. Tech.Assessment UseInstructional InfluencesInstructional PreparationTeacher OpinionsProfessional DevelopmentTeacher CharacteristicsInstructional Content

13. Classroom Practices Student ActivitiesProfessional DevelopmentInstructional Content

14. PDAL Classroom Practices Student Activities Professional Development Instructional Content Classroom Practices Student Activities Professional Development Instructional Content

15. PDAL Instructional Practices Student Activities Professional Development Instructional Content Summer 2003 Summer 2005 Instructional Practices Student Activities Professional Development Instructional Content

16. Year 1 Data Collection Timeline April 2003 May 2003 June 2003 P D Activity Logs Survey orientation Teacher Incentives Admin. Training Survey Administration Site Coordinators Group Admin.

17. Teacher Incentives Review Reflection Planning On-line access to survey results Secure access to individual results Compare instruction across grades Compare instruction & standards Honoraria

18. Survey Administration Group Administration 60 to 90 minutes Brief teachers Survey Return Procedures Back cover information Voluntary (return blank) Confidential (seal envelope) Target Class / Reporting period IP instructions / Response metric Content instructions Feedback forms

19. Survey Back – Cover Respondent Information

20. Instructions for Selecting the Target Class – Mathematics Instruction -- For all questions about classroom practices please refer only to activities related to mathematics instruction. If you teach more than one mathematics class, select the first class that you teach each week. If you teach a split class (i.e. the class is split into more than one group for mathematics instruction) select only one group to describe as the target class.

27. PDAL Data Analysis Plan December 11, 200 Kwang Suk Yoon American Institutes for Research Professional Development Activity Log: A New Approach to Design, Measurement, Data Collection, and Analysis* * To be presented at AERA 2004 Annual Meeting

28. Whether professional development (PD) activities supported by MSP programs are consistent with research-based qualities of PD The effects of PD on mathematics/science content alignment and instructional practice The purpose of the study is to determine: Assessing the Effects of Professional Development on Improving Mathematics and Science Instruction:

29. Limitations of Existing Instruments for Collecting PD Data Rely on teachers’ retrospective reports on cumulative professional development experiences that have occurred over an extended time period Teachers may have to aggregate across a number of distinct activities Focus on a particular set of PD activities supported by certain programs (e.g., Eisenhower) and may exclude many others Solicit from teachers their opinions about their PD activities rather than their actual behavioral engagement with the activities

30. Features of Professional Development Activities Log (PDAL) Help teachers create an ongoing monthly log of any professional learning activity in which they participate Web-based, self-administered survey Longitudinal design: Data collected over 15 months Aligned with SEC items (e.g., content focus) Inclusive approach to PD –Includes MSP-sponsored and non-MSP-sponsored activities –Documents one-time and recurring activities –Captures formal and informal professional development activities

31. PDAL Entries Name of activity Number of hours spent on each activity and its duration Whether the activity is a one-time event or a continuous one (i.e., recurring over a number of months) Type of activity (e.g., workshop, summer institute, study group) Purpose of activity (e.g., strengthening subject matter knowledge) Content focus (e.g., algebraic concepts: absolute values, use of variables, etc.) Instructional practice – instructional topics covered in each activity (e.g., use of calculators, computers, or other educational technology) PD quality features (e.g., active learning, coherence, collective participation) Materials used during each activity

32. Advantages of PDAL Collects disaggregate information about specific PD activities – Reduces bias introduced by gross data aggregation Gathers accurate, time-sensitive information; Minimizes recall problem with retrospective reports Allows teachers to review their own logs – Teachers can reflect on their own PD experiences Generates context sensitive questions Alleviates teachers’ response burden Minimizes data entry errors Be able to tailor technical assistance to teachers based on their response patterns

33. Obstacles to PDAL Requires users’ technical knowledge & skills –In the use of computer and web –In the specific features of the PDAL –Solution: Provide Technical Assistance (TA) as needed Requires Web access –Solution: No-web-access is considered as out of scope The length of data collection period and teachers’ willingness –Sample attrition –Frequency of reporting –Missing data –Solution: Solicit teachers’ cooperation through reminder calls and postcards and local MSP project leader’s initiatives

34. Sample and Multi-level Contexts MSP Sites (N=4-5) –Treatment vs. Control, Nature of partnership School Districts (N=16-23) –Content Standards, Urban/Rural, Size Schools (N=67-79) –SES, School Culture – Trust Teachers (N=450-570) –Subject, Gender, Teaching Experience, Credential Target Class –Content Alignment; Content Match; Student Diversity (Minority, Achievement Levels)

35. Hypothetical PDAL Data: Configurations and Aggregation Disaggregate log entry data –Monthly log (as a basic unit of observation) –Teacher –Activity –Time Data aggregation –Time –Activities –Teachers –Sites

36. Descriptive Analysis Descriptions & correlates of PD activities –patterns of responses (logs) Timing of log entries (peak and low); average time per log –Patterns of PD activities Average number of PD activities per teacher; seasonal variation in PD activities (overall participation rate, rate by type of activity); mean quality by site or PD type –Correlates Which types of PD activities are associated with high quality –Classes of teachers Some latent classes of teachers (e.g., minimalists, enthusiasts, informal learners)

37. Impact Analysis Main Outcome Variables of PDAL Analysis –Quantity and Quality of PD Main Outcome Variables of SEC Analysis –Change or Stability in Content Alignment –Change or Stability in Instructional Practice Linking PD Variables to Instructional Outcomes

38. Impact Analysis (Cont’d) Main Effects of: –MSP Site –Treatment –PD Quality –School Culture (e.g., Trust) –Subject Interaction Effects of: –Treatment by MSP Site –Treatment by PD Quality –Treatment by School Culture –Treatment by Subject –Treatment by PD Quality by School Culture

39. Example of Interaction Effect: The Effect of Treatment on the Use of Instructional Practice Moderated by School Culture (Trust)

40. Effects of Professional Development on Change in Teacher’s Instructional Practice: By PD Activity’s Focus on Specific Instructional Strategy, PD Quality, and School Culture