1 Studying Teacher Professional Development: Challenges and Possible Solutions Nicole Kersting Research Scientist LessonLab Research Institute, Santa Monica, CA

2 Overview of Presentation PD Program under study Challenges to implementation & possible solutions Challenges to evaluation & possible solutions

3 Study Background and Goals Builds on key findings of TIMSS 1999 Video Study. Aims at helping 6 th -grade mathematics teachers to maintain the cognitive level of rich mathematics problems. Key topic areas: fractions, ratio/proportion, and variables/expressions

4 PD Program Can we get teachers to effectively engage students in thinking about math concepts? Three video-based modules, each focusing on: Mathematics content knowledge Analysis of videotaped lessons (strategies for maintaining problems’ richness and student lens) Teaching of videotaped lesson Sharing meetings Total of 45 face-to-face hours –7 pull-out days (42 hours) distributed during the year –3 sharing meetings (3 hours)

5 Research Questions Does the PD program impact teacher knowledge? Does the PD program impact teaching practices? Does the PD program impact student learning?

6 Sample Predominantly low-income district in Los Angeles 5 Title-1 middle schools All 6 th -grade teachers (=71); PD program made mandatory by the district All 6 th -grade students (approx 4400 in 137 classes) Student population: 73.6% Latino and 25.9% African American; 40% are English learners 6.2% of students are mathematics proficient (as measured by state tests in 2004)

7 Design Year 1 Random assignment within schools and tracks: Treatment and no- treatment group Year 2 All teachers participated in PD program (1 year vs. 2 years of PD)

8 Key Measures - Teachers Content and pedagogical content knowledge – Multiple-choice survey: Items from Study of Instructional Improvement (Ball & Hill, University of Michigan); California Professional Development Institutes (MPDI - UC office of the president); and developed in-house – Administered at 3 points in time; 2 equivalent forms Video Analysis Assessment Administered prior and at the end of Year 2 Compliance Ratings

9 Key Measures – Teaching Practices Videotapes of lessons including a rich problem Fieldnotes from classroom observations

10 Key Measures - Students State standardized tests (most distal) Subset of district quarterly assessment items targeting key content areas In-house student performance assessment items (most proximal)

11 Key Features of PD Program District mandated teacher participation Multi-site randomized field trial Title 1 schools Multi-track schools and calendars

12 Challenges to Faithful Program Implementation Possible Solutions Full participation of all teachers Opposing instructional goals  Involve every level at the school and district Complications of multiple track calendars  Constant contact with the person at the school “Try to get on their calendar and try to stay on it”.

13 Challenges to Faithful Program Implementation Possible Solutions Role of math coaches  Don’t make assumptions about district and school personnel position responsibilities. Changes in school and district personnel  Don’t presume personnel stability.

14 Challenges to Experimental Design Possible Solutions Group contamination: students are moved across classrooms  obtain class rosters to assess percent of students moved. Teacher attrition (especially for programs with more than one year implementation).  be conservative to ensure sufficiently large sample size for inferential statistics

15 Challenges to Experimental Design Possible Solutions Compliance: 6/28 teachers in Year 1 and 5/38 teachers in Year 2 were rated as fully compliant  Create compliance scale to be able to evaluate differential program effects for compliant and non-compliant teachers.

16 Challenges to Experimental Design Possible Solutions Threats to the validity of measures –Low motivation (PCK survey is time consuming) –Resistance to being assessed (exposes teachers’ lack of knowledge)  embedded measures of teacher knowledge Lack of control over collecting student measures  Use own resources to collect data

17 Conclusions Conducting experimental research in schools (particularly in disadvantaged schools) is time consuming and difficult Partnerships between researchers and schools are extremely valuable Disadvantaged schools deserve our attention