1. Improving Teacher Quality Grants, Cycle 4: External Evaluation Report December 7 th, 2007 University of Missouri-Columbia Evaluation Team

2. Principal Investigators Sandra Abell Fran Arbaugh Mark Ehlert John Lannin Rose Marra Graduate Research Assistants Kristen Hutchins Ya-Wen Cheng Zeynep Kaymaz Michele Lee Dominike Merle Chia-Yu Wang

3. Context of the Evaluation Improving Teacher Quality Grant program, Cycle 4, 2006-2007 –Focus on high-need schools –14 funded professional development projects –Science and mathematics, grades 4-8

4. Participant Summary 383 participants; 84.0% white and 84.2% female; Most held regular certification in elementary/early childhood (n=236) and/or middle level mathematics (n=122) and/or science (n=203); 29% had been teaching for 1-5 years, 22.7% for 6-10 years, 34.5% for 11-15 years, and 9.5% for 26 or more years; 64.7% had participated in some form of PD the previous year; however, few participated in mathematics or science-specific, district-sponsored PD; 62.6% were new to the ITQG program; Taught in 124 different Missouri school districts, 3 private schools, and 3 charter schools; Directly impacted 33,580 students in the 2006-2007 school year.

5. Percentage of Participants from High-Need Districts

6. Were Cycle 4 ITQG Objectives Met? ObjectiveData Responding to Each Improve student achievement in math/science Students learned new content (MAP scores, student content tests) Increase teachers’ content knowledge and improve teachers’ pedagogical knowledge and skills Teachers learned new content and instructional strategies, and some began to use more inquiry- based instructional practices (evaluation instruments, teacher content knowledge tests) Impact on pre-service teachers Higher ed institutions improved their preparation of future teachers (evaluation instruments) Enhance use of assessmentTeacher knowledge/use of assessment was not improved (evaluation instruments, teacher content knowledge tests)

7. Four Effective PD Practices PD content PD delivery PD sustainability PD as part of a system

8. PD Content--Effective Practices What the research says: Effective PD provides opportunities for teachers to build their content and pedagogical content knowledge and examine practice. (Loucks-Horsley et al., 2003) What the standards say: Professional development for teachers of science requires integrating knowledge of science, learning, pedagogy, and students; it also requires applying that knowledge to science teaching. (NRC, 1996) What the Cycle 4 RFP said: Provide content knowledge related to standards and classroom instruction. Provide instructional strategies related to content being taught in the classroom. Provide research-based instructional strategies to the participating teachers.

9. PD Content—What We Found 1.Science and mathematics subject matter content --Relevant to teachers --Less is more 2.Integration of science and mathematics content --Need for explicit connections 3. Integration of subject matter and pedagogy -- PIs must make their expectations for teacher learning clear, make their pedagogical models explicit, and integrate theory/practice as well as content/pedagogy.

10. PD Content--Participant Preferences Teachers preferred when PD content: Related to their grade level and subject Aligned with local curriculum and GLEs Reflected teacher knowledge needs Related to student knowledge needs Included both science/mathematics content and pedagogical content Representative comments: “I liked the different project ideas that relate to the GLEs.” “Working with professors that could help us combine the GLEs with our lesson design was probably the most valuable part of the experience for me.” “The most valuable part was working in groups to create inquiry-based lessons that are aligned to the GLEs.”

11. PD Content—Relevant Learning 0=No gain in content knowledge 1=None of the gain was relevant to my teaching assignment 2=Very little was relevant 3=A Moderate amount was relevant 4=Most of the gain was relevant 5=All of the gain was relevant Frequencies Descriptive Statistics Subject area012345n/anMSD Mathematics711947593941723.61.2 Science33746758462184.01.0 End of Summer Institute

12. PD Content—Science/Math Subject Matter Amount Projects delivered PD to teachers across a range of grade levels. –largest range: 6 grade levels (grades 3-8) –smallest range: 2 grade levels (grades 6-7) Projects listed 2-8 subject matter foci Teachers were more satisfied when fewer content areas were covered

13. How to Design for Effectiveness in Science/Math Subject Matter UCM—Grade-level appropriateness Lincoln—Less is more

14. PD Content—Science/Math Integration 14 Cycle 4 projects –5 targeted science –2 targeted mathematics –7 seven targeted both science and mathematics

15. PD Content—Science/Math Integration What participants saw as “least valuable” The higher level math since I am lower level science. I am an elementary teacher and many of the lessons were designed for high school math. I did not find this helpful to me. This experience wasn't related to teaching math. I am a math teacher and this experience was hard to incorporate into my classroom. I am a math teacher exclusively, so the time spent on science was not well spent.

16. How to Design for Effective Science/Math Integration UMR—Science and mathematics integration

17. PD Content—Subject Matter and Pedagogy Contribution of Professional Development Emphasis Areas to Practice Reported by Participants at End of Professional Development Projects 0=none to 10=very much

18. How to Achieve Effective Balance of Subject Matter and Pedagogy SEMO—Integrating pedagogy into math content week NMSU—The learning cycle and mathematics Rockhurst—The learning cycle and science

19. PD Delivery What the research says: Effective PD is driven by a well-defined image of effective classroom learning and teaching (Loucks-Horsley et al., 2003)

20. PD Delivery How to use a framework to guide effective PD: Lincoln Rockhurst

21. PD Delivery Challenges: Diversity of project teams Shared vision develops over time Involvement of local school district personnel

22. PD Delivery What the research says: Effective PD engages teachers as adult learners in the learning approaches they will use with their students. (Loucks-Horsley et al., 2003)  Create a high level of cognitive dissonance, and  Provide sufficient time, structure, and support for teachers to think through the dissonance experienced (Thompson and Zeuli, 1999) What the Cycle 4 RFP said:  Provide research-based instructional strategies to participating teachers

23. PD Delivery PIs and Project Staff, Cycle 4 (p. 38)

24. PD Delivery PIs and Project Staff, Cycle 4 (p. 38) 0=Never; 1=Rarely; 2= Sometimes; 3=Often; 4=All/almost all lessons.

25. PD Delivery PIs and Project Staff, Cycle 4 (p. 35) 0=Never; 1=Rarely; 2= Sometimes; 3=Often; 4=All/almost all lessons.

26. PD Delivery Teacher comments (Cycle 4) I learn really well with this hands-on approach, and until these workshops, I have had limited knowledge of physics, especially how to formally teach energy transfers, etc. We were able to work through the activity in the role of the student and then received the content knowledge that went along with it. By doing the activity myself I was able to completely understand how to recreate the activity for my students during the regular classroom. Learning in the mode we expect our students to learn in was a very valuable tool in my teaching. It allowed me to feel some of the frustration my students would later feel when I changed from being the "giver" of information to the facilitator of students "finding" the information during the inquiry part of each lesson. (p. 48)

27. PD Delivery How to engage teachers in approaches they will use with their students: UMSL - literacy strategies MU - notebooking

28. PD Delivery What the research says: Effective PD provides opportunities for teachers to collaborate with colleagues and other experts to improve their practice. (Loucks-Horsley et al., 2003) What the Cycle 4 RFP said: –Contain grade-level and/or content-area collaboration and work –Provide teachers with the opportunity to reflect on their practices and to give the district feedback on the effectiveness of participation in this activity/experience

29. PD Delivery Teacher comments (Cycle 4) To me, as a fairly new teacher the chance to collaborate with other professionals for that amount of time on our unit was very valuable. I loved having the chance to get to know others and build a team situation rather than feeling like I was on my own. I think talking and visiting with other teachers was invaluable! Being able to collaborate with other classroom teachers that also taught my grade level! I am the only 4th grade teacher within my district and I enjoyed meeting and planning some lessons with other teachers who also taught 4th grade. Time to collaborate with other teachers, share teacher techniques, classroom management strategies, plan and share ideas for lessons -- all of these things allowed me to expand my practices and methods in the classroom Networking with other teachers on the projects. I think I gained more knowledge from them as we worked through the activities. It was great to talk to teachers and realize you had similar problems in your classes. (p. 46)

30. PD Delivery A vehicle for teacher collaboration: MSSU--E-conferencing

31. PD Sustainability What the research says: Effective PD builds capacity for sustainability (Loucks-Horsley et al., 2003) What the Cycle 4 RFP said: Proposal narrative should:… discuss how teachers and other participants will be actively engaged over the life of the project and the potential for the project’s sustainability beyond the end date of the grant

32. PD Sustainability Sustainability in ITQG projects: Year-long sustainability Multi-year sustainability Post-project sustainability

33. Year-long Sustainability What did you do in your Cycle 4 project to promote teacher engagement throughout the year?

34. Teachers’ Views PD Delivery Mechanisms nMSDmax/min Summer institute 2198.32.01/10 School-year call backs 2217.12.50/10 Other school-based activities 2136.63.10/10 0=no contribution to practice, 10=very much contribution.

35. Year-long Sustainability: Call-back Sessions The Saturday call-back sessions. Much of the time we were not engaged in a meaningful task. I came to the first follow-up session and I felt that I wasn't getting anything out of it. I guess this is why I chose not to come to most of the sessions. The call back sessions because of the stress created trying to juggle my extra-curricular school responsibilities to allow me time to come.

36. Year-long Sustainability: Classroom-based Support Focus on PD content Clarity about the purpose(s) of classroom- based support How to sustain classroom-based support: –UCM –UMKC1

37. PD Sustainability Multi-year sustainability –PD improvement over time –Repeat participants Post-project sustainability –Small number of districts to concentrate impact –Multiple teachers from the same school

38. PD Sustainability Challenges: Participant attrition (within a year; from year to year) “Mile-wide, inch deep” approach to participant recruitment (issues of scale) District priorities Continued funding

39. PD as Part of a System What the research says: School culture, leadership and support, and time for teacher learning influence the quality of PD (Garet et al., 1999; Sparks, 2002) Effective PD links with other parts of the education system (Loucks- Horsley et al., 2003) What the Cycle 4 RFP said: Directly link to district and building school improvement plans; Be developed with extensive participation of teachers, principals, and other administrators; Provide time and other resources for learning, practice, and follow- up; Be supported by district and building leadership.

40. Sample Teacher Comments “Although I enjoyed meeting and discussing content and objectives with other teachers I found that planning and writing lessons … with teachers outside of my own district was not helpful. Each district has their own focus so it was hard to work it out.” “Some of the lessons really did not apply to my district’s alignment.” “This project was wonderful. I only regret that I was not able to use it to the fullest extent based upon decisions made by the district I worked for. “

41. PD as Part of a System How to achieve various partnerships: UMKC2—Partnership with KC schools at planning phase UMSL—Partnership of science and literacy 3 Rivers—Key role of school-based personnel in planning and instructional team (Also links to Math Academy) MSSU—ITQG project as part of larger PD effort MSU—networking with rural schools

42. PD as Part of a System Challenges for future cycles: Improving participation of high-need districts Forming partnerships with K-12 –During design phase –Around assessment issues –For sustainability Linking higher education components

43. Four Effective PD Practices PD content PD delivery PD sustainability PD as part of a system

44. Future of Missouri METS Importance of connecting ITQG with other METS initiatives: –DESE-funded MSP projects –METS Alliance –Teacher Leadership Academy For example: –CMU—Competition for middle school students

45. Questions Copies of the report and Executive Summary available at: www.pdeval.missouri.edu