1. Improving Teacher Quality Grants, Cycle 3: External Evaluation Report December 8, 2006 University of Missouri-Columbia Evaluation Team

2. Principal Investigators Sandra Abell Fran Arbaugh James Cole Mark Ehlert John Lannin Rose Marra Graduate Research Assistants Kristy Halverson Kristen Hutchins Zeynep Kaymaz Michele Lee Dominike Merle Meredith Park Rogers Chia-Yu Wang

3. Context of the Evaluation Improving Teacher Quality grants program, Cycle 3, 2005-2006 Focus on high-need schools 9 professional development projects Science and Mathematics, grades 4-8

4. Evaluation Model Adapted from Guskey, 2000

5. Purpose of Evaluation Formative evaluation PD environment evaluation Summative evaluation –Participant reaction –Participant learning—content knowledge and inquiry –Participant use of knowledge –Organization change –Student learning

6. Methods—Formative Site visits –Interviews: teachers and staff –Observations –Formative feedback report

7. Methods—PD Environment Teacher Participant Data Questionnaire Site visits –Interviews: teachers and staff –Observations Surveys to PIs (Teaching Philosophy Survey and Seven Principles) PI preliminary report

8. Methods-Outcomes Participant reactions –Site visits –Teacher Participant Survey 1 and 2 Participant learning—content knowledge –Project-specific tests (all 9 projects) Participant learning—inquiry –Teaching Philosophy Survey –Seven Principles Participant use of knowledge –Teacher Participant Survey –Interviews –Seven Principles –Implementation Logs

9. Methods--Outcomes Organization change –Higher Education Impact Survey Student learning –Teacher-assessed (3 projects) –Teacher Participant Survey –MAP analyses

10. Participant Summary 252 participants 86% female; 81% white 40% held a masters degree or higher 76% held their first Bachelor’s degree in a field other than science or math Represented 76 different Missouri school districts, 6 private schools, and 2 charter schools Directly impacted 16,747 students in 2005-2006

11. Assigned Teaching Levels and Subjects of Participants

12. Change in Teaching Assignment

13. Teaching Experience

14. Elem/Middle/Junior High Certification Status

15. High School Certification Status

16. PD Hours Completed in Past 3 Yrs

17. Experience with Web-based PD in Past 3 Years

18. Percentage of Participants from High-Need Districts

19. PD Coverage – Schools and Teachers

20. Results PD Environment Participant Reactions Outcomes –Participant Content Knowledge –Participant Knowledge of Inquiry –Participant Use of Knowledge of Inquiry –Organization Change –Student Learning

21. PD Environment--Projects by Standards Area Science # PD Projects Matter & Energy7 Force, Motion & Mechanical Energy6 Living Organisms1 Inquiry Approach to Science Education9 Ecosystems2 Earth Science4 Astronomy2 Scientific Inquiry4 Science & Technology0 Mathematics # PD Projects Numbers & Operations3 Algebraic Relationships1 Geometric & Spatial Relationships0 Measurement0 Data & Probability1 Inquiry- based/Problem- centered Instruction3

22. PD Environment—PI Beliefs (n=19) MSD Explainer vs. Facilitator 4.20.8 Whole class activity vs. Many things going on 4.20.7 Curriculum coverage vs. Sense-making 3.81.1 Textbook content vs. Student interests 4.30.7 Breadth of concepts vs. Depth of concepts 3.91.3 least constructivist response = 1, neutral = 3, most constructivist = 5

23. Participant Reactions End of SummerEnd of Project n MSD n M Confidence in my content knowledge improved1303.00.81172.90.7 Confidence in my ability to teach improved1302.80.81163.00.8 I will use/have used materials and activities from this project in my classes during the year1303.40.81163.11.0 The PD project was relevant to my teaching assignment1303.40.91163.30.8 High quality instruction was delivered in this project1303.60.61173.50.7 Instructors modeled good practice1293.60.71173.40.8 Overall, I am satisfied with my experiences in this project1283.70.61173.50.8 1-5 scale

24. Participant Performance on Content Knowledge—Post/Pre Tests Posttest scores presented as a percent of pretest scores.

25. Participant Change in Inquiry Knowledge Change (post PD- pre PD; n = 124) Principlediff t 1. Communication0.1 0.6 2. Social Learning0.1 1.8 3. Active/Inquiry-Based0.2 3.2** 4. Prompt Feedback0.1 1.1 5. Problem Solving0.1 1.4 6. Content Knowledge0.2 3.6*** 7. Diversity0.1 1.6 *p <.05. **p <.01. ***p <.001

26. Participant Change in Inquiry Usage Change (post PD- pre PD; n = 124) Principlediff t 1. Communication0.2 6.3*** 2. Social Learning0.4 7.2*** 3. Active/Inquiry-Based0.3 8.4*** 4. Prompt Feedback0.4 9.1*** 5. Problem Solving0.3 6.5*** 6. Content Knowledge0.3 7.6*** 7. Diversity0.2 5.9*** *p <.05. **p <.01. ***p <.001

27. Participant Use of Knowledge Based on PD Components MSD Content Improving content knowledge3.00.8 Materials Using technology effectively to enhance your teaching2.51.1 Developing materials for use with your students3.00.9 Collaboration Collaborating with other teachers3.01.0 n=116 0-4 scale

28. Impact on Participant Use of Knowledge (cont) MSD Assessment Assessing student learning2.60.9 Analyzing student performance data2.61.1 Pedagogy Creating lessons related to content standards2.8 Creating inquiry-based / problem-centered classrooms3.20.9 Increasing student motivation3.00.7 Using inquiry-based / problem-centered teaching3.20.7 Participating in classroom activities as your students would3.00.8 How to implement activities3.00.8 Managing inquiry-based / problem-centered classrooms2.80.9

29. Organization Change--Impact on Higher Education Team members from five projects responded to HEI Survey –Establishment of new science courses related to the PD projects –Establishment of new education courses –Redesign of courses to include more inquiry- based labs –New or strengthened collaborations between education and science –Increased grant writing activity on campus

30. School Level Performance on MAP Map Index and % Top 2 Levels Served vs not served schools by High Needs status Science – 2005-06 compared prior years’ average performance Math – no historical comparison possible: examined performance levels by group

31. Overall Impact of PD Projects

32. Performance Levels in Science

34. Changes in MAP Science Performance – Index Scores

35. Changes in MAP Science Performance -- Proficiency

36. Average MAP Index Scores by Grade Level

37. Average MAP Math Proficiency by Grade Level

38. Summary of Results Teachers were overall satisfied with PD experiences –Valued most: staff, engaging in activities as students would, opportunity to improve content knowledge, working with other teachers; –Valued least: lectures,activities geared toward a different grade level or subject matter than they taught, loosely structured follow-up sessions with no clear purpose.

39. Summary of Results (cont) Assessment components less emphasized than content and inquiry components. Teachers gained content knowledge Evidence of some improved teacher practice attributed to projects. Student learning data mixed. Evidence of impact on higher education is limited but promising in some projects.

40. Conclusions: Effective Project Design Features Projects demonstrated effective practice to varying degrees. Alignment of content emphasis areas between projects and teacher/school needs is critical. Shared vision/collaboration with team implemented in a variety of ways. Effective emphasis areas: learning science/math through inquiry; collegial learning with teachers; long-term PD activities; sense of community.

41. Conclusions (cont.) The “smorgasbord” approach – while well intentioned seemed difficult to carry out. Emphasis on mathematics in overall cycle 3 ITQG program was somewhat limited. Individual projects improve over time. Evaluator role balance between program and projects continues to be an issue.

42. Limitations Necessity of sampling. Instruments align with overall program not specific projects. Low overall response rates –Implementation Logs –End of Project instruments –Higher education impact Overall evaluation vs. project specific. Lack of and alignment of student achievement data. Impact on evaluation due to ongoing team collaboration with PIs and K-12 partners.

43. Recommendations Project Directors: Continue to build strong relations among PIs and instructional staff. Build stronger K-12 partnerships. Balance content and pedagogy. Emphasize and provide opportunities for practice and feedback on classroom assessment. Encourage participation in evaluation activities. Take advantage of formative feedback. Use literature on best practice when designing and implementing PD.

44. Recommendations External Evaluators: Explore ways to reduce participant time on evaluation. Be proactive in working with PIs and K-12 organizations. Continue to work with PIs through all phases of evaluation. Work with MDHE to examine our roles as evaluators.

45. Recommendations MDHE: Continue funding multi-year projects. Encourage true partnerships via RFP wording and reward systems. Require that the majority of participants are from high-needs districts. Require minimum hours of PD per project. Support PI cross-fertilization of best practices.

46. Questions Copies of the report and Executive Summary available at: www.pdeval.missouri.edu