2. Evaluation Team M.A. Henry Consulting, LLC Co-Lead Marty Henry, EdD, CEP EvaluatorsKeith S. Murray, CEP Quantitative Mark Hogrebe, PhD Analyst Qualitative AnalystKatherine A. Phillips, PhD ContentBob Coulter, PhD AdvisorsCharles R. Granger, PhD

4. Improving Teacher Quality Program Theory of Change

5. The Cycle 12 Projects

6. ITQG Cycle 12 Projects Early Elementary Environmental Education: Field Based Approach Missouri State University, West Plains (4E) (Year 3 of 3) Alice (Jill) Black, Project Director Transforming Mathematics Instruction Using Inquiry and One-to- One Environments (TRIM I+121) Missouri State University, Springfield (Year 3 of 3) Lynda Plymate, Project Director Jan Van-Gilder, Project Director Diana Piccolo, Key Project Leader

7. ITQG Cycle 12 Projects Science Education and Quantitative Literacy: An Inquiry-based Approach Missouri University of Science & Technology, Rolla (SEQL) (Year 1 of 3) V.A. Samaranayake, Co-Director Oran Allan Pringle, Co-Director David Westenberg, Co-Director Ronald Bieniek, Co-Director Evalee Lasater, Co-Director Janna Neiss, Co-Director Southeast Missouri State University: Making Mathematicians: Learning to Think and Apply, Cape Girardeau (MM) (Year 2 of 3) Rita Fisher, Project Director Linda Null, Co-Director Julie Antill, Co-Director

8. ITQG Cycle 12 Projects University of Missouri: Preparing Elementary Mathematics Specialists to Serve Missouri Schools (PrepEMS) (Year 1 of 1) Barbara Reys, Co-Director Ann McCoy, Co-Director

9. Location of Projects in ITQG Cycle 12

10. Location of Project Schools in ITQG Cycle 12

11. Location of Teachers’ Schools in ITQG Cycle 12

12. Location of Students in ITQG Cycle 12

13. All Full-Time Participants in Cycle 12 were LEA Educators

14. Teachers’ Educational Attainment in ITQG Cycle 12 (reported n=159)

15. Percent of Teachers by Grade Levels in ITQG Cycle 12 (reported n=327)

16. Grade-Level Distribution by Project in ITQG Cycle 12

17. Federal Funding for Missouri’s ITQG Program Across Cycles

18. Mean Cost per Teacher by Cycle

19. Selected Characteristics of High-Quality PD Cycle 12 Areas of Strength  Directly linked to improved standards-based student learning  Developed with teacher  Provides time/resources for learning, practice, follow-up  Grade-level collaboration and work  Content-area collaboration and work  Specialization-area collaboration and work  Action research and sharing of findings

20.  Ongoing participation by school and district leadership  Strategies for students of limited English proficiency  Methods of teaching children with special needs  Provides teachers with the opportunity to give the district feedback on the projects Selected Characteristics of High-Quality PD Cycle 12 Areas of Challenge

22. What the ITQG Program Aims to Accomplish through Its Projects Objectives for Teachers  Increase teachers’ knowledge in targeted mathematics and/or science content.  Improve teachers’ pedagogical knowledge and practices in inquiry-based instruction.  Improve teachers’ knowledge and skills in designing, implementing and using of assessment instruments and data.

23. Sample Content Test Item A study reported that in a random sampling of 100 women under the age of 35 showed that 8 of the women were married 2 or more times. Based on the study results, how many women in a group of 5,000 women under the age of 35 would likely be married 2 or more times? Show your work or explain your thinking.

24. Sample Content Test Item Based on the data here, what can you conclude about women over 45 years old?

25. Sample Content Test Item

26. Sample Environmental Education Content Test Item Jose lives in Silver, Monica lives in Lincoln, and Blake lives in Nobel. They want to meet in a city on the map. No one wants to travel for more the 15 minutes. 1. Where could they meet? 2. Explain your answer.

27. Cycle 12 Impact on Teachers: Content

28. Teacher Content Test Results (N=137) P = 0.06 No statistically significant increase pre to post

29. Cycle 12 Impact on Teachers: Practice Classroom Observations  Focus Teachers for Case Study  2+ Teachers per Project  Lessons Observed Two Times Each  Use of Inside the Classroom Observation and Analytic Protocol, Horizon Research, Inc., 2002)

30. Ratings: 1=ineffective instruction, 2=elements of effective instruction, 3=beginning stages of effective instruction, 4=accomplished, effective instruction, 5=exemplary instruction. ITQG Cycle 12 Impact on Teachers’ Practice Ratings of 1 st & 2 nd Classroom Observations: Design & Implementation (N=22 observations among 11 teachers)

31. ITQG Cycle 12 Impact on Teachers’ Practice Ratings of 1 st & 2 nd Classroom Observations: Content & Classroom Culture (N=22 observations among 11 teachers) Ratings: 1=ineffective instruction, 2=elements of effective instruction, 3=beginning stages of effective instruction, 4=accomplished, effective instruction, 5=exemplary instruction.

32. ITQG Cycle 12 Classroom Observation Capsule Ratings Using a 7-Point Scale (N=22 observations among 11 teachers)

33. Self-Reported Obstacles to Cycle 12 Teachers’ Teaching Performance (n=98, 107 items reported)

34. Self-Reported Benefits & Improvements to Cycle 12 Teachers’ Teaching Performance (n=98, 121 items reported)

35. Knowledge and Skills in Designing and Using Assessment Instruments and Data  Many variations of implementation  One day focus, integrated throughout programming, use of local district data to inform instruction  Designed to meet needs of local teachers in each project

37. What the ITQG Program Aims to Accomplish through Its Projects Objectives for Students  Improve student achievement in targeted mathematics and/or science content areas.

38. Cycle 12 Impact on Students: Results Change in analysis due to lack of DESE student state data Examination of schools with treatment teachers vs. comparison schools a. Propensity Score Matching b. Regression Models All grades Science and Math Content c. Compared % Achieving Proficient and Advanced

39. Cycle 12 Impact on Students: Results Regression Analysis

40. Cycle 12 Impact on Students: Results Impact on ITQG Schools Examination of ITQG-only schools for a. Effects of # of Students with ITQG Teachers b. Effects of # of Teachers with Master’s Degrees c. Effects of multiple years of ITQG Participation

41. Cycle 12 Impact on Students: Results # of Students with ITQG Teachers Schools with greater % of students who had ITQG Teachers had a higher % of students scoring Proficient/Advanced on MAP tests. r = 0.24, p < 0.05 for Math r = 0.36, p < 0.01 for Science

42. Cycle 12 Impact on Students: Results Percent of Teachers in ITQG School with Master’s Degrees Schools with greater % of students who had Teachers with Master’s Degrees had a higher % of students scoring Proficient/Advanced on MAP tests r = 0.34, p < 0.01 for Math r = 0.26, p < 0.05 for Science

43. Cycle 12 Impact on Students: Results Number of Years Teachers Have Been in ITQG Schools with greater % of teachers with multiple years in an ITQG program had a higher % of students scoring Proficient/Advanced on science MAP tests. r = 0.32, p < 0.05 for Science No relationship for Math

45. What the ITQG Program Aims to Accomplish through Its Projects Objectives for Educational Institutions  Improve the preparation of pre-service teachers through improvements in mathematics/science content and/or pedagogy courses. AND  Sustain the successful efforts of the ITQG program.

46. Cycle 12 Impact on Educational Institutions  Effects on university pre-service programs Change in preservice ed courses Change in some content courses Preservice effects vary across institutions from none to moderate  Administration supportive – mostly disengaged with direct project activities  Effects on schools and school districts is teacher- dependent and will diminish if/when teachers leave

47. Cycle 12 Case Study Aims: To Investigate  Effects or effectiveness of ITQG participation by multiple teachers in isolated rural schools  Degree to which ITQG participation affected collaborative learning, school curriculum, teacher performance  Role of administrative support in ITQG impact  Motivations and risk factors involved for participation

48. Cycle 12 Case Study Methods  Mixed-method, intensive study of two rural ITQG schools with multiple teachers in multiple projects through more than one cycle  Seven teachers  Three projects represented  Data from surveys, content tests, observation results, interviews with school faculty and staff, extant school and community sources

49. Cycle 12 Case Study Does participation in an ITQG project by multiple teachers in small, rural schools serve to intensify positive impact? Yes – With multiple ITQG teachers, schools experienced  Enhanced teacher collaboration and group planning  Greater cross-grade integration of content  Discussion of STEM content and sharing of activities  School curriculum effects related to adoption of ITQG components (where teachers responsible for school curriculum planning)

50. Does participation in an ITQG project by multiple teachers in small, rural schools serve to intensify positive impact? But  Effects wholly teacher-determined and activated  Decision to participate affected by PD distance, stipends  Lack of administrator engagement in ITGQ challenged sustained incorporation of improvements  Lack of administrator engagement in ITQG had directly negative consequences for teachers, in their view  Institutionalization impossible without institutional intent Cycle 12 Case Study

51. Cycle 12 Institutional Impact  Sustainability – promising results for local district sustainability through teacher networks  Projects teams are strong  Will sustain effects if they remain engaged with teachers and schools  Much depends on integration with school and district planning and administrative connectedness

52. Sample Pre/Post SNA Potential for Sustaining Support Network at Start of First Cycle of Funding (Project 1) Support Network At End of Three Cycles of Funding (Project 2) Circles = Project Teachers, Orange Circles = former or other project teachers, Squares = Administrators; Triangles = PIs; Down Triangles = Other teachers; Diamonds = Relatives of Participants; + = Others, Yellow Figure = Internet

53. Question: To what extent does school/district administration participation/partnership in ITQG control or mediate achievement of program and project objectives?

54. Feedback to MDHE You have completed at least one year of your project. What feedback or questions do you have for DHE or evaluators that would help you do your work better?

55. Feedback from Projects Consider administering external evaluation posttest at next to last follow up session; follow up with absentees; projects will assist in follow-up Clarify external eval needs regarding student internal test data (one copy of each test instrument with grade level, teacher, total test is worth, students coded with pre/post scores) Have teachers send student data to projects as soon as they give the posttest

56. Contact Marty Henry mahenry@mahenryconsulting.com Keith Murray keithsmurray@mahenryconsulting.com St. Louis Office Phone: 314-771-4991 Fairfax Office Phone: 703-204-4760