1. Mathematics and Science Partnerships Program Regional Conference Washington D.C December 11, 2006 Getting Results in the Classroom F. Joseph Merlino, PI/PD The Math Science Partnership of Greater Philadelphia Jill Feldman, Ph.D. Research for Better Schools

2. Exhibit 1. Conceptual Model of Mathematics and Science Partnerships Program Develop partnership of high- need school districts and an IHE’s science, technology, engineering, mathematics faculty Provide professional development to strengthen teachers’ content knowledge Improve classroom instruction Improve student achievement in mathematics and science

4. MathScience Elem Sch Middle Sch High Sch Teacher Recep- tivity to Prof Dev High Medium Low Math Science Content Know- ledge Low Medium High Relative Number of Teachers to Train 5 2 1 Number of Students Affected per Teacher 1 class 2.5 classes 5 classes Professional Development Considerations and Tradeoffs

5. “Wide Net” Approach : Casting for Participating Teachers from Multiple Districts, Schools or Departments Summer Institute Academic Year PositiveNegative  Easier to fill classes  Ease of scheduling at a single location, e.g. College campus  College Professors have much more time available to present  Willing teachers, higher receptivity to PD  Can more easier schedule intensive multi day and week PD  Individual teachers volunteer or have to be paid to participate  Partial departmental or school participation  More difficult academic year follow-up if across many schools or districts  More difficult to ascertain whether summer PD has translated into the academic year classroom  More difficult to gauge classroom effects  Schools Can compel teacher attendance as part of regular work day  Do not have to pay teachers to attend  Easier to match training with actual classroom practice as it is occurring  More difficult to schedule common training dates and sites across more than one district  More difficult to obtain the services of PD college professor  More difficult to differentiate PD or different math and science teachers within a school of department  Effects of a lack of common curriculum among participants more of an issue

6. “Narrow Cast” Approach : Casting for Participating Teachers from A Single District, School or Department Summer Institute Academic Year PositiveNegative  Ease of scheduling at a single location, e.g. College campus  College Professors have much more time available to present  Willing teachers, higher receptivity to PD  Can more easier schedule intensive multi day and week PD  Less difficult academic year follow-up  Less difficult to gauge classroom effects  More difficult to fill classes  Individual teachers volunteer or have to be paid to participate  Partial departmental or school participation  Somewhat less difficult to ascertain whether summer PD has translated into the academic year classroom  More vulnerable to not providing for new hire needs in August  Schools can compel teacher attendance as part of regular work day  Do not have to pay teachers to attend  Easier to match training with actual classroom practice as it is occurring  Easier to enlist building and district administrators for follow thorough  Less effects of a lack of common curriculum among participants less of an issue  More difficult to fill classes with willing teachers  More difficult to obtain the services of PD college professor  More restrictive scheduling dates available  More vulnerable to last minute school scheduling changes  More vulnerable to school or district driven PD priorities that are not math or science related

7. One Year or Multiple Years? One Year Two Years PositiveNegative  Clear Beginning and End  Easier to replicate Three Years Four Years  Teachers need a year with new learning to fully incorporate it  Limitations on depth and breadth  More depth and breadth  Teachers can reflect upon and perfect past year’s PD  More difficult to sequence training for new hires  Teachers missing days becomes more problematic  More depth and breadth of ideas  Curriculum and assessment changes becomes more robust  Teacher leadership emerges  Teachers can start to train others  Teacher leadership emerges  Teacher and school culture changes  Teacher attrition becomes more noticeable  More difficult to sequence training for different years  Teacher attrition becomes a serious problem  Need to make-up many training days to build coherence

8. Reluctant Realizations Teacher and administrator attrition and turnover rates are not an aberration, they are a constant feature of school reality that means PD needs to be ongoing every year. We still have early 20 th century designed schools that were not designed by schedule, organization or employment contracts to be professional learning communities.

9. Slogging through the “Wetlands” Using Evaluation Findings to Inform Decision Making

10. Operationally Defining “the Wetlands” Linking the evolving theory of action to evaluation plans Developing meaningful/measurable objectives Selecting/designing data collection measures Reflecting upon/refining evaluation plans to support decision making

11. Using an Evolving Theory of Action to Evaluate Program Impact Implies ongoing need to review (and revise) originally proposed: –Objectives –Benchmarks –Measures –Impact

12. One Practice-Based Example: Breaking Down Conceptual Thinking: Goal 1: To ensure that all students have access to, are prepared for, and are encouraged to participate and succeed in challenging and advanced mathematics and science courses. Proposed Objective: Empower all middle school and high school math and science teachers to be able to teach challenging and advanced math and science courses.

13. Step 1:Refining the Objective

14. Step 2:Establishing Benchmarks

15. Step 3: Identifying/Developing Measures

16. Step 3: Identifying/Developing Measures (continued) Revised Objective 1.4: Increase ability of mathematics and science teachers’ to implement high quality curricula. Benchmark Measure Responsible Party Ongoing Issues 75 % observed PD will receive ratings of 3 or 4 on scale of 5. # ratings 3 or higher # total ratings RBS Value-added of evaluator data compare to data provided by participants 75% participating teachers will show significant increase in knowledge. # teachers demonstrating increased content knowledge # total teachers PD Facilitators/RBS Systematic use of rubric (developed by RBS)

17. Step 4: Based on the New Information: Improving Evaluation Plans Analyze collected data Compare findings to targeted benchmarks Provide opportunities to reflect on reasons for findings Consider revisions to improve usefulness of data collected for decision-making

18. Comparing Results to Benchmarks: Benchmark 1

19. Comparing Results to Benchmarks: Benchmark 2

20. Comparing Results to Benchmarks: Benchmark 3

21. Summary  To support the needs of project decision-makers, evaluation plans need to respond effectively to:  inevitable program changes  unanticipated evaluation findings.  To be used for planning purposes, data need to be shared:  in manageable “chunks”  at the “right time(s)” (i.e., with adequate time for sense-making)  in a coherent format (given the audience).