1. Milwaukee Mathematics Partnership Program Evaluation MSP Regional Conference November 8 2006

2. November 6-7, 20062 MMP Goals  Comprehensive Math Framework  Distributed Leadership  Teacher Learning Continuum  Student Learning Continuum

3. November 6-7, 20063 MMP Core Partners  University of Wisconsin—Milwaukee  Milwaukee Public Schools  Milwaukee Area Technical College

4. November 6-7, 20064 Evaluation Goals  Help the MMP better serve its constituents and improve its effectiveness  Serve the broader mathematics education community through documentation and dissemination of MMP activities

5. November 6-7, 20065 Evaluation Logic Model Student Achievement Teacher Content & Pedagogical Knowledge Math Faculty Involvement Learning Team Effort School Buy-in Teacher Involvement New Courses District Buy-in MPA Ownership MATC Buy-In UWM Buy-In Classroom Practice MMP Activities Proximal Outcomes Distal Outcomes

6. November 6-7, 20066 2005-06 Evaluation Activities  MMP Online Survey  MTS Survey  Learning Team Observations  Classroom Observations  Assessment of Teacher MKT  Social Network Analysis  MPS Data Mining

7. November 6-7, 20067 Presentation Overview  Part I: District Wide Analysis  Part II: School Case Studies

8. November 6-7, 20068 Part I: District Wide Indicators Student Achievement Learning Team Effort School Buy-in Teacher Involvement Classroom Practice

9. November 6-7, 20069 Part I Activities  MMP Survey Designed to measure differences in the quantity and quality of MMP related activities  MTS Survey Designed to measure how well MTS perceived school to be, in terms of meeting the goals of the MMP  MKT Assessment Designed to assess teachers’ mathematical knowledge for teaching  WKCE Tests Designed to assess students’ mathematical content knowledge

10. November 6-7, 200610 Respondents Number of MPS Respondents by Role in the MMP Academic Year 2004-05 2005-06 Math Teacher Leader124140 Learning Team Member & Mathematics Teacher 167284 LT Member (Administrative)80171 Math Teacher Only6761340 Literacy Coach4794 Total10942029

11. November 6-7, 200611 Research Questions 1. Validity of MMP Survey 2. Change in MMP Impact 3. Characteristics of math-focused schools 4. MMP Impact on student achievement 5. Characteristics of quality learning teams 6. Characteristics of quality MTLs

12. November 6-7, 200612 Research Question #2  Has the perceived impact or focus of the MMP changed since last year? Analytical Approach  Dependent t-tests conducted at the school level for all school level variables obtained in both administrations of the MMP Survey

13. November 6-7, 200613 Results – Trends in Impact of MMP  This year statistically significant increases MTLs reported discussing mathematics with others at their school (t(90) = 12.06, p <.001) Teachers reported engaging in activities designed to align their curriculum to standards (t(111) = 8.53, p <.001) Teachers reported engaging in activities designed around CABS or student work samples (t(106) = 7.04, p <.001)

14. November 6-7, 200614 Research Question #3  What variables characterize schools that are more focused on increasing student achievement in mathematics? Analytical Approach  Stepwise multiple regression

15. November 6-7, 200615 Results – Characteristics of Schools with Greater Math Focus  68% of variability in schools’ overall self- reported focus on mathematics could be explained by differences in: Teachers reported working together to improve content and pedagogical knowledge (b =.46, t = 6.7, p <.001) Teachers reported consistent instructional practices used at their school (b =.14, t = 2.4, p =.018) Teachers perceived the Learning Team to be supportive of efforts to improve math teaching and learning (b =.38, t = 5.6, p <.001)

16. November 6-7, 200616 Research Question #4  What variables help to explain differences in the percentage of students classified as proficient in mathematics? Analytical Approach  Stepwise multiple regressions controlling for previous achievement and SES

17. November 6-7, 200617 Results – Impact of MMP on Increasing Student Achievement  Schools with a stronger focus on increasing student achievement in mathematics are have a higher percentage of students proficient in mathematics, after controlling for SES and previous achievement (b =.26, t = 3.7, p =.001)  An additional 7% of variability in student proficiency rates explained by the addition of this predictor

18. November 6-7, 200618 Research Question #5  What variables characterize Learning Teams that are perceived to be more helpful in terms of increasing student achievement in mathematics? Analytical Approach  Stepwise multiple regression

19. November 6-7, 200619 Results – Characteristics of Supportive Learning Teams  64% of variability in schools’ overall perception of the level of support provided by the Learning Team could be explained by differences in: Teachers reported working together on improvement activities designed around CABS or student work samples (b =.41, t = 5.5, p <.001) Teachers reported a greater alignment between their school’s adopted curriculum and standards/learning targets (b =.18, t = 2.4, p =.021) Teachers perceived the MTL to be supportive of efforts to improve mathematics teaching and learning (b =.46, t = 5.9, p <.001)

20. November 6-7, 200620 Research Question #6  What variables characterize Math Teacher Leaders that are perceived to be more helpful, in terms of increasing student achievement in mathematics? Analytical Approach  Stepwise multiple regression

21. November 6-7, 200621 Results – Characteristics of Supportive Math Teacher Leaders  42% of variability in schools’ overall perception of the level of support provided by the MTL be explained by differences in: Teachers reported working together on improvement activities designed around CABS or student work samples (b =.38, t = 4.5, p <.001) Teachers reported a greater alignment between their school’s adopted curriculum and standards/learning targets (b =.26, t = 3.0, p =.004) MTLs perceived themselves as being supported by others at their school (b =.27, t = 3.2, p =.002)

22. November 6-7, 200622 Conclusions  MTSs in general have a strong sense of what is going on with school leadership, but less awareness about activity at the classroom level.  MMP efforts are being felt beyond the learning team and MTL by school staff  MMP activities are helping schools become more focused on increasing student achievement in mathematics  Schools that are more focused have increased the proportion of students proficient in mathematics  Adoption of MMP-related principles is reported to be related to supportive learning teams and to supportive Math Teacher Leaders

23. November 6-7, 200623 Part II: Case Study Schools Student Achievement Teacher Content & Pedagogical Knowledge Learning Team Effort School Buy-in Teacher Involvement Classroom Practice Collaboration

24. November 6-7, 200624 Eleven Case Study Schools  Schools were diverse in terms of Type Geography Student demographics

25. November 6-7, 200625 Case Study Data Collection  22 learning team observations—2 in each school  44 classroom observations—4 in each school; 2 teachers observed 2 times each  MKT Assessment for math teachers  SNA Survey for math education ‘stakeholders’

26. November 6-7, 200626 Results of Learning Team Observations Team Functioning Leadership Participation Organization & Structure Results Overall Functioning MMP Issues Math Vision Integration Math Leadership MMP Work Overall MMP Overall, strongest areas were participation & mathematics leadership Biggest areas for improvement were math vision & results

27. November 6-7, 200627 Characteristics of Hi-Lo Scoring Learning Teams—Team Functioning  Distributed leadership  Positional authority is less important  Multiple views are represented and heard  Multiple segments of the school are represented  Written agenda, note taker, facilitator  Explicit action items  Participants have hi knowledge and skill levels  Principal does all the talking  A few individuals dominate the discussion  No agenda or team is easily distracted from the agenda  Little follow-through on assignments  No clear action items Hi Lo

28. November 6-7, 200628 Characteristics of Hi-Lo Scoring Learning Teams—MMP Issues  Consistent curriculum  Math is addressed alongside and in combination with other subjects  Coherent within grades and (at times) across grades  MTL clearly in charge with respect to math  Attention to CABS; reference to MMP courses; reviewing student work  Variation in curriculum  Math not addressed at the meeting  No clear math leader—i.e., hard to tell who the MTL is  Confusion about the MMP and CMF  Too much non-academic housekeeping  School climate is the priority Hi Lo

29. November 6-7, 200629 Results of Classroom Observations General Practice Articulating Math Task Formative Assessment Overall Comprehensive Math Framework Understanding Computing Application Reasoning Engagement Overall, strongest areas were articulating the math task & understanding Biggest areas for improvement were use of formative assessment & engagement

30. November 6-7, 200630 Characteristics of Hi-Lo Scoring Classroom Performance—General  Math task within the lesson was easy to identify  Math task was discrete and level-appropriate  Encouraging self- assessment and peer- assessment  Establish criteria for proficiency  Promoting problem solving and independent thinking  Math task was to complex or obscure  Only feedback provided was if answer was correct  Little teacher involvement in the lesson  Feedback focuses on student behavior Hi Lo

31. November 6-7, 200631 Characteristics of Hi-Lo Scoring Classroom Performance—CMF  Student explanations sought  Computation is presented as a means to an end  Problem solving was emphasized  Students had to justify solutions  Lessons are made relevant by using everyday things like money or time and seeking examples from students’ lives  Close ended questions are emphasized  Only one way to solve problems presented  Minimal time allowed to share solutions  Students not accountable for responding to questions  Problems not presented in context Hi Lo

32. November 6-7, 200632 Results of MKT Assessment Number & Operations 43 item assessment addressed 3 content areas: AlgebraGeometry Overall Score &&

33. November 6-7, 200633 Results of MKT Assessment Average IRT Scores n Number & Operations AlgebraGeometryTotal Lo 12-0.78-0.60-0.65-0.59 HI 320.120.310.200.22 Mean-0.36-0.13-0.32-0.31 Median-0.38-0.09-0.39-0.44 SD0.700.770.790.75

34. November 6-7, 200634 Social Network Analysis  Math stakeholders in each school were asked to name individuals with whom the communicated about mathematics  Statistical analysis focused on 1. Network and in-school density 2. Importance of MTL and MTS

35. November 6-7, 200635 Metricn Total Named Network density Density in school MTL Role--In Degree MTS Role--In Degree Lo13313.9%7.6%3.331.14 Hi329111.4%31.1%28.2418.52 Mean21.154.06.7%17.6%13.815.31 SD6.817.62.6%9.6%7.24.9 Median19486.2%15.4%13.073.75 Overall SNA Results Density—a perfect score is 100% where everyone names everyone else In-Degree scores are relative measures

36. November 6-7, 200636 Example Network

37. November 6-7, 200637 Example Network

38. November 6-7, 200638 Report Card Indicators  19 indicators in 7 domains based on in- school data collection, online surveys, and MPS data 1. MTS Assessment 2. Collaboration 3. Learning Teams 4. Classroom Practice 5. Professional Development 6. Teacher MKT 7. Student Achievement

39. November 6-7, 200639 Report Card Results Student Achievement Teacher Content & Pedagogical Knowledge Learning Team Effort School Buy-in Teacher Involvement Classroom Practice WKCE Mean % Proficient = 44% Overall rating = 3.5 Gap MTL v. other teacher =.2 Teacher Engagement = 3.2 Overall IRT = -0.34 Algebra IRT = -0.18 Team Functioning = 3.5 MMP Principles = 3.6 LT Quality = 3.1 PD Hrs. = 17.8 Facilitation Hrs. = 1.0 PD Quality = 3.1 Network density = 6.7% / School density = 17.6% MTL Role = 13.8 / MTS Role = 5.3 SR MTL Engagement = 4.4 / MTS Quality = 3.0 MTS Assessment = 38.3 of 55

40. November 6-7, 200640 Student Achievement & In-School Network Density

41. November 6-7, 200641 Student Achievement & Learning Team MMP

42. November 6-7, 200642 Student Achievement & Professional Development

43. November 6-7, 200643 Conclusions  MMP is advancing concepts, ideas, & principles that can help schools improve student achievement results in math.  Schools that score well with regards to MMP-related metrics have higher student achievement.  Learning team adoption of MMP ideas and dense in-school communication networks are predictors of high student achievement

44. November 6-7, 200644 Conclusions  At the same time… Some MPS schools are lagging behind in terms of adopting MMP ideas. These schools perform do not score as well on MMP metrics, which is consistent with student achievement results.  We know that other factors—prior year student achievement and SES—are stronger predictors

45. November 6-7, 200645 Evaluation Next Steps District Wide Analysis  Continue online survey & data mining  Improve ability to link student and teacher data working with MPS Case Study Schools 1. Recruit case study schoolsNov 2. Plan observationsNov-Dec 3. Observations Round 1Jan-Feb 4. Observations Round 2March-April 5. MKT AssessmentMay 6. SNA SurveyMay