1. 1 Milwaukee Mathematics Partnership Program Evaluation Year 4 Results Carl Hanssen Hanssen Consulting, LLC Cindy Walker University of Wisconsin-Milwaukee November 2007 MTL Meeting This material is based upon work supported by the National Science Foundation under Grant No. 0314898. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).

2. 2 Evaluation Goals  Help the MMP better serve its constituents and improve its effectiveness  Serve the broader mathematics education community through documentation and dissemination of MMP activities

3. 3 MMP Evaluation Logic Model Student Achievement Teacher Content & Pedagogical Knowledge Math Faculty Involvement Learning Team Effort School Buy-in Teacher Involvement New Courses District Buy-in MPA Ownership MATC Buy-In UWM Buy-In Classroom Practice MMP Activities Proximal Outcomes Distal Outcomes

4. 4 Presentation Overview 1. 2007 MMP Online Survey Results & Trends 2. MMP Impact on Student Achievement Gains 3. Detailed Work in 10 MPS Schools

5. 5 1. 2007 MMP Online Survey Results & Trends Learning Team Effort School Buy-in Teacher Involvement Classroom Practice

6. 6 Online Survey Responses Academic Year 2004-052005-062006-07 Math Teacher Leader124140143 Learning Team Member & Mathematics Teacher 167284335 LT Member (Administrative, other) 127165225 Math Teacher Only6761,3401,388 Total1,0942,0292,091 Responses are aggregated within a school so that each school counts as 1 case in the analysis

7. 7 Online Survey Variables 75+ Survey items 17 Composite Variables Example Composite Variable: AlignmentHow aligned a school’s curriculum is to standards and learning targets ItemsI feel the mathematics program my school uses aligns with: MPS learning targets. Wisconsin state standards. Goals of the Comprehensive Math Framework. State/district assessments (WKCE/Terra Nova).

8. 8 Context for these results n Stage 1. Learning Targets Stage 2. Align Targets Stage 3. Designing CABS Stage 4. Examine CABS Stage 5. Descriptive Feedback Year 1, 2003-04 10138%53%9%0%1% Year 2, 2004-05 9718%34%38%5%4% Year 3, 2005-06 8913%26%41%18%2% Year 4, 2006-07 891%9%25%43%23% These data show that there has been progress toward embracing MMP principles in schools

9. 9 Statistically Significant Improvements Quantity of PD Consistency in math instruction Engaging in activities to align curriculum to learning targets Engaging in activities using CABS and student work samples Engaging in activities to gauge student progress Talking about teaching & learning Mathematics with others 2.84 3.01 3.06 3.72 3.42 3.60 3.17 2.88 3.25 3.17 2.99 2.88 Spring 2006Spring 2007 Engagement

10. 10 School Math Focus Consistent curriculum + Teachers working together + PD perceived as valuable Predicts Strong Math Focus

11. 11 Supportive Learning Teams Predicts Perception of a supportive Learning Team MTL perceived as supportive + Curriculum aligned to targets + Learning Team focuses on math + Teachers working together + PD perceived as valuable

12. 12 Supportive MTL Predicts Perception of a supportive MTL PD perceived as valuable + MTS perceived as supportive + Teachers working together + Learning Team focuses on math - Curriculum aligned to targets Schools with a supportive MTL likely aligned curriculum to targets last year

13. 13 1. Conclusion  Across the district, schools are reporting higher levels of involvement with MMP  Similarly, schools report more frequently engaging in activities that the MMP encourages and promotes

14. 14 2. MMP Impact on Student Achievement Student Achievement Learning Team Effort School Buy-in Teacher Involvement Classroom Practice

15. 15 MMP Impact on 2006 Student Achievement Are student achievement gains greater in schools that have more fully embraced MMP principles?

16. 16 Sep 04 Sep 05 Sep 06 Sep 07 Sep 08 State Test Fall 2004 MMP Online Survey Spring 2005 State Test Fall 2005 State Test Fall 2006 State Test Fall 2007 MMP Online Survey Spring 2006 MMP Online Survey Spring 2007 MMP Online Survey Spring 2008 2004-2005 School Year 2005-2006 School Year 2006-2007 School Year 2007-2008 School Year Data Collection Timeline

17. 17 Analytical Approach Use Student Achievement Data from 2005 + MMP Online Survey Results from 2006 to explain variability in Student Achievement Gains from 2005 to 2006

18. 18 Sources of Variability in Student Achievement Scores Variability In Student Achievement In 2006 81% Student 19% School 12% MMP Alignment 79% Other 52% Student Achievement In 2005 48% Other 4 th Grade 9% LT Quality

19. 19 Sources of Variability in Student Achievement Scores Variability In Student Achievement In 2006 78% Student 22% School 9% MMP Alignment 79% Other 56% Student Achievement in 2005 44% Other 5 th Grade 5% LT Quality

20. 20 Sources of Variability in Student Achievement Scores Variability In Student Achievement in 2006 78% Student 22% School 4% MMP Alignment 96% Other 50% Student Achievement In 2005 50% Other 6 th Grade

21. 21 Sources of Variability in Student Achievement Scores Variability In Student Achievement In 2006 76% Student 24% School 10% MMP Alignment 90% Other 58% Student Achievement in 2005 42% Other 7 th Grade

22. 22 Sources of Variability in Student Achievement Scores Variability In Student Achievement In 2006 79% Student 21% School 7% MMP Alignment 93% Other 56% Student Achievement in 2005 44% Other 8 th Grade

23. 23 2. Conclusion  Schools that more fully embrace MMP principles are more likely to show gains in student achievement  MMP influence is perhaps felt most strongly by students in the lower grades

24. 24 3. Detailed Work in 10 MPS Schools Student Achievement Teacher Content & Pedagogical Knowledge Learning Team Effort School Buy-in Teacher Involvement Classroom Practice Collaboration

25. 25 Ten Case Study Schools  Diverse set of schools  School Type 5 K-5 3 K-8 2 6-8  Geography 7 North 3 South  Median students = 430

26. 26 Case Study Data Collection  20 learning team observations—2 in each school  40 classroom observations—4 in each school; 2 teachers observed 2 times each  MKT Assessment for math teachers  SNA Survey for math teachers and administrators

27. 27 Results of Learning Team Observations Team Functioning Leadership Participation Organization/Structure Results Overall Functioning MMP Issues Math Vision Consistency Math Leadership MMP Work Overall MMP Strengths Participation Organization/Structure Areas to Improve Meeting Results Strengths Math Leadership Vision, Consistency Areas to Improve MMP Work

28. 28 Authoritarian Directive leader Little discussion Reporting out Two Learning Team Models Emerging Participatory Active discussion Consensus building Planning Key Observation: to what degree are LT meetings about learning versus school administration?

29. 29 Characteristics of High &Low Rated Learning Teams—Team Functioning  Focus on learning  Distributed leadership  Positional authority is less important  Multiple views are represented and heard  Multiple segments of the school are represented  Written agenda, note taker, facilitator  Explicit action items  Participants have hi knowledge and skill levels  Focus on administration  Principal does all the talking  A few individuals dominate the discussion  No agenda or team is easily distracted from the agenda  Little follow-through on assignments  No clear action items High Low

30. 30 Characteristics of High & Low Rated Learning Teams—MMP Issues  Consistent curriculum  Math is addressed alongside and in combination with other subjects  Coherent within grades and across grades  MTL clearly in charge with respect to math  Attention to CABS; reference to MMP courses; reviewing student work  Variation in curriculum  Math not addressed at the meeting  No clear math leader— i.e., hard to tell who the MTL is  Confusion about the MMP and CMF High Low

31. 31 Results of Classroom Observations General Practice Identify the Math Task Is the Math Correct? Formative Assessment Comprehensive Math Framework Understanding Computing Application Reasoning Engagement Strengths Identify the math task Correct Mathematics Areas to Improve Formative assessment* Strengths Understanding Reasoning Areas to Improve Application Engagement

32. 32 Characteristics of Strong & Weak Rated Classroom—General Practice  Math is correct  Math task within the lesson was easy to identify  Math task was discrete and level-appropriate  Encouraging self- assessment and peer- assessment  Establish criteria for proficiency  Promoting problem solving and independent thinking  Incorrect Math  Math task was too complex or obscure  Only feedback provided was if answer was correct  Little teacher involvement in the lesson  Feedback focuses on student behavior Strong Weak

33. 33 Characteristics of Strong & Weak Rated Classroom Performance—CMF  Student explanations sought  Computation is presented as a means to an end  Problem solving was emphasized  Students had to justify solutions  Lessons are made relevant by using everyday things like money or time and seeking examples from students’ lives  Close ended questions are emphasized  Only one way to solve problems presented  Minimal time allowed to share solutions  Students not accountable for responding to questions  Problems not presented in context Strong Weak

34. 34 Results of MKT Assessment Number & Operations 43 item assessment addressed 3 content areas: Algebra Geometry (2006-07 Focus) Overall Score &&

35. 35 Results of MKT Assessment MKT scores can be interpreted like z-scores Results were aggregated within schools There is tremendous variability in the results Geometry was the MMP PD focus in 2006-07

36. 36 Social Network Analysis  Teachers and administrators in each school were asked to name individuals with whom they communicated about mathematics  This is a key indicator of distributed leadership

37. 37 Mathematics Distributed Leadership Continuum HighLow Tight Network MTL Central Many Links to MTL MTS Inside Many Links to MTS Loose Network MTL Not Central Few Links to MTL MTS Outside Few Links to MTS

38. 38 Low Student Achievement: 2006: 20% Proficient 4-year trend: -4%

39. 39 Medium Student Achievement: 2006: 21% Proficient 4-year trend: -19%

40. 40 High Student Achievement: 2006: 50% Proficient 4-year trend: +7%

41. 41 3. Conclusion  No single factor—e.g., distributed leadership, teacher MKT, learning team performance—is sufficient for success, but all may be necessary  Schools that are performing well do many of the things MMP promotes well, and realize synergy between many of these activities and principles

42. 42 Overall Conclusions  There is support for the argument that schools that have more fully adopted MMP principles are demonstrating stronger outcomes—though there is still a lot of work to do.  MMP Impact, though, is not being felt in all schools—there is tremendous variability in MMP adoption and progress across the district

43. 43 Overall Conclusions  Important considerations for sustaining MMP work Creating Distributed Leadership in a school takes time—and communication is critical Last year the Learning Team was perceived as the most important actor for improving mathematics teaching and learning. This year, in schools that report high levels of math focus, that responsibility seems to be dispersed throughout the school.

44. 44 Overall Conclusions  Important considerations for sustaining MMP work MTL role may be shifting from focal point to facilitator—we see a shift in the perception of who is responsible for helping the school focus on improving mathematics teaching and learning MTS role may more important than ever—schools using the MTS appear further down the path

45. 45 Focus Question  What message will you be taking back about… Your ongoing work to improve math in your school? Specific areas where your school can improve its math education?