Milwaukee Mathematics Partnership Program Evaluation MSP Regional Conference November

November 6-7, MMP Goals  Comprehensive Math Framework  Distributed Leadership  Teacher Learning Continuum  Student Learning Continuum

November 6-7, MMP Core Partners  University of Wisconsin—Milwaukee  Milwaukee Public Schools  Milwaukee Area Technical College

November 6-7, Evaluation Goals  Help the MMP better serve its constituents and improve its effectiveness  Serve the broader mathematics education community through documentation and dissemination of MMP activities

November 6-7, Evaluation Logic Model Student Achievement Teacher Content & Pedagogical Knowledge Math Faculty Involvement Learning Team Effort School Buy-in Teacher Involvement New Courses District Buy-in MPA Ownership MATC Buy-In UWM Buy-In Classroom Practice MMP Activities Proximal Outcomes Distal Outcomes

November 6-7, Evaluation Activities  MMP Online Survey  MTS Survey  Learning Team Observations  Classroom Observations  Assessment of Teacher MKT  Social Network Analysis  MPS Data Mining

November 6-7, Presentation Overview  Part I: District Wide Analysis  Part II: School Case Studies

November 6-7, Part I: District Wide Indicators Student Achievement Learning Team Effort School Buy-in Teacher Involvement Classroom Practice

November 6-7, Part I Activities  MMP Survey Designed to measure differences in the quantity and quality of MMP related activities  MTS Survey Designed to measure how well MTS perceived school to be, in terms of meeting the goals of the MMP  MKT Assessment Designed to assess teachers’ mathematical knowledge for teaching  WKCE Tests Designed to assess students’ mathematical content knowledge

November 6-7, Respondents Number of MPS Respondents by Role in the MMP Academic Year Math Teacher Leader Learning Team Member & Mathematics Teacher LT Member (Administrative)80171 Math Teacher Only Literacy Coach4794 Total

November 6-7, Research Questions 1. Validity of MMP Survey 2. Change in MMP Impact 3. Characteristics of math-focused schools 4. MMP Impact on student achievement 5. Characteristics of quality learning teams 6. Characteristics of quality MTLs

November 6-7, Research Question #2  Has the perceived impact or focus of the MMP changed since last year? Analytical Approach  Dependent t-tests conducted at the school level for all school level variables obtained in both administrations of the MMP Survey

November 6-7, Results – Trends in Impact of MMP  This year statistically significant increases MTLs reported discussing mathematics with others at their school (t(90) = 12.06, p <.001) Teachers reported engaging in activities designed to align their curriculum to standards (t(111) = 8.53, p <.001) Teachers reported engaging in activities designed around CABS or student work samples (t(106) = 7.04, p <.001)

November 6-7, Research Question #3  What variables characterize schools that are more focused on increasing student achievement in mathematics? Analytical Approach  Stepwise multiple regression

November 6-7, Results – Characteristics of Schools with Greater Math Focus  68% of variability in schools’ overall self- reported focus on mathematics could be explained by differences in: Teachers reported working together to improve content and pedagogical knowledge (b =.46, t = 6.7, p <.001) Teachers reported consistent instructional practices used at their school (b =.14, t = 2.4, p =.018) Teachers perceived the Learning Team to be supportive of efforts to improve math teaching and learning (b =.38, t = 5.6, p <.001)

November 6-7, Research Question #4  What variables help to explain differences in the percentage of students classified as proficient in mathematics? Analytical Approach  Stepwise multiple regressions controlling for previous achievement and SES

November 6-7, Results – Impact of MMP on Increasing Student Achievement  Schools with a stronger focus on increasing student achievement in mathematics are have a higher percentage of students proficient in mathematics, after controlling for SES and previous achievement (b =.26, t = 3.7, p =.001)  An additional 7% of variability in student proficiency rates explained by the addition of this predictor

November 6-7, Research Question #5  What variables characterize Learning Teams that are perceived to be more helpful in terms of increasing student achievement in mathematics? Analytical Approach  Stepwise multiple regression

November 6-7, Results – Characteristics of Supportive Learning Teams  64% of variability in schools’ overall perception of the level of support provided by the Learning Team could be explained by differences in: Teachers reported working together on improvement activities designed around CABS or student work samples (b =.41, t = 5.5, p <.001) Teachers reported a greater alignment between their school’s adopted curriculum and standards/learning targets (b =.18, t = 2.4, p =.021) Teachers perceived the MTL to be supportive of efforts to improve mathematics teaching and learning (b =.46, t = 5.9, p <.001)

November 6-7, Research Question #6  What variables characterize Math Teacher Leaders that are perceived to be more helpful, in terms of increasing student achievement in mathematics? Analytical Approach  Stepwise multiple regression

November 6-7, Results – Characteristics of Supportive Math Teacher Leaders  42% of variability in schools’ overall perception of the level of support provided by the MTL be explained by differences in: Teachers reported working together on improvement activities designed around CABS or student work samples (b =.38, t = 4.5, p <.001) Teachers reported a greater alignment between their school’s adopted curriculum and standards/learning targets (b =.26, t = 3.0, p =.004) MTLs perceived themselves as being supported by others at their school (b =.27, t = 3.2, p =.002)

November 6-7, Conclusions  MTSs in general have a strong sense of what is going on with school leadership, but less awareness about activity at the classroom level.  MMP efforts are being felt beyond the learning team and MTL by school staff  MMP activities are helping schools become more focused on increasing student achievement in mathematics  Schools that are more focused have increased the proportion of students proficient in mathematics  Adoption of MMP-related principles is reported to be related to supportive learning teams and to supportive Math Teacher Leaders

November 6-7, Part II: Case Study Schools Student Achievement Teacher Content & Pedagogical Knowledge Learning Team Effort School Buy-in Teacher Involvement Classroom Practice Collaboration

November 6-7, Eleven Case Study Schools  Schools were diverse in terms of Type Geography Student demographics

November 6-7, Case Study Data Collection  22 learning team observations—2 in each school  44 classroom observations—4 in each school; 2 teachers observed 2 times each  MKT Assessment for math teachers  SNA Survey for math education ‘stakeholders’

November 6-7, Results of Learning Team Observations Team Functioning Leadership Participation Organization & Structure Results Overall Functioning MMP Issues Math Vision Integration Math Leadership MMP Work Overall MMP Overall, strongest areas were participation & mathematics leadership Biggest areas for improvement were math vision & results

November 6-7, Characteristics of Hi-Lo Scoring Learning Teams—Team Functioning  Distributed leadership  Positional authority is less important  Multiple views are represented and heard  Multiple segments of the school are represented  Written agenda, note taker, facilitator  Explicit action items  Participants have hi knowledge and skill levels  Principal does all the talking  A few individuals dominate the discussion  No agenda or team is easily distracted from the agenda  Little follow-through on assignments  No clear action items Hi Lo

November 6-7, Characteristics of Hi-Lo Scoring Learning Teams—MMP Issues  Consistent curriculum  Math is addressed alongside and in combination with other subjects  Coherent within grades and (at times) across grades  MTL clearly in charge with respect to math  Attention to CABS; reference to MMP courses; reviewing student work  Variation in curriculum  Math not addressed at the meeting  No clear math leader—i.e., hard to tell who the MTL is  Confusion about the MMP and CMF  Too much non-academic housekeeping  School climate is the priority Hi Lo

November 6-7, Results of Classroom Observations General Practice Articulating Math Task Formative Assessment Overall Comprehensive Math Framework Understanding Computing Application Reasoning Engagement Overall, strongest areas were articulating the math task & understanding Biggest areas for improvement were use of formative assessment & engagement

November 6-7, Characteristics of Hi-Lo Scoring Classroom Performance—General  Math task within the lesson was easy to identify  Math task was discrete and level-appropriate  Encouraging self- assessment and peer- assessment  Establish criteria for proficiency  Promoting problem solving and independent thinking  Math task was to complex or obscure  Only feedback provided was if answer was correct  Little teacher involvement in the lesson  Feedback focuses on student behavior Hi Lo

November 6-7, Characteristics of Hi-Lo Scoring Classroom Performance—CMF  Student explanations sought  Computation is presented as a means to an end  Problem solving was emphasized  Students had to justify solutions  Lessons are made relevant by using everyday things like money or time and seeking examples from students’ lives  Close ended questions are emphasized  Only one way to solve problems presented  Minimal time allowed to share solutions  Students not accountable for responding to questions  Problems not presented in context Hi Lo

November 6-7, Results of MKT Assessment Number & Operations 43 item assessment addressed 3 content areas: AlgebraGeometry Overall Score &&

November 6-7, Results of MKT Assessment Average IRT Scores n Number & Operations AlgebraGeometryTotal Lo HI Mean Median SD

November 6-7, Social Network Analysis  Math stakeholders in each school were asked to name individuals with whom the communicated about mathematics  Statistical analysis focused on 1. Network and in-school density 2. Importance of MTL and MTS

November 6-7, Metricn Total Named Network density Density in school MTL Role--In Degree MTS Role--In Degree Lo %7.6% Hi %31.1% Mean %17.6% SD %9.6% Median %15.4% Overall SNA Results Density—a perfect score is 100% where everyone names everyone else In-Degree scores are relative measures

November 6-7, Example Network

November 6-7, Example Network

November 6-7, Report Card Indicators  19 indicators in 7 domains based on in- school data collection, online surveys, and MPS data 1. MTS Assessment 2. Collaboration 3. Learning Teams 4. Classroom Practice 5. Professional Development 6. Teacher MKT 7. Student Achievement

November 6-7, Report Card Results Student Achievement Teacher Content & Pedagogical Knowledge Learning Team Effort School Buy-in Teacher Involvement Classroom Practice WKCE Mean % Proficient = 44% Overall rating = 3.5 Gap MTL v. other teacher =.2 Teacher Engagement = 3.2 Overall IRT = Algebra IRT = Team Functioning = 3.5 MMP Principles = 3.6 LT Quality = 3.1 PD Hrs. = 17.8 Facilitation Hrs. = 1.0 PD Quality = 3.1 Network density = 6.7% / School density = 17.6% MTL Role = 13.8 / MTS Role = 5.3 SR MTL Engagement = 4.4 / MTS Quality = 3.0 MTS Assessment = 38.3 of 55

November 6-7, Student Achievement & In-School Network Density

November 6-7, Student Achievement & Learning Team MMP

November 6-7, Student Achievement & Professional Development

November 6-7, Conclusions  MMP is advancing concepts, ideas, & principles that can help schools improve student achievement results in math.  Schools that score well with regards to MMP-related metrics have higher student achievement.  Learning team adoption of MMP ideas and dense in-school communication networks are predictors of high student achievement

November 6-7, Conclusions  At the same time… Some MPS schools are lagging behind in terms of adopting MMP ideas. These schools perform do not score as well on MMP metrics, which is consistent with student achievement results.  We know that other factors—prior year student achievement and SES—are stronger predictors

November 6-7, Evaluation Next Steps District Wide Analysis  Continue online survey & data mining  Improve ability to link student and teacher data working with MPS Case Study Schools 1. Recruit case study schoolsNov 2. Plan observationsNov-Dec 3. Observations Round 1Jan-Feb 4. Observations Round 2March-April 5. MKT AssessmentMay 6. SNA SurveyMay