1. Measuring the Effect of the Milwaukee Mathematics Partnership on Student Achievement Cindy M. Walker Jacqueline K. Gosz DeAnn Huinker University of Wisconsin - Milwaukee

2. Structure of MMP Math Teaching Specialist (MTS) Learning Team (LT) Literacy Coach (LC) Math Teacher Leader (MTL) Math Teacher

3. Concerns Difficulty measuring impact of MMP on student achievement in mathematics –Much of the effect of the MMP is not under the direct control of the MMP –There are uncontrollable variables at the school level that must be taken into consideration Finding an appropriate statistical model –Status-based models –Value-added Assessment (VAA) models

4. Status-Based Growth Models Rely upon cohort-to-cohort comparison to assess improvement in achievement Compare unadjusted average student performance on standardized tests across successive cohorts Inherent shortcomings –Based on a non-equivalent group case study design –Do not take into consideration pre-existing differences between students

5. VAA Growth Models Consider the growth of students from one year to the next through repeated measures analyses Use longitudinal data to compare a students’ progress to his/her past academic achievements Provides some control over the individual differences between students because students serve as their own controls

6. Purposes 1)To determine if reported differences in how the model of professional development is being implemented by the MMP are related to differences in student achievement gains in mathematics 2)To compare and contrast status-based and VAA models in terms of assessing the impact of the effectiveness of the MMP 3)To address issues specific to the MMP that have been encountered in attempting to implement a system of accountability

7. Methodology – Participants Schools –92 of the 143 (64%) elementary & middle schools that participated in the MMP during its 1 st year were considered in this study. Students – 3 rd – 8 th graders who participated in the district’s standardized testing –VAA Only students who had test scores for both the 2003-4 and 2004-5 school years, did not change schools, and did not repeat/skip a grade could be considered 14,259 total students –Status-based 27,117 students for the 2003-4 school year 27,079 students for the 2004-5 school year

8. Methodology - Instrumentation Self-report instrument administered to MTLs 1)How often MTLs meet with their Math Specialist and others in their building; 2)How satisfied MTLs feel with the amount of time they have to discuss issues centered around mathematics with their Math Specialist and others in their building; 3)How supported MTLs feel in their role as a MTL by their Math Specialist and others in their building; 4)The MTLs perceptions of the functioning of school’s Learning Team in terms of the general climate of the meetings as well as the overall cohesion in accomplishing goals related to their mathematics curriculum. MTL Tracking variables –Number of MMP meetings attended and whether or not a vision statement and/or action plan was turned in

9. Methodology – Instrumentation Mathematics scale scores from the November administration of the Terra Nova (grades 3, 5-7) and WKCE (grades 4 and 8) for 2003-4 and 2004-5. –Both tests published by CTB-McGraw Hill –Mathematics scale scores from Terra Nova and WKCE shown to be highly correlated by CTB and therefore, according to the test publishers, it is appropriate to directly compare the results of the WKCE and Terra Nova

10. DV = school level difference score representing growth in student mathematics achievement Weighted average of individual difference scores was calculated to obtain an overall measure of growth for each school Weighted average used because expected growth for each student is not equal to zero (  (D j-i ) ≠ 0 ) I = the lowest tested grade level served by a particular school j = the highest grade level served by a particular school n(j - i) = the # of students with scale scores for both grades i and j = the average of individual difference scores for students from grade j to grade I, and N = total number of students used to calculate the overall growth Dependent Variables - VAA

11. Dependent Variables – Status based DV = school level difference score representing growth in student mathematics achievement The difference in average scale scores for each grade between the 2004-5 and 2003-4 school years was calculated and then the average difference across all grade levels was calculated for an overall measure of growth for each school Simple average used because the expected difference for each grade level is zero (  (D 2-1 ) ≠ 0 )

12. Does the degree in which schools’ learning teams embrace the vision of the MMP influence student growth in mathematics achievement test scores? –Stepwise regression analysis used with five independent variables with four interaction terms Measure of the frequency in which the MTL met with the LT and had useful discussions about how to help students succeed in mathematics Climate of the LTLT collaboration Vision Plan submissionAction Plan Submission –Analyses exploratory in nature Evaluation Question #1

13. Evaluation Question #2 Does the level of satisfaction with time spent discussing mathematics, as reported by MTLs, and frequency of MTL attendance at district-wide MMP meetings influence student growth in mathematics achievement test scores? –Stepwise regression analyses used with two independent variables with one interaction term Number of district-wide MMP meetings attended by the MTL Satisfaction the MTL felt with the amount of time spent discussing mathematics with others in their school The interaction of the above two variables –Analysis exploratory in nature

14. Evaluation Questions #3 and #4 3)Is there a difference in student growth in mathematics for schools for which MTLs report feeling a greater ability to accomplish goals regarding mathematics? 4)Is there a difference in student growth in mathematics for schools in which MTLs feel more supported by other math teachers in their school? Each used a one-way ANOVA with three levels on the DV

15. Results Very little statistically significant results were found –This is not surprising given that the analyses conducted were based only on approximately six months of MMP activity Evaluation Question #1 – degree in which LTs embrace vision of the MMP –VAA – no IVs were significant enough to be entered into the regression model –Status based – LT collaboration found to be a significant predictor of student growth in math (F (1,88) = 5.06, p=.027,  = -2.23, R 2 =.05)

16. Results (con’t) Evaluation Question #2 – MTL level of satisfaction with time spent math and frequency of MTL attendance at MMP meetings –VAA and Status based – MTL attendance at district-wide MMP meetings found to be a statistically significant predicator of overall school level growth in math VAA: F (1,88) = 4.17, p=.044,  =.98, R 2 =.05 Status based: F (1,90) = 6.37, p=.011,  = 1.31, R 2 =.07 –Encouraging that attendance seems to play a role in increasing student achievement

17. Results (con’t) Evaluation Questions #3 and #4 – MTLs feeling of ability to accomplish goals and perception of support from other math teachers –No statistically significant results were found for either evaluation question for either accountability system

18. Discussion Statistical analyses did not detect many relationships between survey responses (IVs) and overall school level growth in mathematics –Not surprising given that the data collected from the survey was obtained only six months after the inception of MMP activities –Encouraging that a small positive effect on overall growth in student achievement in mathematics was found in terms of MTL attendance at MMP meetings

19. Discussion Differences found between the models highlight major flaws of using a status based growth model –Namely, the status based model utilizes different cohorts of students to ascertain a measure of change in achievement

20. Discussion Data exemplifying the major flaw of status based growth models Means and SD of Difference Scores Obtained from VAA and Status Based Accountability Systems VAAMeanSDStatus BasedMeanSD Overall16.2728.30Overall-2.557.39 4 th - 3 rd Gr23.2933.303 rd Gr-3.0514.77 5 th - 4 th Gr8.3726.554 th Gr-5.1010.77 6 th - 5 th Gr11.6926.385 th Gr-5.5911.52 7 th - 6 th Gr15.1726.016 th Gr0.379.83 8 th - 7 th Gr20.6925.587 th Gr-3.6612.03 8 th Gr7.9111.79

21. Discussion Support for VAA models –Use of longitudinal data to estimate the impact of large-scale educational reform on student achievement Compares student’s progress to his/her past academic achievement rather than to a generic cohort of students Offers a picture of the rate at which a student is learning By tracking individual student’s progress from one year to the next, VAA models provide a control for pre-existing student differences

22. Discussion Practical disadvantages of VAA models –Require an extensive database of longitudinal data Student achievement scores Student characteristic variables Means of linking students to educational reform activities –Requires reliable and valid methods of quantifying educational reform endeavors and outcomes

23. Discussion Theoretical concerns with using VAA –Model specification –Viability of using the model to claim causal effects of educational reform on student learning –Inclusion of covariates and possible confounding variables –Use of achievement test data as the primary outcome measure

24. Discussion Theoretical concerns (con’t) –Effect of having missing data that is NOT completely missing at random (CMAR) CMAR data – there are not discernable features that characterize the students who have missed a test from those who have not However, students who are more likely to miss tests often have certain characteristics that distinguish them from the population of students who take the tests Data that is NOT missing at random cannot be ignored because it introduces bias into the estimates of school-level effects

25. Discussion Theoretical concerns (con’t) –How best to take into account the time at which student achievement tests are administered Unclear how to best model growth in achievement so that the growth is accurately distributed among the schools responsible for that growth Especially salient when school level variables are collected at a different point in time as when students are tested

26. Discussion Theoretical concerns (con’t) –How best to take into account the time at which student achievement tests are administered is a major issue in trying to model how variability at the school level, which is related to the efforts of MMP, is having an impact on increasing student achievement in mathematics.  MMP CONUNDRUM: While the efforts of the MMP start in September and end in August, the school year runs from September to June, students are tested in November, and data is collected from teachers, learning team members, and MTLs in June.

27. Discussion Theoretical concerns (con’t) –Using growth as a DV implies the scale of measurement is constant across years Vertical scaling assumes each test is measuring a common trait so that the scales have identical meanings However, as students progress from one grade to the next, their curriculum progresses in content and difficulty and so tests designed for different grade levels may not allow for a measure of true growth –Standardized achievement tests are subject to measurement error and the amount of error varies depending on student’s level of achievement

28. Discussion Problems encountered in MMP evaluation –Currently only school level variables can be utilized –Lack of means to link teacher responses to student achievement Issues associated with teacher confidentiality –Lack of participation in data collection efforts Likely that modeling classroom effects would be statistically biased due to the magnitude of missing responses

29. Future Direction Conduct more refined analyses using data collected at the end of year two –On-line survey made available to all MPS participants in the MMP that may have an effect on increasing student achievement in mathematics including Learning Team Members Literacy Coaches Math Teacher Leaders Math Teachers

30. Future Direction Data collected at the end of year two –Six primary areas were evaluated 1.Professional Development 2.School/District Functioning 3.Learning Team Functioning 4.Classroom Practice 5.Role of the MTL 6.Role of the LC

31. Future Direction Data collected at the end of year two –Professional Development (PD)– the extent to which those surveyed participated in, and were influenced by, PD activities supported by the MMP Both formal and informal professional development activities were considered PD items were administered to all participants

32. Future Direction Data collected at the end of year two –School/District Functioning The extent to which those surveyed feel involved in helping shape their school’s mathematics program Participant’s perception regarding the degree of alignment between the school and district goals Participant’s perception regarding the level of support provided by the Math Teaching Specialists and Learning Targets Items were administered to all participants

33. Future Direction Data collected at the end of year two –Learning Team Functioning The extent to which the Learning Team is functioning as a cohesive unit The degree to which the Learning Team is emphasizing school improvement efforts in mathematics in their work Items administered to all MTLs and all Learning Team members

34. Future Direction Data collected at the end of year two –Classroom Practice The extent to which math teachers engage their students in interactive vs. didactic instruction The amount of time math teachers devoted to each of the learning targets in MPS Items administered to all participants that reported teaching math in the classroom

35. Future Direction Data collected at the end of year two –Role of Math Teacher Leader (MTL) The extent to which MTLs met with others in their school to discuss the teaching and learning of mathematics The level of support MTLs felt was provided to them by others in their school and the MTSs Items administered only to MTLs –Role of the Literacy Coach (LC) The extent to which the LC collaborated with and supported the MTL at their school Items only administered to the LCs