Evaluating the Vermont Mathematics Initiative (VMI) in a Value Added Context H. ‘Bud’ Meyers, Ph.D. College of Education and Social Services University of Vermont Douglas Harris, Ph.D. The Evaluation Center The Vermont Institutes, Montpelier, VT

Purpose Describe the context, purpose, structure, staffing and content of the instructional program for teachers Describe the logic model that provides the framework for evaluation of the program Examine the methodology for assessing the value added to student learning by teacher preparation in mathematics Present findings from cross-sectional analyses of eight cohorts of students who have been taught by VMI teachers and four cohorts of control students. Present findings from a qualitative evaluation of the process of implementation of the components of the program from the perspectives of participants and administrators Present recommendations arising from the findings of the evaluation

Context, purpose, structure, staffing A field based masters degree in Education with a primary emphasis on mathematics content Partially funded by local districts, partially MSP Focus on Grades k-8 Includes focus on teacher leadership Includes action research Three years Mathematicians and educators

Content-12 Courses Course 1: Mathematics as a Second Language (3 credits) Course 2: Functions and Algebra for Elementary Teachers (3 credits) Course 3: Trigonometry for Elementary Teachers (2 credits) and Algebra and Geometry II (1 credit) Course 4: Measurement, Geometry, and Probability for Elementary Teachers (3 credits) Course 5: Number Theory for Elementary Teachers (3 credits)

Courses…contd. Course 6: Statistics, Action Research, and Inquiry into Effective Practice, I (3 credits) Course 7: Statistics, Action Research, and Inquiry into Effective Practice, II (3 credits) Course 8: Algebra and Geometry for Elementary Teachers, III (3 credits) Course 9: Statistics, Action Research, and Inquiry into Effective Practice, III (3 credits) Course 10: Calculus for Elementary Teachers, I (3 credits)

Courses…contd. Course 11: Calculus for Elementary Teachers, II (3 credits) Course 12: Capstone VMI Experience (3 credits)

Methods Quantitative- Cross-sectional and longitudinal Two intervention groups Group 1=16 schools, multiple VMI teachers (HiEnd) Group 2=22 schools, single teacher per school (RegIntervention) Control group=22 schools, matched to Groups 1 and 2 (combined) on demographics Longitudinal comparisons for 2 cohorts, 1999 and 2000, grades 4 to 8. (See Figure 2)

Methods Qualitative Interviews Participants Graduates Administrators Observations Review of course materials and participant portfolios Course evaluations Interviews of School Staff and Leadership

Sample Characteristics Figure 1

Current and Potential Trend Studies with 3 data points for VT Data Others are cross-sectional, independent group comparisons YearGr 4Gr 5Gr 6Gr 7Gr 8Gr 9Gr x 2000x 2001x x 2004xx 2005x xx 2006 x x = Comparisons Longitudinal comparisons Figure 2

Question 1: What do comparisons with comparable schools show, over time, when students are matched from grade 4 through grade 8 and grade 10 during the two cohort groups ( , and )? Significant differences in t=3.22, p<.001, df 1, Significant differences in t = 5.27, p<.001, df 1, 1488 Significant differences in t = 3.39, p<.001, df 1, 1107 Figure 3 Figure 4

Discussion As indicated in Figures 3 and 4, each of the VMI cohorts outperforms the Control schools in similar patterns of difference. Results for comparisons at the first data point for each cohort (1999 and 2000) are not significantly different while they emerge as significant in 2005 and 2006.

Significant differences in adjusted for differences in 1999, HiEnd v Reg and Control, F = 7.96, df 2, 1235, p <.001 Question 2: How do levels of implementation of the VMI curriculum in classrooms relate to levels of achievement of students who experience them? (1999 Cohort) Figure 5

Significant differences in 2006 adjusted for differences in 2000, HiEnd v Reg and Control, F = 9.91, df 2, 1093, p <.001 Note: The average standard deviation for groups is about 40 points across all groups. Mean differences between groups is thus approximately.25 S.D. between HiEnd and Control groups. Question 2: How do levels of implementation of the VMI curriculum in classrooms relate to levels of achievement of students who experience them? (2000 Cohort) Figure 6

Discussion: VMI HiEnd Schools significantly out-performed the matched schools and appear to carry most of the difference between the VMI and matched schools in both cohorts from 1999 through Analyses of covariance adjusted the score differences at the first two data points in each cohort. The final data point for each cohort represents group differences at grade 10.

Question 3. Are school level effects related to income level of students and levels of content implementation of the VMI curriculum? (1999 Cohort) Do gaps close? Figure 7

Question 3. Are school level effects related to income level of students and levels of content implementation of the VMI curriculum? (2000 Cohort) Do gaps close? Figure 8

Discussion:

Quantitative Summary Longitudinal comparisons of VMI and Control schools indicate a persistent pattern of advantage for VMI schools Longitudinal comparisons indicate that the VMI schools, with more VMI teachers, had a significantly greater long term gain on percentile ranks. Stronger teachers may not have as great an effect in the shorter term when isolated. Longitudinal comparisons suggest that SES differences are reduced over time when high quality mathematics instruction is offered to all students

Qualitative Findings (2005 and 2006) Impact on Participants and Their Teaching Finding 1: Mathematics Content An overarching theme is the impact of the VMI experience on the teacher’s own understanding of mathematics content. This is consistent among participants in all three years of the VMI experience and remains the highest priority of graduates as they reflect on the VMI experience Finding 2: Increased Confidence Related to Mathematics Increased understanding of mathematics content impacts the confidence of participants as related to mathematics, to teaching mathematics, and to their enthusiasm for and enjoyment of mathematics. Finding 3: Impact on Instructional and Assessment Practice Participants graduates (many of who are math teacher leaders), and principals report that the instruction in VMI, increased content knowledge, and increased confidence have had major impact on instructional and assessment practice in the classrooms of participants and of teachers impacted by VMI-trained math teacher leaders Finding 4: The Impact of Action Research on Classroom Practice The impact of action research is mixed, with some participants, graduates, and principals reporting considerable impact, others less so. Finding 5: Principal Support The active support of principals makes a profound difference in VMI participants’ work in the classroom and in leadership positions.

More Qualitative Findings Finding 6: Personal Impact on Participants Beyond the impact of VMI on teachers in relation to math content, instruction, and leadership, the program also has profound personal impact on many participants. Impact on Students Finding 7: Transfer of VMI Content to the Classroom Teachers report direct transfer of mathematics content used in VMI to the math experiences of their students. Finding 8: Impact on Student Problem Solving Teachers and teacher leaders report that the problem solving emphasis in VMI has significant impact on their understanding of how to engage students in problem solving in the classroom. Finding 9: Impact of Action Research on Students Some teachers believe that the interventions begun in their action research projects will continue to impact their students over time. Impact on Teacher Leadership in Schools and Districts Finding 10: Impact on Teacher Leaders Teachers working as teacher leaders credit VMI for providing the knowledge, confidence, and support for them to take on leadership roles. This is true of teachers working in leadership at the school, district, and state levels. Finding 11: Impact on curriculum, instruction, and assessment Teacher leaders report that the knowledge base and contacts developed during VMI have significant impact on the development and implementation of mathematics curriculum, instruction, and assessment. Finding 12: Need for Ongoing Professional Community There is a common desire among VMI graduates to maintain the type of professional learning community afforded them through VMI.

Recommendations

Educational or Scientific Importance of the Study The National Research Council, (2004) US Department of Education (2004) and researchers at RAND (2004) have cited the need for better evaluations of teacher professional development programs. The combination of mixed method approaches with multiple strategies such as longitudinal studies of students acting as their own controls over time is the present attempt to respond to current concerns. In addition, intervention students in combination with matched groups at multiple intervals is a further attempt to provide additional confidence in results.

Want a copy of the paper or PowerPoint??