Symposium: Assessment, Accountability, Instruction, and Learning in Urban Districts Research funded by the Joyce Foundation and Helen Bader Foundation Center for Systemic Reform in the Milwaukee Public Schools (SSR-MPS) Wisconsin Center for Education Research (WCER) University of Wisconsin-Madison 3 papers: - William Clune, et al.Milwaukee Middle School Proficiencies - Robert MeyerValue-added & School Performance - Norman WebbAssessment Literacy All focus on Milwaukee district 3 Commentators: - Warren ChapmanJoyce Foundation - Deborah LindseyMilwaukee Public Schools - Andrew PorterWCER Papers on website:

The Milwaukee Middle School Proficiencies: Systemic school reform through high stakes assessments and a network of schools William H. Clune, with Sarah Mason, Cecilia Pohs, Chris Thiel, and Paula A. White Paper prepared for the annual meeting of the AERA, New Orleans, April 2, 2002 Paper on website:

Overview of Points What are the Proficiencies? Research Methods Impact on instruction and achievement Centralized/ decentralized policy & implementation Value of standardized and performance assessments Importance of evaluation

The Milwaukee Middle School Proficiencies: What are they? Promotion requirements from 8 th grade, year through Areas: Communications, Mathematics, Science, Research Paper Multiple assessments, multiple opportunities to pass, grades 6-8 Purpose: assure readiness for high school (not just pass or fail)

Assessment Types Traditional standardized tests (e.g., State of Wisconsin) On-demand district performance assessments Performance assessments embedded in instruction (tasks, scoring rubrics) Most weight put on embedded assessments Can pass w/o "proficient" on State test Alternative completion project (all 4 proficiencies) (Adopt-a-City)

Research Methods 17 interviews in Spring school sites (range of reputed success) 9 interviews with learning coordinators, 3 principals 5 with respondents from school network ("Middle School Collaborative") Taped, transcribed, coded with NUD*IST 4 Names & schools not disclosed in paper

Findings Strong impact on instruction, learning, school organization Unclear impact on student achievement, weak positive evidence Centralized/ decentralized implementation/ policy formation

Instruction Provided a focus for teaching(9 respondents) Students re-do work w teachers(4) Increased hands-on work(4) Aligned with curriculum(9) Learning Students: - take greater responsibility(6) - improved in writing(5) - improved in reading(3) - improved in math/ science(6) - improved on standardized tests(2) School organization Schools did major re-organization(4) Special proficiency classes helpful(4) Summer school programs helpful(3) Finding 1: Strong Impact on Instruction, Learning, School Organization

Finding 2: Unclear Impact on Student Achievement, weak positive evidence Intended to improve achievement (proficiency beyond assessments) Performance assessments not statistically reliable Annual 8th grade testing poor method of evaluation Wisconsin 8th grade scores rose for two years then fell Technical problems: change of test date, test forms District study: increase in high school grades and 9th grade promotion Also more transitional ("8-T") students in grades 8-9 Much better: value-added from annual standardized tests (Meyer paper)

Finding 3: Centralized/ decentralized implementation/ policy formation In general Unfolding requirements and implementation (incremental "roll out") Much formative activity by District staff, Middle School Collaborative, Learning Coordinators, Lead Principals The Middle School Collaborative (network of Middle School Principals) Had independent funding (Danforth) Summer retreat for guiding vision ("all children can succeed") Prevented repeal of Proficiencies Successfully advocated alternative completion mechanism (Adopt-a-City) Advocated fewer midstream policy changes

Discussion/ Significance of Findings Policy strength from the top and bottom Tradeoff of measurement reliability and instructional validity

Discussion point 1: Policy strength from the top and bottom Good fit with Porter et al (1988) framework (authority, power, consistency, specificity) Importance of infrastructure in systemic reform (e.g., Clune, 2001) At the top - Authority (School Board, broad support for performance assessments) - Power (promotion for students, high resources) - Specificity (clear expectations for students) - Consistency (coherent design across subjects, grades) From the bottom - Authority (Principals in Collaborative) - Power (huge voluntary resources for implementation) - Specificity (many details worked out between and w/i schools) - Consistency (Vision and details managed decentrally)

Discussion point 2: Tradeoff Of Measurement Reliability And Instructional Validity Traditional standardized tests: high reliability, low instructional validity District performance assessments: low reliability, high instructional validity Standardized performance assessments rejected as too expensive Two good options for district: - Combination of both, but move to annual standardized testing for evaluation - Revisit standardized performance assessments

Conclusion: Importance of Evaluation Important for both successful and unsuccessful programs To discontinue, continue, refine Proficiencies used enormous resources (mostly labor) Intensive resources = political vulnerability = need for good evaluation