Bill Breisch and Ed O’Connor Midwest Instructional Leadership Council

 Nonprofit organization formed June 2010  Focus of the organization is “building and sustaining capacity for improving the achievement of all children”  Pursuing that mission by: ◦ Professional development conferences and institutes ◦ Embedded coaching and support ◦ Bridging research and practice Midwest Instructional Leadership Council2

 Ed O’Connor: Educational Consultant/Instructional Data Coach Midwest Instructional Leadership Council (miLc)  Trained as a School Psychologist  MS/PhD University of Wisconsin-Madison  15 Years in the Monona Grove School District  School Psychologist at all levels 4K-12  Director of Continuous Improvement and Assessment  Currently working with 15+ districts across MN/WI  Continuous School Improvement/RtI  Systems Analysis  Using Data to Improve Instruction

 Bill Breisch Recently retired after serving entire professional career (44 years) in the Monona Grove School District 15 years – Fifth and Sixth Grade Teacher 17 years – Elementary School Principal 12 years – Director of Instruction Served on the School Administrators Alliance Assessment Project Team that drafted recommendations regarding the development of new statewide student assessment Educational Consultant & Board of Directors - miLc

Participants in this session will:  be able to connect conceptually the ACT measures to the larger context of Continuous School Improvement (CSI).  become aware of (or review) the national correlations (grades 3-11) between Measures of Academic Progress (MAP) and WKCE/ACT.  have discussed the relevance of “summative assessments” for local school improvement efforts through ◦ examples of K-12 Benchmark Targets Developed from Local Data. ◦ application examples from WI school districts.

The Big Picture

 We are doing so much assessment there is no time for teaching.  We are being forced to “teach to the test” because there is too much emphasis on test results.  We are collecting so much assessment data, but I still don’t know what I am supposed to do with it.  I don’t know how all this assessment is helping us. 3/3/10(c) Midwest Instructional Leadership Council7

 “Highly Effective Schools” Research:  Reliable and Valid data collection procedures  Regular review of outcomes against benchmark targets  Routine reporting of results to stakeholders  Structures for using data to determine how to allocate resources  Structures for evaluating the relationship between action plans and outcomes observed. 3/3/10(c) Midwest Instructional Leadership Council8

 Quality assessment procedures are necessary but not sufficient to produce exceptional learning outcomes.  “You can’t fatten the pig by weighing it” But… “You can’t tell if you are approaching your destination if you don’t have any data” 3/3/10(c) Midwest Instructional Leadership Council9

 The fundamental purpose of assessment is to allow us to make decisions: ◦ Big Picture:  Are we achieving our system’s purpose?  What is that purpose? ◦ Intermediate Picture  What is working?  What is not working? ◦ Immediate Picture  What should I/we do? 3/3/10(c) Midwest Instructional Leadership Council10

MAP, Explore, Plan, ACT, and WKCE What are our targets? 3/3/10(c) Midwest Instructional Leadership Council11

David Conley, Director of the Center for Educational Policy and Research at the University of Oregon Broad Definition of CCR

Results of this ACT study provide empirical evidence that, whether planning to enter college or workforce training programs after graduation, high school students need to be educated to a comparable level of readiness in reading and mathematics. Graduates need this level of readiness if they are to succeed in college-level courses without remediation and to enter workforce training programs ready to learn job-specific skills.

 Identifying the level of reading and mathematics skills students need to be ready for entry-level jobs that require less than a bachelor’s degree, pay a wage sufficient so support a family, and offer the potential for career advancement.  Comparing student performance on ACT tests that measure workforce readiness (WorkKeys tests: Reading for Information and Applied Mathematics) with those that measure college readiness (ACT)  Determining if the levels of performance needed for college and workforce readiness are the same or different

WKCE “Proficient” and “Advanced” Scores, Measures of Academic Progress (MAP) RIT Scores, and National Percentiles (8 th Grade) WKCEMAP RIT ScoreNational Percentile Reading – “Old Proficient” th Reading – “New Proficient” th Reading – “Old Advanced” th Reading – “New Advanced” th Math – “Old Proficient” th Math – “New Proficient” st Math – “Old Advanced” nd Math – “New Advanced” th Source: Northwest Evaluation Association (May 2010 and August 2012). Wisconsin Linking Study: A Study of the Alignment of the NWEA RIT Scale with the Wisconsin Knowledge and Concepts Examination. Lake Oswego, OR: The Kingsbury Center at Northwest Evaluation Association.

 Active NWEA districts that use EXPLORE, PLAN, and ACT were recruited.  ACT data was matched to corresponding MAP data at the individual level  Total number of matched record pairs 108,000  No formal sampling strategies employed other than to cut extreme residuals

MAP Mathematics RIT Score as Predictor – Same Season Cut Scores and Normative Percentile Ranks on MAP Corresponding to College Readiness Benchmarks Grade Mathematics College Readiness Test Benchmark MAP Cut Score MAP Normative Percentile Rank 8EXPLORE Math PLAN Math ACT Math

MAP Reading RIT Score as Predictor – Same Season Cut Scores and Normative Percentile Ranks on MAP Corresponding to College Readiness Benchmarks Grade Reading College Readiness Test Bench mark MAP Cut Score MAP Normed Percentile Rank English College Readiness Test Bench mark MAP Cut Score MAP Normed Percentile Rank 8 EXPLORE Reading EXPLORE English PLAN Reading PLAN English ACT Reading ACT English

MAP Language Usage RIT Score as Predictor – Same Season Cut Scores and Normative Percentile Ranks on MAP Corresponding to College Readiness Benchmarks Grade Reading College Readiness Test Bench mark MAP Cut Score MAP Normed Percentile Rank English College Readiness Test Bench mark MAP Cut Score MAP Normed Percentile Rank 8 EXPLORE Reading EXPLORE English PLAN Reading PLAN English ACT Reading ACT English

The Big Picture

The BIG picture: To achieve exceptional student learning outcomes we must:  Employ continuous school improvement frameworks that organize and systematize our “acts of improvement” so that we can determine when we are making progress and when we are not.  Establish clear and measurable goals.  Review and discuss progress openly and often.  In other words we must… ◦ Use data to drive instruction (c) miLc 12/01/201024

What framework guides your thinking?

 System  System units ◦ Building ◦ Grade ◦ Classroom ◦ Department  Individual students

(c) miLc 12/01/201029

Summative Assessment Data-Aggregated Measures of Academic ProgressGrades Explore/Plan/ACT Minnesota Comprehensive Assessments Strengths Opportunities for Improvement Goals – District – Building – Grade – Student Root Cause Analysis: Why do the observed strengths and OFI’s (gaps) exist? Action Planning: If we do _________ we should sustain strengths and reduce gaps. Summative Assessment Data -Disaggregated Interim Assessment Data Identify and validate gaps at unit level. Suggest or validate contributing causes. Action plan implementation - Adequate training, resources and support along with fidelity monitoring. Frequent formative Assessment Interim Assessment: Indicates short-term effect and predicts impact on later summative assessment performance.

Tracking Student Progress Toward College and Career Readiness 3/3/10(c) Midwest Instructional Leadership Council31

 Small sample sizes ◦ Each grade group contains between 200 and 300 students  Observed probabilities ◦ The proportion of students meeting or exceeding the cut score that also exceed the predicted cut (e.g., exceeds MAP cut and also meets CCR Target on the Explore).  Receiver Operating Characteristics (ROC) Curve Analysis procedures ◦ Cut scores attempt to statistically maximize the number of students correctly identified as needing acceleration support (sensitivity) while at the same time maximizing the number of students correctly identified as “on-track” (specificity) ◦ In other words high sensitivity means there is a high probability that a “positive” test result is truly positive and high specificity indicates a high probability that a “negative” result is truly negative. ◦ We elect to prioritize specificity because we are most concerned about “missing” students who truly need additional attention. 3/3/10(c) Midwest Instructional Leadership Council32

3/3/10(c) Midwest Instructional Leadership Council34 True Positives True Negatives False Positives False Negatives

3/3/10(c) Midwest Instructional Leadership Council35

Contact Information:  Ed O’Connor(608) ◦  Bill Breisch(608) ◦