Establishing MME and MEAP Cut Scores Consistent with College and Career Readiness A study conducted by the Michigan Department of Education (MDE) and ACT, Inc.

Overview of MEAP and MME MEAP (Michigan Educational Assessment Program) MME (Michigan Merit Examination)

Why Reset Cut Scores Now? Disjoint cut scores  Relative difficulty of cut scores increases by grade in grades 3-8  Relative difficulty spikes in high school (grade 11) Relatively low cut scores  Based on standard setting panelists’ interpretation of “basic skills” New economy  Requires a redefinition of “basic skills”  Need students to be prepared for technical career training or college State Board of Education History  Adopted new, rigorous high school content standards in 2006  Adopted new high school exam in 2006  Did not want to adopt college ready cut scores at that time (wanted to give time for schools to implement the new content standards)  Requested that in five years time the issue be revisited to raise cut scores

Previous Impact Data

Purpose of the study Set new cut scores representing being on track to career and college readiness in the grades and subjects in red Not needed in writing, since the writing standards were originally set to be consistent with ACT’s college readiness benchmarks Need three cut scores to create four performance levels  Advanced  Proficient  Partially Proficient  Not Proficient (no cut score needed)

Standard Setting Reasoned Judgment  Subjective (panelist judgment)  With objective elements (e.g., ordered item booklets) Empirical  Objective External criterion (grade in related freshman course) was not created for the purpose of standard setting  With subjective elements External criterion (freshman college course grade) is itself a subjective judgment on the part of the college professor Choice  Empirical  Meaning

Two types of linkages were needed to identify the cut scores: Link 11th grade MME scores to freshman college grades to identify cut scores on the MME. Link MEAP scores to MME scores to identify cut scores on one or more grades of the MEAP.

Step 1: The MME To do list  Define what is meant by college success  Collect data on students with MME scores and college outcomes  Decide on a method that links MME scores and college success

Measure of college success First year courses Course grade- either average grade, or specific grade or better Examples:  average grade of 3.0  Proportion of students with a grade of B or better

The MME - Data MME Test data available with MI student ID CEPI data available with MI student ID This is a very good thing

The MME data College grade data taken from Fall 2009 and Fall 2010 Course grade data from 39 public colleges  19 4-year institutions  20 2-year institutions Each college was given the opportunity to define their “first year courses” in each course area

The MME - data Number of courses varied by subject Mathematics 1 course College Algebra Science 4 courses Biology/Life Science General Chemistry Physics Other

The MME – College Grade Data Total sample sizes:  Reading40,164  Mathematics 6,567  Science 16,180  Social Science 41,780 >100,000 records

The MME - Methods Signal Detection Theory (SDT) Logistic Regression (LR) Conditional mean Very Similar

The SDT Method College grade Cut score (To be determined) Successful/ Successful Not Successful/ Not Successful Not Successful/ Successful Successful/ Not Successful

The Logistic Method 0.5 Cut score

The MME To get a cutscore, we must choose a success criterion and a method  B or higher grade  SDT method

MME – the results SubjectProficient Cut Score Past Proficient Cut Score Mathematics Reading Science Social Science

Step 2: MEAP to MME What does it mean to say that a 5 th grade student is proficient? General answer: A student is CCR if his/her score in grade 5 indicates that the student is on track to be college ready in grade 11

College Readiness in Grade 5 College ReadyNot College Ready

What link to make? Option 1 – Link Grade 8 to Grade 11, Grade 7 to Grade 8, etc. Option 2 – Link Grade 8 to Grade 11, Grade 7 to Grade 11, etc Advantage: Less Measurement Error Advantage: Larger Sample Size

Cohort with at Least Two Years of Data Cohort GradeSample Size a 1112 N_ Total b N_Full_ Matched c ,468107, ,896105, ,557101, ,26898, ,72994, ,95093, ,04798, ,86988, ,24693, ,81397, ,255104,352

Fewer links – smaller(?) sample MME scores Grade 11 MEAP scores Grade 7 Grade 8 Grade 9 MEAP scores Grade 3 Grade 4 Grade 5 Grade 6

MEAP – the results (mathematics) GradeNew Cut ScoreOld Cut Score

Validity checks Consistency of classification across years  Should be high, but not too high Consistency of classification with different tests (like the ACT)  Note that MME contains items from the ACT as one of its constituent parts for Reading, Math, and Science

Classification Consistency Rates Year to Year Mathematics 11-grade-65% %86%95% 7-881%84%95% 6-782%83%96% 5-681%84%95% 4-580%82%94% 3-477%80%95% Grade Partially Proficient Advanced

Consistency of MME with ACT ACT Math Meeting College and Career Readiness Benchmarks YesNo MME Math Proficient Yes 30, % 1, % No 7, % 81, % 93%

Consistency of MME with ACT SubjectConsistency Rate More difficult Mathematics93%MME Reading86%ACT Science90%ACT

Consequences – Reading (Student Level)

Consequences – Reading (Elementary and Middle School Level)

Consequences – Reading (High School Level)

Consequences – Mathematics (Student Level)

Consequences – Mathematics (Elementary and Middle School Level)

Consequences – Mathematics (High School Level)

Message Michigan schools are not adequately preparing students for freshman-level college/career-training courses Michigan schools are better preparing students for entry-level reading-heavy (humanities, social science) courses than for entry-level mathematics courses

Message Florida recently attempted to do this in writing, but the increase in cut scores was rejected We believe Michigan was able to successfully make the transition to more rigorous cut scores because of advance notice to the field, the press, and the state board of education regarding the anticipated impact and a strong argument for the need to revisit cut scores

Thank you for your attention Any Questions or Comments?