1. Establishing MME and MEAP Cut Scores Consistent with College and Career Readiness A study conducted by the Michigan Department of Education (MDE) and ACT, Inc.

2. Overview of MEAP and MME MEAP (Michigan Educational Assessment Program) MME (Michigan Merit Examination)

3. Why Reset Cut Scores Now? Disjoint cut scores  Relative difficulty of cut scores increases by grade in grades 3-8  Relative difficulty spikes in high school (grade 11) Relatively low cut scores  Based on standard setting panelists’ interpretation of “basic skills” New economy  Requires a redefinition of “basic skills”  Need students to be prepared for technical career training or college State Board of Education History  Adopted new, rigorous high school content standards in 2006  Adopted new high school exam in 2006  Did not want to adopt college ready cut scores at that time (wanted to give time for schools to implement the new content standards)  Requested that in five years time the issue be revisited to raise cut scores

4. Previous Impact Data

5. Purpose of the study Set new cut scores representing being on track to career and college readiness in the grades and subjects in red Not needed in writing, since the writing standards were originally set to be consistent with ACT’s college readiness benchmarks Need three cut scores to create four performance levels  Advanced  Proficient  Partially Proficient  Not Proficient (no cut score needed)

6. Standard Setting Reasoned Judgment  Subjective (panelist judgment)  With objective elements (e.g., ordered item booklets) Empirical  Objective External criterion (grade in related freshman course) was not created for the purpose of standard setting  With subjective elements External criterion (freshman college course grade) is itself a subjective judgment on the part of the college professor Choice  Empirical  Meaning

7. Two types of linkages were needed to identify the cut scores: Link 11th grade MME scores to freshman college grades to identify cut scores on the MME. Link MEAP scores to MME scores to identify cut scores on one or more grades of the MEAP.

8. Step 1: The MME To do list  Define what is meant by college success  Collect data on students with MME scores and college outcomes  Decide on a method that links MME scores and college success

9. Measure of college success First year courses Course grade- either average grade, or specific grade or better Examples:  average grade of 3.0  Proportion of students with a grade of B or better

10. The MME - Data MME Test data available with MI student ID CEPI data available with MI student ID This is a very good thing

11. The MME data College grade data taken from Fall 2009 and Fall 2010 Course grade data from 39 public colleges  19 4-year institutions  20 2-year institutions Each college was given the opportunity to define their “first year courses” in each course area

12. The MME - data Number of courses varied by subject Mathematics 1 course College Algebra Science 4 courses Biology/Life Science General Chemistry Physics Other

13. The MME – College Grade Data Total sample sizes:  Reading40,164  Mathematics 6,567  Science 16,180  Social Science 41,780 >100,000 records

14. The MME - Methods Signal Detection Theory (SDT) Logistic Regression (LR) Conditional mean Very Similar

15. The SDT Method College grade Cut score (To be determined) Successful/ Successful Not Successful/ Not Successful Not Successful/ Successful Successful/ Not Successful

16. The Logistic Method 0.5 Cut score

17. The MME To get a cutscore, we must choose a success criterion and a method  B or higher grade  SDT method

18. MME – the results SubjectProficient Cut Score Past Proficient Cut Score Mathematics11161100 Reading11081100 Science11261100 Social Science11291100

19. Step 2: MEAP to MME What does it mean to say that a 5 th grade student is proficient? General answer: A student is CCR if his/her score in grade 5 indicates that the student is on track to be college ready in grade 11

20. College Readiness in Grade 5 College ReadyNot College Ready

21. What link to make? Option 1 – Link Grade 8 to Grade 11, Grade 7 to Grade 8, etc. Option 2 – Link Grade 8 to Grade 11, Grade 7 to Grade 11, etc Advantage: Less Measurement Error Advantage: Larger Sample Size

22. Cohort with at Least Two Years of Data Cohort GradeSample Size 345678910 a 1112 N_ Total b N_Full_ Matched c 109-1010-11--------118,468107,844 208-0909-1010-11-------123,896105,019 307-0808-0909-1010-11------128,557101,277 406-0707-0808-0909-1010-11-----135,26898,131 505-0606-0707-0808-0909-1010-11----140,72994,044 6-05-0606-0707-0808-0909-1010-11---149,95093,447 7--05-0606-0707-0808-0909-1010-11--151,04798,155 8---05-0606-0707-0808-0909-1010-11-161,86988,526 9----05-0606-0707-0808-0909-1010-11163,24693,004 10-----05-0606-0707-0808-0909-10160,81397,172 11------05-0606-0707-0808-09158,255104,352

23. Fewer links – smaller(?) sample MME scores Grade 11 MEAP scores Grade 7 Grade 8 Grade 9 MEAP scores Grade 3 Grade 4 Grade 5 Grade 6

24. MEAP – the results (mathematics) GradeNew Cut ScoreOld Cut Score 8830800 7731700 6629600 5531500 4434400 3338300

25. Validity checks Consistency of classification across years  Should be high, but not too high Consistency of classification with different tests (like the ACT)  Note that MME contains items from the ACT as one of its constituent parts for Reading, Math, and Science

26. Classification Consistency Rates Year to Year Mathematics 11-grade-65%- 8-1183%86%95% 7-881%84%95% 6-782%83%96% 5-681%84%95% 4-580%82%94% 3-477%80%95% Grade Partially Proficient Advanced

27. Consistency of MME with ACT ACT Math Meeting College and Career Readiness Benchmarks YesNo MME Math Proficient Yes 30,534 25.5% 1,241 1.0% No 7,004 5.8% 81,041 67.6% 93%

28. Consistency of MME with ACT SubjectConsistency Rate More difficult Mathematics93%MME Reading86%ACT Science90%ACT

29. Consequences – Reading (Student Level)

30. Consequences – Reading (Elementary and Middle School Level)

31. Consequences – Reading (High School Level)

32. Consequences – Mathematics (Student Level)

33. Consequences – Mathematics (Elementary and Middle School Level)

34. Consequences – Mathematics (High School Level)

35. Message Michigan schools are not adequately preparing students for freshman-level college/career-training courses Michigan schools are better preparing students for entry-level reading-heavy (humanities, social science) courses than for entry-level mathematics courses

36. Message Florida recently attempted to do this in writing, but the increase in cut scores was rejected We believe Michigan was able to successfully make the transition to more rigorous cut scores because of advance notice to the field, the press, and the state board of education regarding the anticipated impact and a strong argument for the need to revisit cut scores

37. Thank you for your attention Any Questions or Comments?