Presentation to the Michigan State Board of Education September 13, 2011

2  ACT Research & Development Unit  National Center for Educational Achievement  Michigan Technical Advisory Committee  MDE/BAA Measurement Research & Psychometrics Unit

3  Identify a score for Proficient on the MME that represents being on track for career and college  Identify a score for Proficient on the highest grade level of MEAP that represents being on track to high school success  Identify a score for Proficient on the lower grade levels of MEAP that represent being on track to success in the next higher grade  Identify a score in each grade level of MEAP or MME that represents attainment of Partial Proficiency  Identify a score in each grade level of MEAP or MME that represents attainment of Advanced skills

4  Assumed that if a student is on track to success in college, then the student is also on track to success in technical career training  Therefore, focus on success in college  Included 2-year college programs (including job training programs at 2-year institutions)  Success could mean many things:  A, B, or C in college?  In a 2-year or 4-year college?  Conducted analyses of all of these scenarios  Conducted analyses only of academic success, not of every factor that leads students to be successful in college

5  2-year versus 4-year colleges  Separate analyses were run regarding college success in 2-year and 4-year institutions  The cut scores identified for 2 year versus 4 year institutions were within measurement error of each other  Therefore, all final analyses combined all students from 2-year and 4-year colleges into a single group

6  Separate analyses were run using students achieving an A versus B versus C in their first credit-bearing freshman courses  A and C analyses did not produce usable results  Therefore, all final analyses used the criterion of B or better as the measure of college success  This is also the criterion for success used by ACT in its college readiness benchmarking study

7  Three types of analyses conducted  Logistic Regression (LR)  Signal Detection Theory (SDT)  Equipercentile Cohort Matching (ECM)

8 Started with data from Michigan Public Institutions of Higher Education  Identified appropriate credit-bearing freshman courses against which to analyze the relationship between MME scores and course grades  Thanks to 2- and 4-year institutions for providing the data  Thanks to President’s Council for providing listings of courses appropriate to tie to MME scores Using those data, conducted SDT analyses to connect MME to college freshman grades

9 Used SDT to map backward from 11 th grade MME to 7 th and 8 th grade MEAP  Identified the score on MEAP that would maximize the consistent classifications from MEAP to MME  Used SDT to map from 7 th grade MEAP to all other MEAP grades  Identified the score on each MEAP grade that would maximize consistent classifications from grade to grade

10  Recommend retaining the labels for the four performance categories  Not Proficient(Considered “Off Track”)  Partially Proficient(Considered “Off Track”)  Proficient(Considered “On Track”)  Advanced(Considered “On Track”)

11  SDT can also identify scores on the MME scale that give certain probabilities of obtaining a B or better in the first credit bearing freshman course  Identified two probabilities that have strong meaning and give cut scores sufficiently far from the MME Proficient (On Track) cut scores.  1/3 probability of B or better (Partially Proficient cut score)  2/3 probability of B or better (Advanced cut score)  These cut scores were identified using SDT for the MME  These cut scores were also mapped back to the MEAP using ECM

12 Grade Partially ProficientProficientAdvanced

13

14

15

16 Grade Partially ProficientProficientAdvanced

17

18

19

20 Grade Partially ProficientProficientAdvanced

21

22

23

24 Grade Partially ProficientProficientAdvanced

25

26

27

28  MDE staff will apply the cut scores retroactively as if the new cut scores had been in place  Applied to four years of data  Within one month of approval  No retroactive accountability implications for schools, just to allow for following trends over time

29  Joseph Martineau  Executive Director  Bureau of Assessment & Accountability  Michigan Department of Education 