Presentation at the Fall 2011 Meeting of the Michigan Educational Research Association

 2 Identifying MME Cut Scores

3 University2-Year Institution Central Michigan UniversityAlpena Community CollegeMid Michigan Community College Eastern Michigan UniversityDelta CollegeMonroe County Community College Ferris State UniversityGlen Oaks Community CollegeMontcalm Community College Grand Valley State UniversityGogebic Community CollegeMott Community College Michigan Technological UniversityGrand Rapids Community CollegeMuskegon Community College Michigan State UniversityHenry Ford Community CollegeNorth Central Michigan College Oakland UniversityJackson Community CollegeNorthwestern Community College Northern Michigan UniversityKalamazoo Valley Community CollegeOakland Community College Saginaw Valley State UniversityKellogg Community CollegeSchoolcraft College The University of Michigan-Ann ArborKirtland Community CollegeSouthwestern Michigan College University of Michigan-DearbornLake Michigan CollegeSt. Clair County Community College University of Michigan-FlintLansing Community CollegeWashtenaw Community College Wayne State UniversityMacomb Community CollegeWest Shore Community College Western Michigan University

4 MME content areaCollege courses used MathematicsCollege Algebra. Reading Courses identified by 4-year universities. Reading-heavy courses such as entry-level literature, history, philosophy, or psychology for 2-year universities. Science Courses identified by 4-year universities. Entry level biology, chemistry, physics, or geology for 2-year universities. Social Studies Courses identified by 4-year universities. Entry level history, geography, or economics for 2-year universities.

5  Grades were put on a numeric scale from 0-4  0 = F  1 = D  2 = C  3 = B  4 = A  Not used o AU, AWF, DR, R, RA, FR, T, TR, X  Coded as 3.0 o P, CR  Coded as 0.0 o IN, N, NC, NE, NS, W, WF, WP, WX, and U

6 MME Content Area Sample Size Percent B or higher Course GradeMME Score MeanSDMeanSD Math6, Reading37, Social Studies 39, Science15,

7 MME Content AreaCourse TypeNumber of Students MathematicsCollege Algebra6567 Reading Literature456 American History1731 Other History3010 Psychology16231 Sociology8236 Political Science6114 Philosophy1869 Other2517

8 MME Subject AreaCourse TypeNumber of Students Science Biology/Life Science8355 General Chemistry5807 Physics535 Other1483 Social Studies American History1734 Other History3006 Psychology16230 Sociology8231 Geography612 Political Science6108 Economics3498 Other2361

9  Students receiving an A  Students receiving a B or better  Students receiving a C or better  Students receiving a B or better in 4-year universities  Students receiving a B or better in 2-year institutions

10  Logistic Regression (LR) o Identify score that gives a 50% probability of achieving an A o Identify score that gives a 50% probability of achieving a B or better o Identify score that gives a 50% probability of achieving a C or better  Signal Detection Theory (SDT) o Identify scores that maximize the proportion receiving consistent classifications from MME to college grades i.e., both proficient/advanced and receiving a A/B/C or better i.e., both not proficient/partially proficient and receiving a A-/B-/C- or worse o Equivalent to LR under mild monotonicity assumptions  Selected SDT as the preferred method because of its purpose (maximizing consistent classification)

11 Where success is obtaining an A/B/C or better e is the base of the natural logarithm β 0 is the intercept of the logistic regression β 1 is the slope of the logistic regression x is the MME score

12

13

14

15

16 Basic Idea  Set the MME cut score to…  Maximize the number of students in the Consistent cells  Minimize the number of students in the Inconsistent cells  Maximize consistent classification from MME to first-year college grades MME (unknown cut score) Freshman Grade (known cut score) B- or LowerB or Higher College ReadyInconsistentConsistent Not College ReadyConsistentInconsistent

17

18

19 Adjust the unknown cut score to maximize consistent classification

20

21

22  Analyses treating grades of A as the success criterion produced unusable results (i.e., the highest possible MME scale scores  Analyses treating grades of C as the success criterion produced unusable results (i.e., MME scale scores below chance level)  Analyses treating 4-year and 2-year institutions did produce different cut scores, but they were within measurement error of each other  Used analyses based on all institutions and grades of B or better to produce MME cut scores  Used probability of success of 33% and 67% to set the “partially proficient” and “advanced” cut scores  SDT and LR produced very similar results  Used SDT because it was the preferred methodology

23 Content Area Classification Consistency Partially Proficient Cut Score Proficient Cut Score Advanced Cut Score Mathematics65% Reading63% Science67% Social Studies63%

 24 Identifying MEAP Cut Scores

25 Cohort Grade

26  Logistic Regression (LR) o Identify score that gives a 50% probability of achieving proficiency on a later- grade test (i.e., MME or MEAP)  Signal Detection Theory (SDT) o Identify scores that maximize the proportion receiving consistent classifications from one grade to a later grade i.e., proficient/advanced on both tests i.e., not proficient/partially proficient on both tests o Equivalent to LR under mild monotonicity assumptions  Equipercentile Cohort Matching (ECM) o Identify scores that give the same percentage of students proficient/advanced on both tests  Selected SDT as the preferred method because of its purpose (maximizing consistent classification)  However, SDT and LR are susceptible to regression away from the mean

27  Same as for identifying MME cut scores  Criterion for success is proficiency on a later grade test rather than getting a B or better in a related college course

28 Each dot represents a plot of test scores in grade 8 and grade 11 for a single student

29 Grade 11: ProficientGrade 11: Not proficient

30 Grade 8: Proficient Grade 11: Not proficient Grade 8: Proficient Grade 11: Proficient Grade 8: Not proficient Grade 11: Not proficient Grade 8: Not Proficient Grade 11: Proficient

31

32

33  The more links in the chain, the greater the effects of regression  Original plan for Math and Reading o Link grade 11 MME to college grades o Link grade 8 MEAP to grade 11 MME o Link grade 7 MEAP to grade 8 MEAP o Link grade 6 MEAP to grade 7 MEAP o Link grade 5 MEAP to grade 6 MEAP o Link grade 4 MEAP to grade 5 MEAP o Link grade 3 MEAP to grade 4 MEAP  Original plan results in 7 links by the time the grade 3 cut is set  Original plan results in inflated cut scores in lower grades

34  Revised plan for Math and Reading  For Grade 3, 4, 5, 6 o Link grade 11 MME to college grades o Link grade 7 MEAP to grade 11 MME o Link grade 3, 4, 5, or 6 MEAP to grade 7 MME  For Grade 7, 8 o Link grade 11 MME to college grades o Link grade 7 or 8 MEAP to grade 11 MME  Results in a maximum of three links for any one grade

35  No evidence of regression away from the mean in identifying MEAP “proficient” cut scores o Looking for a consistently lower percentage of students proficient as one goes down in grades o Used SDT to identify MEAP “proficient” cut scores  Evidence of regression away from the mean in identifying MEAP “partially proficient” and “advanced” cut scores o Increasingly smaller percentages of students in the “Not proficient” and “Advanced” categories as one goes down in grade o Used ECM instead to identify MEAP “Not Proficient” and “Advanced” cut scores

36  No evidence of regression away from the mean in identifying MEAP “proficient” cut scores o Looking for a consistently lower percentage of students proficient as one goes down in grades o Used SDT to identify MEAP “proficient” cut scores  Evidence of regression away from the mean in identifying MEAP “partially proficient” and “advanced” cut scores o Increasingly smaller percentages of students in the “Not proficient” and “Advanced” categories as one goes down in grade o Used ECM instead to identify MEAP “Not Proficient” and “Advanced” cut scores

37  Classification Consistency Rates for MEAP Cut Scores in Mathematics Grade Cut Score Partially ProficientProficientAdvanced 883%86%95% 781%84%95% 682%83%96% 581%84%95% 480%82%94% 377%80%95%

38  Classification Consistency Rates for MEAP Cut Scores in Reading Grade Cut Score Partially ProficientProficientAdvanced 883%78%87% 786%76%85% 6 74%83% 588%75%84% 480%82%94% 380%72%86%

39  Classification Consistency Rates for MEAP Cut Scores in Science Grade Cut Score Partially ProficientProficientAdvanced 880%84%92% 576%82%92%

40  Classification Consistency Rates for MEAP Cut Scores in Science Grade Cut Score Partially ProficientProficientAdvanced 985%81%91% 681%77%91%

 41 Creating Mini-Cuts for PLC Calculations in Reading and Mathematics

42

43

44

45

46

 47 New Versus Old Cut Scores

48

49

50

51

52

53

54

 55 New Versus Old Cut Scores

56

57

58

59

60

61

62

 63 New Versus Old Cut Scores

64

65

66

 67 New Versus Old Cut Scores

68

69

70

71  Joseph A. Martineau o Executive Director o Bureau of Assessment & Accountability o Michigan Department of Education o o