1 PRESENTED TO THE FALL 2011 MEETING OF THE MICHIGAN EDUCATIONAL RESEARCH ASSOCIATION MAY 17, 2011 Methods for Setting Cut Scores to Represent College Readiness

2 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Resources Contributors  MDE Measurement Research & Psychometrics Unit  ACT Research & Development Unit  National Center for Educational Achievement  Michigan Technical Advisory Committee

3 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining College Readiness ACT Methodology  Good basis for a starting point  What score on the ACT gives an 11 th grader a 50/50 chance of obtaining a B or better in a… first-semester of freshman year credit-bearing (non-remedial)  …course in a related subject?

4 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining College Readiness ACT Methodology ACT Methodology for identifying College Ready Benchmarks   A logistic regression methodology identifying the point on the ACT subject score scale where a student has a 50% or better probability of achieving a B or better in the first-semester, credit-bearing, adequately related, freshman course Start with data from Michigan Public Institutions of Higher Education  Identify appropriate credit-bearing freshman courses against which to analyze the relationship between MME scores and course grades  Analyze proportion of students receiving a B or better against MME scale scores

5 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining College Readiness ACT Methodology Each dot represents a group of students with a specific test score, plotting the test score against the percent of students with that test score who earned a B or better

6 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining College Readiness ACT Methodology The curved line is the logistic regression line of best fit through the cloud of points

7 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining College Readiness ACT Methodology Horizontal line represents a 50% probability of earning a B or better

8 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining College Readiness ACT Methodology Vertical line represents the test score that give a 50% probability of earning a B or better

9 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining College Readiness ACT Methodology Issues With the ACT Methodology for the Purpose of Setting Cut Scores on the MME  Non-symmetric, regression-based method  Focused on probability prediction ACT’s purpose Appropriate for ACT’s purpose  Identifying equivalent points on different scales requires the use of a symmetric procedure.  Using a non-symmetric procedure can result in a biased cut score because of regression to the mean when identifying a cut score not at the state average

10 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining College Readiness Proposed Methodology Proposed Methodology  A Method Based on Signal Detection Theory (SDT)  A symmetric method  Focused on maximizing classification consistency Maximizes the percentage of students who are both 1.Classified as proficient on 11 th grade MME 2.Achieved a B or better in their first-semester, freshman-year, credit- bearing college course Classification consistency is the core of identifying both college readiness in high school and being on track in lower grades  Results in an unbiased cut score because regression to the mean is not an issue

11 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining College Readiness Proposed Methodology SDT in a nutshell, for our purposes  Focused on the ability to…  Make correct decisions when data contain both “signal” and “noise”  Identify thresholds of “signal” in the data that maximize the ability to make correct classification decisions

12 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining College Readiness Proposed Methodology Other uses of SDT-based methods germane to our purposes  Maximally accurate detection of…  Medical anomalies in reading radiological reports  Severe cases in Emergency Room triage  Medical impacts in clinical drug trials  Objects in naval sonar, military radar, and civilian air traffic control  Impact of interventions on memory and cognition Our purpose  Maximally accurate detection of…  College readiness on MME  Being on-track to college readiness on MEAP

13 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining College Readiness Proposed Methodology Basic application of SDT-based methods for our purposes  Using the known outcomes of college course grades, set the MME cut score to…  Maximize the number of students who are consistently classified  Minimize the number of students who are inconsistently classified Grade 11 MME (unknown cut score) College Grades (known outcome) Less than BB or Better ProficientInconsistentConsistent Not ProficientConsistentInconsistent

14 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining College Readiness Proposed Methodology Start with data from Michigan Public Institutions of Higher Education  Identify appropriate first-semester, freshman, credit-bearing (non- remedial) courses against which to analyze the relationship between MME scores and course grades  Identify students who took both the MME as 11 th graders and took those courses  Split those students into those who received a B or higher and those who received a B- or lower  Depicted graphically in the next slides

15 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Each dot on the left side of the vertical line represents one student who achieved a B- or lower in a first-semester freshman year, credit- bearing course Each dot on the right side of the vertical line represents one student who achieved a B or higher in a first-semester freshman year, credit- bearing course

16 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Each dot on the upper side of the horizontal line represents one student who achieved proficiency on the MME Each dot under the horizontal line represents one student who was not proficient on the MME

17 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Inconsistent Proficient in 11 th grade B- or lower in college Consistent Not proficient in 11 th grade B- or lower in college Consistent Proficient in 11 th grade B or higher in college

18 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA

19 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA

20 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA 75.00% Consistent

21 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA 79.69% Consistent

22 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining College Readiness Proposed Methodology  Intend to create three cut scores based on this methodology  Produces four performance levels  All three cut scores can have externally validated meaning:  Example Cut Score Set 1:  A or better in college course as Advanced  B or better in college course as Proficient  C or better in college course as Partially Proficient  Example Cut Score Set 2:  B or better in selective enrollment 4-year Universities as Advanced  B or better in non-selective 4-year Universities as Proficient  B or better in community college as Partially Proficient  Graphical example shows one cut at a time being set  For convenience  Statistical methodology can do all three cuts simultaneously

23 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining On-Track for the Highest Grade Level of Each MEAP Subject From the new MME cut scores, identify new cut scores for MEAP as follows…  Mathematics – grade 8  Reading – grade 8  Science – grade 8  Social Studies – grade 9 Use test scores of students who took both the MME in Spring 2010 and the grade 9 MEAP in Fall of 2007 for Social Studies Use test scores of students who took both the MME in Spring 2010 and the grade 8 MEAP in Fall of 2006 for Mathematics, Reading, and Science

24 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining On-Track for the Highest Grade Level of Each MEAP Subject Each dot on the graph represents the grade 11 MME score and the score on the highest grade level of MEAP for an individual student

25 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining On-Track for the Highest Grade Level of Each MEAP Subject MME cut scores previously identified using SDT-based methods

26 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining On-Track for the Highest Grade Level of Each MEAP Subject Not Proficient on MME Partially Proficient on MME Proficient on MME Advanced on MME

27 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining On-Track for the Highest Grade Level of Each MEAP Subject MEAP grade 8 cut scores to be identified

28 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining On-Track for the Highest Grade Level of Each MEAP Subject

29 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining On-Track for the Highest Grade Level of Each MEAP Subject Adjust the Grade 8 MEAP Cut Scores Up or Down to Obtain Maximum Classification Consistency

30 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining On-Track for the Remaining Grade Levels of Each MEAP Subject Use test scores of students who took MEAP in both Fall 2009 and Fall 2010 for Reading and Math Use test scores of students who took MEAP in both Fall 2007 and Fall 2010 for Science and Social Studies Why not use the same cohort all the way back to the lowest MEAP grade?  Data are not available  Need to use the most recent data for identifying each grade level’s cut score  Need to reflect the current state of Michigan education

31 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Defining On-Track for the Remaining Grade Levels of Each MEAP Subject Use same SDT-based method to go systematically down the line one grade at a time  Mathematics and Reading  Connect grade 8 to grade 7  Connect grade 7 to grade 6  Connect grade 6 to grade 5  Connect grade 5 to grade 4  Connect grade 4 to grade 3  Science  Connect grade 8 to grade 5  Social Studies  Connect grade 9 to grade 6

32 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Next Steps 1. Obtain data from Institutions of Higher Education (currently being done, from CEPI) 2. Determine the appropriate credit-bearing freshman courses against which to analyze MME scores (currently being done, with ACT and CEPI) 3. Identify college-ready cut scores for all subjects on the MME 4. Identify on-track cut scores for all subjects on the MEAP 5. Recommend to the State Board of Education the final cut scores for Fall 2011 MEAP and Spring 2012 MME

33 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA References Boutis, K., Pecaric, M., Seeto, B. and Pusic, M. (2010). Using signal detection theory to model changes in serial learning of radiological image interpretation. Advances in health sciences education: theory and practice, 15(5), Despins, L., Scott-Cawiezell, J., and Rouder, J. (2010). Detection of patient risk by nurses: a theoretical framework. Journal of advanced nursing, 66(2), Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley. Klein, S. and Levi, D. (2009) Stochastic model for detection of signals in noise. Journal of the Optical Society of America: Optics, image science, and vision, 26(11), Merlo-Pich, E. and Gomeni, R. (2008). Model-based approach and signal detection theory to evaluate the performance of recruitment centers in clinical trials with antidepressant drugs. Clinical pharmacology and therapeutics, 84(3), Neal, A. and Kwantes, P. (2009) An evidence accumulation model for conflict detection performance in a simulated air traffic control task. Human factors, 51(2), Pigeau, R., Angus, R., O’Neill, P. and Mack, I. (1995). Vigilance latencies to aircraft detection among NORAD surveillance operators. Human Factors, 37(3),

34 College Ready Cut Scores for MEAP and MME May 17, 2011 MERA Contact Information Joseph A. Martineau, Ph.D.  Executive Director  Bureau of Assessment & Accountability  Michigan Department of Education 