MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY March 2, 2016 Loris Fagioli, PhD Research and Planning Analyst, Irvine Valley College Research Analyst, RP Group MMAP team

OVERVIEW Standardized assessment, in its current form, has led us to systematically underestimate student capacity Underestimating students is not cost-free Evidence-based, strong multiple measures provide a key tool in the effort to transform basic skills in California and nationally Powerful completion, equity, and real world implications Based on years of substantial, convergent research in California and in other states Multiple measures can greatly increase placement accuracy

3

CURRENT PRACTICE 97% of CCCs place students with standardized assessment (REL West, 2011) Majority of students are placed below college-level 68% of students nationwide (Scott-Clayton & Belfield, 2015). >85% of CCC students Most CCCs collect multiple measures data. Few leverage it. 81% collect information on high school performance 65% collect high school GPA Multiple measures are typically assigned a relatively minor weight (e.g., Ngo & Kwon, 2014) Using Multiple Measures to place students is required by law

WHAT DOES THIS MEAN FOR STUDENTS? Implied message: You are not ready for college and are likely to fail ~30% of those assigned to remediation never attempt a course in the sequence

COULD CONVENTIONAL WISDOM BE WRONG? Research increasingly questions effectiveness of standardized assessment as the only measure to understanding student capacity Little relation to college course outcomes ( e.g., Belfield & Crosta, 2012; Edgecombe, 2011; Scott-Clayton, 2012; Scott-Clayton & Rodriguez, 2012 ) bit.ly/CCRCAssess bit.ly/CCRCAssess Underestimates capability of students of color, women, first generation college students, low SES ( Hiss & Franks, 2014) bit.ly/DefiningPromise bit.ly/DefiningPromise Incredible variability in cutscores; CCs often use higher cutscores than 4-year institutions bit.ly/NAGB2012 bit.ly/NAGB2012

A NEW HOPE Multiple measures systems are Easily employed Work seamlessly with your current placement test/CAI Have been developed and refined for 3 years by RP Group Educational Results/Cal-PASS Plus The University of Michigan There is a $1.5 million grant to help you get this system going

MULTIPLE MEASURES ASSESSMENT PROJECT (MMAP) Predicting successful completion of English & math in the CCCs based on high school performance data HS GPA, individual course grades, CST scores Other Focus on predictive validity (success in course) Math, English, Reading, and ESL Statewide support Research base, predictive analytics, decision tree models Pilot colleges and faculty/staff engagement Webinars, convenings/summits, professional development K-12 outreach and data population Data warehouse and tool development Integration with Common Assessment System

GROWING INTEREST AND SCALE Colleges continue to join the project and enthusiastically inquire about participating 41 pilot colleges now committed, (8 more at various stages) >900,000 community college students >40% of community college students statewide >8% of all community college students nationally 11 colleges have pilots in place for Fall colleges plan to use multiple measures for Spring 2016

IT WORKS!!

SUCCESS STORY 1: SIERRA COLLEGE

SUCCESS STORY 2: LONG BEACH

SUCCESS RATE AT TRANSFER LEVEL Success Story 3: Bakersfield

ESTIMATED COSTS AND SAVINGS Old modelAccelerationMIH + Acceleration Semesters 321 Units 1064 Cost to get all to Engl 1A $1,677,552 $1,084,682 $208,851 Placement at Transfer level 18%29%57% N ~3500 Success Story 3: Bakersfield ~ $ 1.5 million savings

GOODIES FOR THE BREAKOUT SESSIONS Looking at how rules were developed specific rules and trees impact on equity impact on placement what is “disjunctive”? how do I get started? Q & A 15

THANK YOU! For more information about MMAP visit: And/or contact: Mallory Newell, Project Manager, MMAP Ken Sorey, CalPASS Plus

MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY BREAKOUT SESSION March 2, 2016 Loris Fagioli, PhD Research and Planning Analyst, Irvine Valley College Research Analyst, RP Group MMAP team

VARIABLES EXPLORED FOR MODEL- BUILDING Cumulative HS GPA (11 th and 12 th grade) Grades in English and Math classes Scores on CST Early Assessment Project (EAP) AP class grades Remedial class grades (if applicable) Expository class grades (if applicable) ESL class grades (if applicable) 18

DECISION TREE EXAMPLE: STATISTICS – NDM Percent of sample in leaf Success rate Node Leaf

SAMPLE PHASE II RULE SETS English LevelRule (NDMs) TransferHS 12 GPA >= 2.6 One level below Transfer HS 12 GPA >= 2.2 AND C or better in 12 th grade English course Math LevelRule (DMs) College Algebra HS 12 GPA >=3.2 OR HS 12 GPA >=2.9 AND Pre-Calculus C or better Intermediate Algebra HS 12 GPA >=2.9 OR HS 12 GPA >=2.5 AND Algebra II CST >= 302 Find the full rule set here:

Outreach to feeder high schools?

EXAMPLE PROJECTED PLACEMENT IN ENGLISH Irvine Valley College (Incoming Cohort Fall 2015)

ACTUAL PLACEMENT EXAMPLE San Diego CCD Fall 2015 Pilot

DISJUNCTIVE APPLICATION OF MULTIPLE MEASURES Disjunctive Placement (recommended) Conjunctive Placement Test ScoreOR High School Transcript OROther Measures (Non-cognitive, essay, other evidence) Test ScoreAND(High School Transcript OROther Measures)

WHAT MIGHT THIS MEAN FOR STUDENTS? LBCC saved students over 10,000 semesters of unneeded remediation in first three years. $250 per course for student (plus books!), $750 per course for state At scale (113 CCs) this could lead to 250,000 years of time, each year, saved for our students Dramatic opportunity costs of college reduced Median 2012 salary of “some college” is ~$30,000/year Don’t lose their first year or median salary though, they lose their last year.