1. MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY March 2, 2016 Loris Fagioli, PhD Research and Planning Analyst, Irvine Valley College Research Analyst, RP Group MMAP team

2. OVERVIEW Standardized assessment, in its current form, has led us to systematically underestimate student capacity Underestimating students is not cost-free Evidence-based, strong multiple measures provide a key tool in the effort to transform basic skills in California and nationally Powerful completion, equity, and real world implications Based on years of substantial, convergent research in California and in other states Multiple measures can greatly increase placement accuracy

3. 3

4. CURRENT PRACTICE 97% of CCCs place students with standardized assessment (REL West, 2011) http://bit.ly/20wYEZT Majority of students are placed below college-level 68% of students nationwide (Scott-Clayton & Belfield, 2015). http://bit.ly/CCRCPlacementAccuracy >85% of CCC students http://bit.ly/11_12_Basic_Skills_Report Most CCCs collect multiple measures data. Few leverage it. 81% collect information on high school performance 65% collect high school GPA Multiple measures are typically assigned a relatively minor weight (e.g., Ngo & Kwon, 2014) http://bit.ly/1KP8NPZ Using Multiple Measures to place students is required by law

5. WHAT DOES THIS MEAN FOR STUDENTS? Implied message: You are not ready for college and are likely to fail ~30% of those assigned to remediation never attempt a course in the sequence

6. COULD CONVENTIONAL WISDOM BE WRONG? Research increasingly questions effectiveness of standardized assessment as the only measure to understanding student capacity Little relation to college course outcomes ( e.g., Belfield & Crosta, 2012; Edgecombe, 2011; Scott-Clayton, 2012; Scott-Clayton & Rodriguez, 2012 ) bit.ly/CCRCAssess bit.ly/CCRCAssess Underestimates capability of students of color, women, first generation college students, low SES ( Hiss & Franks, 2014) bit.ly/DefiningPromise bit.ly/DefiningPromise Incredible variability in cutscores; CCs often use higher cutscores than 4-year institutions bit.ly/NAGB2012 bit.ly/NAGB2012

7. A NEW HOPE Multiple measures systems are Easily employed Work seamlessly with your current placement test/CAI Have been developed and refined for 3 years by RP Group Educational Results/Cal-PASS Plus The University of Michigan There is a $1.5 million grant to help you get this system going

8. MULTIPLE MEASURES ASSESSMENT PROJECT (MMAP) Predicting successful completion of English & math in the CCCs based on high school performance data HS GPA, individual course grades, CST scores Other Focus on predictive validity (success in course) Math, English, Reading, and ESL Statewide support Research base, predictive analytics, decision tree models Pilot colleges and faculty/staff engagement Webinars, convenings/summits, professional development K-12 outreach and data population Data warehouse and tool development Integration with Common Assessment System

9. GROWING INTEREST AND SCALE Colleges continue to join the project and enthusiastically inquire about participating 41 pilot colleges now committed, (8 more at various stages) >900,000 community college students >40% of community college students statewide >8% of all community college students nationally 11 colleges have pilots in place for Fall 2015 10 colleges plan to use multiple measures for Spring 2016

10. IT WORKS!!

11. SUCCESS STORY 1: SIERRA COLLEGE

12. SUCCESS STORY 2: LONG BEACH

13. SUCCESS RATE AT TRANSFER LEVEL Success Story 3: Bakersfield

14. ESTIMATED COSTS AND SAVINGS Old modelAccelerationMIH + Acceleration Semesters 321 Units 1064 Cost to get all to Engl 1A $1,677,552 $1,084,682 $208,851 Placement at Transfer level 18%29%57% N ~3500 Success Story 3: Bakersfield ~ $ 1.5 million savings

15. GOODIES FOR THE BREAKOUT SESSIONS Looking at how rules were developed specific rules and trees impact on equity impact on placement what is “disjunctive”? how do I get started? Q & A 15

16. THANK YOU! For more information about MMAP visit: http://bit.ly/About_MMAP http://bit.ly/About_MMAP And/or contact: Mallory Newell, Project Manager, MMAP newellmallory@fhda.edu Ken Sorey, CalPASS Plus ken@edresults.org

17. MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY BREAKOUT SESSION March 2, 2016 Loris Fagioli, PhD Research and Planning Analyst, Irvine Valley College Research Analyst, RP Group MMAP team

18. VARIABLES EXPLORED FOR MODEL- BUILDING Cumulative HS GPA (11 th and 12 th grade) Grades in English and Math classes Scores on CST Early Assessment Project (EAP) AP class grades Remedial class grades (if applicable) Expository class grades (if applicable) ESL class grades (if applicable) 18

19. DECISION TREE EXAMPLE: STATISTICS – NDM Percent of sample in leaf Success rate Node Leaf

20. SAMPLE PHASE II RULE SETS English LevelRule (NDMs) TransferHS 12 GPA >= 2.6 One level below Transfer HS 12 GPA >= 2.2 AND C or better in 12 th grade English course Math LevelRule (DMs) College Algebra HS 12 GPA >=3.2 OR HS 12 GPA >=2.9 AND Pre-Calculus C or better Intermediate Algebra HS 12 GPA >=2.9 OR HS 12 GPA >=2.5 AND Algebra II CST >= 302 Find the full rule set here: http://bit.ly/PhaseII_MMAP_rules

24. Outreach to feeder high schools?

25. EXAMPLE PROJECTED PLACEMENT IN ENGLISH Irvine Valley College (Incoming Cohort Fall 2015)

26. ACTUAL PLACEMENT EXAMPLE San Diego CCD Fall 2015 Pilot http://bit.ly/MMAPPilot

27. DISJUNCTIVE APPLICATION OF MULTIPLE MEASURES Disjunctive Placement (recommended) Conjunctive Placement Test ScoreOR High School Transcript OROther Measures (Non-cognitive, essay, other evidence) Test ScoreAND(High School Transcript OROther Measures)

28. WHAT MIGHT THIS MEAN FOR STUDENTS? LBCC saved students over 10,000 semesters of unneeded remediation in first three years. $250 per course for student (plus books!), $750 per course for state At scale (113 CCs) this could lead to 250,000 years of time, each year, saved for our students Dramatic opportunity costs of college reduced Median 2012 salary of “some college” is ~$30,000/year Don’t lose their first year or median salary though, they lose their last year.