1. MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation Irvine Valley College

2. OVERVIEW Standardized assessment, in its current form, has led us to systematically underestimate student capacity Underestimating students is not cost-free Evidence-based, strong multiple measures provide a key tool in the effort to transform basic skills in California and nationally Powerful completion, equity, and real world implications Based on years of substantial, convergent research in California and in other states Multiple measures can greatly increase placement accuracy

3. 3

4. CURRENT PRACTICE CCCs reliant on standardized assessment (REL West, 2011) http://bit.ly/20wYEZT 97% of CCCs place students with a standardized assessment Majority of students are placed below college-level 68% of students in two year institutions take developmental education course (Scott-Clayton & Belfield, 2015). http://bit.ly/CCRCPlacementAccuracy >85% of CCC students place into basic skills http://bit.ly/11_12_Basic_Skills_Report Most CCCs collect multiple measures data. Few leverage it. 81% collect information on high school performance 65% collect high school GPA MMs are typically assigned a relatively minor weight (e.g., Ngo & Kwon, 2014) http://bit.ly/1KP8NPZ

5. WHAT DOES THIS MEAN FOR STUDENTS? First interaction with college is basically a statement of our lack of confidence in them Implied message: You are not ready for college and are likely to fail ~30% of those assigned to remediation never attempt a course in the sequence

6. COULD CONVENTIONAL WISDOM BE WRONG? Research increasingly questions effectiveness of standardized assessment as the only measure to understanding student capacity Little relation to college course outcomes ( e.g., Belfield & Crosta, 2012; Edgecombe, 2011; Scott-Clayton, 2012; Scott- Clayton & Rodriguez, 2012 ): bit.ly/CCRCAssess bit.ly/CCRCAssess Underestimates capability of students of color, women, first generation college students, low SES Hiss & Franks, 2014; bit.ly/DefiningPromisebit.ly/DefiningPromise

7. A NEW HOPE There is a powerful, easily deployed multiple measures system that will work seamlessly with your current placement test and with the incoming common assessment system A team of researchers from the RP Group, Educational Results/Cal-PASS Plus and the University of Michigan has spent three years developing and refining this system There is a $1.5 million grant to help you get this system going

8. MULTIPLE MEASURES ASSESSMENT PROJECT (MMAP) Predicting successful completion of English & math in the CCCs based on high school performance data Focus on predictive validity (success in course) and improving student completion of foundational skills Statewide support Research base, predictive analytics, decision tree models Pilot colleges and faculty/staff engagement Webinars, convenings/summits, professional development K-12 outreach and data population Data warehouse and tool development Integration with Common Assessment System

9. GROWING INTEREST AND SCALE Colleges continue to join the project and enthusiastically inquire about participating 41 pilot colleges now committed, 8 more at various stages of exploration, representing more than: >900,000 community college students >40% of community college students statewide >8% of all community college students nationally 11 had pilots in place for Fall 2015 10 colleges submitted prospective applicants for Spring 2016

11. 11

14. SAMPLE PHASE II RULE SETS English LevelRule (NDMs) TransferHS 12 GPA >= 2.6 One level below Transfer HS 12 GPA >= 2.2 AND C or better in 12 th grade English course Math LevelRule (DMs) College Algebra HS 12 GPA >=3.2 OR HS 12 GPA >=2.9 AND Pre-Calculus C or better Intermediate Algebra HS 12 GPA >=2.9 OR HS 12 GPA >=2.5 AND Algebra II CST >= 302 Find the full rule set here: http://bit.ly/PhaseII_MMAP_rules

15. STATISTICS TREE – NDM

16. DISJUNCTIVE APPLICATION OF MULTIPLE MEASURES Disjunctive Placement Test score OR High School Transcript OR AP score OR EAP score OR Essay score Conjunctive Placement Test score AND (High School Transcript OR AP score OR EAP score OR Essay Score)

17. PROJECTED IMPACT ON COURSE SUCCESS RATES (COMPLETION OF COURSE WITH C OR BETTER)

18. SAN DIEGO CCD MMAP F2015 PILOT http://bit.ly/MMAPPilot

19. LBCC MULTIPLE MEASURES RESEARCH Five cohorts tracking more than 7,000 HS grads who matriculated to LBCC directly Examined predictive utility of wide range of high school achievement data for predicting: How students are assessed and placed How students perform in those classes

20. RE-IMAGINED STUDENT CAPACITY Starting in Fall 2012, students were provided an alternative assessment Reverse-engineered the analysis to place students using: Last high school course in discipline Grade in last course in discipline Overall HSGPA Last standardized test in discipline (and level) Placed students in highest course where projected success rate higher than average success rate for that course.

21. LBCC COHORT 3 (FALL 2014): SUCCESS RATES IN TRANSFER- LEVEL COURSES English difference, p <.001

22. WHAT MIGHT THIS MEAN FOR STUDENTS? LBCC saved students over 10,000 semesters of unneeded remediation in first three years. $250 per course for student (plus books!), $750 per course for state At scale (113 CCs) this could lead to 250,000 years of time, each year, saved for our students Dramatic opportunity costs of college reduced Median 2012 salary of “some college” is ~$30,000/year Don’t lose their first year or median salary though, they lose their last year.

23. THANK YOU! For more information about MMAP visit: http://bit.ly/About_MMAP http://bit.ly/About_MMAP And/or contact: Mallory Newell, Project Manager, MMAP (newellmallory@fhda.edu)newellmallory@fhda.edu or Ken Sorey, CalPASS Plus (ken@edresults.org)ken@edresults.org