1. Let Icarus Fly : Multiple Measures, Assessment, and the Re- imagination of Student Capacity John J. Hetts, Ph.D. Senior Director of Data Science Educational Results Partnership jhetts@edresults.org @jjhetts #LetIcarusFly Oregon Placement Work Group October 23, 2015

2. Overview Standardized assessment has led us to systematically and substantially underestimate student capacity Particularly for students of color, low income students, first generation college students, women Evidence-based, multiple measures is one of four key cornerstones on which to rebuild the foundations of community college education Demonstrates fundamental capacity of far more of our students to succeed if given the chance Powerful completion, equity, and real world implications Based powerfully both on basic principles of assessment and measurement as well as strong

3. First, Daedalus and Icarus Daedalus crafted the labyrinth of inescapable complexity for King Minos To escape from Minos, Daedalus built wings of feather and wax for his son Icarus and himself Don’t fly too high, lest sun melt the wax and you plummet to your doom Dangers of innovation/invention, hubris, Importance of knowing your limits, listening to your wiser elders But most of us forget the rest of that story…

4. Transition to College: Assessment and Placement Community colleges are open enrollment institutions Requires assessing and planning for educational needs of students. Goal Effectively place student at most appropriate level for their skill – where challenge matches skill level Zone of proximal development Optimal performance, flow If you think you can catch the bus, you will run for it.” Lee Peng Yee, Singapore National Institute of Education Mathematician

5. Community college student transition Community colleges rely nearly entirely on standardized assessment (WestEd, 2011) Majority of students placed below college-level Significant barrier to completion (Bailey, Jeong, &Cho, 2010) 50-60% of equity gap in completions (Stoup, 2015) What does this mean? First interaction is to tell students they don’t belong Imply most students not ready for college and likely to fail Convinced nearly everyone Including our students

6. Conventional Wisdom Explaining Assessment Results It is a problem with today’s students Students are simply, vastly unprepared for college Kids these days ….

7. That seems awfully familiar

8. Too familiar (Bye Bye Birdie – 1963)

9. What If that Conventional Wisdom is Wrong? Substantial, long-term increase in IQ: bit.ly/FlynnEffectIQ bit.ly/FlynnEffectIQ 18-24 with HS degree: 91% - highest ever: bit.ly/2014HS18-24bit.ly/2014HS18-24 National Assessment of Educational Progress: at all-time highs in virtually every demographic category: bit.ly/NAEPInfo bit.ly/NAEPInfo

10. NAEP Math and Reading Assessments

11. What If the Conventional Wisdom is Wrong? Research increasingly questions effectiveness of standardized assessment for understanding student capacity Little relation to college course outcomes (e.g., Belfield & Crosta, 2012; Edgescombe, 2011; Scott-Clayton, 2012; Scott-Clayton & Rodriguez, 2012): bit.ly/CCRCAssess bit.ly/CCRCAssess Incredible variability in cutscores and 2-year colleges often use HIGHER cutscores than 4-year bit.ly/NAGB2012 Underestimates capability of students of color, women, first generation college students, low SES Hiss & Franks, 2014; bit.ly/DefiningPromisebit.ly/DefiningPromise

12. What if? What if the problem is not primarily with our students but with limitations in how we have assessed and understood their capacity to do college-level work?

13. It gets worse What if this flawed method of understanding and “remediating” student capacity has actually had the opposite effect? Imagine yourself arriving at college as a community college student…

14. But there’s good news… What if one of the key barriers to our students’ successful transition to and success in college is one that we fully control?

15. Cornerstone 1: Improving assessment through evidence- based multiple measures Resources/references: http://www.lbcc.edu/PromisePathways http://bit.ly/MMAP2015 http://bit.ly/RPSTEPS http://bit.ly/RPMultipleMeasures http://cccassess.org

16. LBCC Multiple Measures Research Five cohorts tracking more than 7,000 HS grads who matriculate to LBCC directly Examined predictive utility of wide range of high school achievement data For predicting: How students are assessed and placed How students perform in those classes (and alignment between them)

17. Alignment in English * p <.05 **, p <.01, *** p<.001, x = p< 1 x 10 -10

18. Alignment in Math * p <.05 **, p <.01, *** p<.001, x = p< 1 x 10 -10

19. Key Takeaways Assessment should predict how students will perform at our colleges Instead: Current standardized tests predict standardized tests Classroom performance predicts classroom performance More info tells us more about student capacity than less info

20. Re-imagined student capacity Starting in Fall 2012, students from LBUSD (now 6 districts covering >30 high schools and growing) were provided an alternative assessment Reverse engineered the analysis to place students using: Last high school course in discipline Grade in last course in discipline Overall HSGPA Last standardized test in discipline (and level) Placed students in highest course where projected success rate higher than average success rate for that course.

21. Implementing Multiple Measures Placement: Transfer-level Placement Rates F2012

22. But …

23. … didn’t that just flood transfer- level courses with unqualified students?

24. Comparison against students in advancing through sequence: Success rates in transfer-level courses Neither of these differences approach significance, p >.30

25. Cohort 1 English 1 Success Rates by Placement (vs. 6 year completion)

26. Comparison against traditional sequence: Success rates in transfer-level courses English difference, p <.001

27. Overall Success Rate in Transfer Level English by Method of Qualification (among students with high school data available)

28. Success Rate by Method of Qualification in Transfer Level English

29. … what about grade inflation in HS?

30. HS GPA Distribution from MMAP dataset

31. ACT research on grade inflation Little evidence for grade inflation over last decade Earlier observations of grade inflation may have been partly artifactual adjustments to GPA for AP/IB/Honors Zhang & Sanchez, 2014: http://bit.ly/ACTGradeInflation http://bit.ly/ACTGradeInflation

32. Concerns about grade inflation and social promotion do not fit evidence Suggests that there should be little to no relation between HS grades and college grades because HS grades unrelated to performance Everyone gets As and Bs would mean no variation to predict outcomes Yet, predictive utility strongly observed Stronger than standardized tests

33. Westrick & Allen, 2014: ACT COMPASS Validation Median Logistic R (Table 4) http://bit.ly/ACTandGPAhttp://bit.ly/ACTandGPA CourseCompass TestCompassHSGPA HSGPA + Compass English 1Writing Skills.31. 57.62 ArithmeticPre-Algebra. 57.34.66 AlgebraPre-Algebra.36. 65.80 Intermediate AlgebraAlgebra.47. 66.84 College AlgebraAlgebra.41. 76.88 College Algebra.51. 76.94

34. Compass Score (30 extremely low to 90 extremely high) High School GPA 306090 2.00 3.00 4.00 Westrick & Allen, 2014: Conditional Success Rates (Table 6) http://bit.ly/ACTandGPAhttp://bit.ly/ACTandGPA Compass Score (30 extremely low to 90 extremely high) High School GPA306090 2.0023%28%32% 3.0043%49%55% 4.0065%70%75%

35. … didn’t that work only because Long Beach is special/has special relationship between LBCC and LBUSD?

36. Not just Long Beach LBCC now includes multiple additional districts including ABC Unified and Los Alamitos Unified Long thread of research in the CCCs alone 2008: Willett, Hayward, & Dahlstrom 11 th grade HS variables as early alert mechanism for discipline assessment 2011: Martinez self-reported HS variables as more powerful predictors of college completion 2014: Willett & Karanjeff replication of LBCC research with 12 additional colleges (STEPS) Replication of implementation Bakersfield College and Sierra College began similar implementation in 2014 CCRC research (Belfield & Crosta, 2012; Scott-Clayton, 2012) MMAP Statewide Research & local replications: bit.ly/MMAP2015 bit.ly/MMAP2015

37. CA Multiple Measures Assessment Project (MMAP) Examination of high school achievement data for predictors of successful completion of English & math in the CCCs Focus on predictive validity (success in course) and improving student completion of foundational skills Statewide support Research base, predictive analytics, decision tree models Pilot colleges and faculty/staff engagement Webinars, convenings/summits, professional development K-12 outreach and data population Data warehouse and tool development

38. High school variables that predict college success English Cumulative HS GPA Grade in last HS English C+ or better in AP English class Score on English CST Non-remedial status in HS English Math Cumulative HS GPA Enrollment and grades in Geometry, Algebra II, Trigonometry, Pre-calculus, Statistics, Calculus Taking a more challenging CST Score on math CST Delay*

39. Sample Transfer-level decision rules English 11 th Grade High School GPA ≥ 2.6 Math (Statistics) 11 th Grade High School GPA ≥ 3.0 & Algebra I C or better OR 11 th Grade High School GPA ≥ 2.3 & Pre-calculus C or better

41. MMAP: Projected impact on course success rates

42. SDCCD MMAP F2015 Pilot (N = ~1000)

43. Sierra College F2014 Transfer-Level English Success Rates

44. … we’re happy with our placement. Why should we change?

45. Powerful reasons for change: 1) Basic assessment theory and methods Self-reported satisfaction with assessment by instructors and students is most common measure Has grave methodological flaws: Selection bias Confirmation bias Effort justification System justification Self-fulfilling prophecy effects

46. Powerful reasons for change: 1) Basic assessment theory and methods Principles of assessment/measurement theory: Measurement = True score + systematic error + random error Methodological gold standard of assessment- triangulation to true score via assessment across: methods content domains evaluators Time GPA assesses capacity assessment methods, content domains, dozens of instructors, and across time Combined with other available evidence/assessments increases quality further

47. Powerful reasons for change: 2) It’s poorly assessing students Substantial evidence of systemic & severe underplacement placing students in developmental education who could get a B or better in the transfer level course 36% percent of students placed into development English 25% of students placed into developmental Math Using multiple measures reduces error, has potential to increase success rates and sequence completion http://bit.ly/CCRCPlacementAccuracy

48. Powerful reasons for change: 3) Transformational impacts for students Potential for dramatic increases in rates and time to completion of Transfer-level course in discipline Subsequent courses in discipline Other early educational milestones.

49. F2012 Promise Pathways vs. Fall 2011 2-year rates of achievement

50. Equity impact LBCC: F2011 Baseline Equity Gaps for 2-year rates of achievement

51. Equity impact LBCC: F2012 2-year rates of achievement

54. … what about students for whom high school transcript data aren’t available/easy to get?

55. Self-reported HSGPA appears to be reliable alternative College of the Canyons Research (Gribbons, 2014) Self-report of last course and grade in Fall term very accurate Errors that do occur in part because of timing of assessment University of California admissions Uses self-report HSGPA but verifies after admission 2008: 9 campuses, 60000 students. No campus had more than 5 discrepancies b/w reported grades and student transcripts: http://bit.ly/UCSelfReportGPA Much of the ACT research uses self-report GPA and finds it to be a more powerful predictor than students actual scores on the standardized tests ACT, 2013: r(1978) =.84

56. ACT, 2013: http://bit.ly/ACTSelf-ReportedGPA http://bit.ly/ACTSelf-ReportedGPA Actual HSGPA Level N Mean HSGPA Mean diff. Accuracy Actual Student- reported % within 0.25 % within 0.50 3.50–4.005993.793.75–0.0487%98% 3.00–3.494513.243.23–0.0160%90% 2.50–2.994082.812.760.0547%82% 2.00–2.492652.242.350.1140%73% 1.50–1.991721.772.040.2730%55% 0.00–1.49851.031.850.8214%35% Total1,9802.953.020.0758%83%

57. … what about non-traditional students?

58. Multiple measures continues to have utility for delayed matriculants Delay in matriculation is a main effect i.e., likelihood of success decreases somewhat Does not interact with HSGPA or course grades “A” students at a delay definitely better than “B students” at a delay “A” students at a delay remain better than “B” Likely grade drops but still very likely to be successful HSGPA continues to be predictively useful up to the point where we have data we can meaningfully connect(delay of ~10 years).

59. How long is High School GPA good for?

60. Westrick & Allen, 2014: ACT COMPASS Validation Standardized Logistic Regression Coefficients(Table 5) http://bit.ly/ACTandGPAhttp://bit.ly/ACTandGPA Course Compass TestStudent TypeCompassHSGPADiff English 1Writing SkillsTraditional.25.72.47 Nontraditional.21.36.15 ArithmeticPre-AlgebraTraditional.67.51 -.16 Nontraditional.43.08 -.35 AlgebraPre-AlgebraTraditional.43.78.35 Nontraditional.32.47.15 Int. AlgebraAlgebraTraditional.52.76.24 Nontraditional.44.25 -.19 Coll. AlgebraAlgebraTraditional.36.88.52 Nontraditional.43.59.16 Coll. Algebra Traditional.50.82.32 Nontraditional.26.47.21

61. … that doesn’t seem like enough evidence that I need to rethink student capacity. What else have you got?

62. Cornerstone 2: Reconsider cut scores Resources/references: http://bit.ly/LetThemIn (Henson & Hern, 2014) http://bit.ly/LetThemIn http://bit.ly/Kalamkarian2015 (Kalamkarian, Raufman, & Edgecombe, 2015) http://bit.ly/Kalamkarian2015 http://bit.ly/Rodriguez2014 (Rodriguez, 2014) http://bit.ly/Rodriguez2014

63. Natural experiment at Butte College In 2011, switched from one placement test to another Old test/cut scores: 23% of incoming students “college ready” in English New test/cut scores: 48% of incoming students “college ready” in English

64. Butte College: Assessment of first-year students

65. Butte College: Completion of Transfer-Level English in 1 st Year

66. Developmental Math Reform – Virginia Community College System Introduced new assessment instrument Intentionally increased percentage assigned to college-level math

67. Additional bodies of work showing higher student capacity Acceleration (e.g., Hayward & Willett, 2014) http://bit.ly/CAPEval http://bit.ly/CAPEval Corequisite developmental education (e.g., Coleman, 2015) http://bit.ly/2015ALPhttp://bit.ly/2015ALP 2-4X transfer-level course completion Comparable or higher success rates Works across demographic group Reduces equity gaps substantially

68. Evaluation of 2011-2012 pilot year of California Acceleration Project Summary of Findings (Hayward & Willett, 2014) Large and robust effects of acceleration that work for Students of all backgrounds Students at all placement levels Not a function of selection/cherry-picking Examples from Math

69. Regression Adjusted Effects – Math

70. Completion of transfer-level math for traditional and accelerated pathways by ethnicity

71. Replication of CCBC ALP model ALP model involves: Enrollment directly in college-level English (mainstreamed) Concurrent enrollment in just-in-time companion developmental English course taught by same instructor Four early implementers at or near scale

72. Completion of College-Level English (of those who take one-level below course)

73. Combining cutscore revision and corequisite expansion in English - VCCS

74. Completion of College English - VCCS

75. Summary of impacts of approaches that re-imagine student capacity Transfer-level success rates (if taken) Developmental Success Rates Transfer-level completion (by entire cohort) Meaningful equity impacts Upfront Development of Curriculum Multiple Measures No change to higher Lower overall (but no change for students that remain) Much higher Substantial Low Acceleration No change to higher Much Higher Substantial High Moderate Corequisite models Higher Much Higher Substantial High Moderate Cutscore revision Slightly lower No change to slightly lower Much Higher Substantial Low

76. What might this mean for students? Bakersfield College has saved over 3000 semesters in first two years LBCC saved students over 10,000 semesters (5000 years!) of unneeded remediation in first three years. $250 per course for student (plus books!), $750 per course for state Dramatic opportunity costs of college reduced Median 2012 salary of “some college” is ~$30,000/year Don’t lose their first year or median salary though, they lose their last year.

77. What might this mean for all of us? These evidence-based cornerstones save students 1-2 semesters of developmental education that: Evidence predicted and research demonstrated that they did not need By law (in some places), fairness, and basic educational practice should not have been required to take

78. The Great Recession of 2008 took 125- 150,000 people of Oregon out of the workforce of Oregon Failure to correctly understand the capacity of Oregon’s community college students may be costing those140,000-150,000 students a year or more of additional time out of the workforce. Sense of the scale

79. What can be gained through assessment and placement reform A clarion call to better understand the true capacity of our students. The ability to transform student outcomes Powerful levers to address student equity gaps Renewed opportunities to collaborate with our K-12 educational colleagues The chance to stop meeting students at front door to tell them they don’t belong A far better future for our students and for us all A reminder of Daedalus’ second instruction to Icarus. It’s just as important not to fly to low.

80. Thank you! Contact information John Hetts Educational Results Partnership jhetts@edresults.org 916-498-8980 ext. 208 714-380-2678 cell Twitter: @jjhetts