1. East Bay Career Pathways Regional Convening #2 December 4, 2014
Let Icarus Fly: Unleashing student achievement through multiple measures assessment John J. Hetts, Ph.D. Senior Director of Data Science Educational Results Partnership
East Bay Career Pathways Regional Convening #2
December 4, 2014
John Hetts, Senior Data Scientist
Ken Sorey, VP ERP
Short Bios go here.

2. Overview
Standardized tests systematically underestimate student capacity
Students of color
First generation college students
Lower SES
Women
Multiple measures (esp. GPA) fairer and far more accurate predictor of college performance/graduation
Increase grad rates, decrease time to completion
Very low cost, exceptionally high ROI
To students
To colleges
To state
(optional intro) 0) Standardized assessments systematically underestimate student’s capacity to do college-level work, particularly students of color, first-generation college students, students with lower SES, and women, and do so so both at four year colleges and community colleges.
1) Using multiple measures, particularly high school GPA, in assessment, admissions, and merit-based aid, provides a fairer, far more accurate assessment of students’ ability both to perform well in college and to graduate.
2) As a result, using multiple measures will increase graduation rates, decrease time to graduation, and meaningfully help close equity gaps and can do so at next to no cost.
3) In fact, using multiple measures will save public education systems and taxpayers tens of millions of dollars annually in California alone in unnecessary costs of reteaching successfully mastered skills and millions of dollars annually in unnecessary assessments.
(for longer elevators) 4) Even more importantly, using multiple measures will save California students billions of dollars in lost salary/other opportunity costs that they currently lose to the time necessary to retake the same courses they successfully completed in high school and will meaningfully improve a wide range of their long-term life outcomes as well as their productive contributions to the economy and their communities.

3. Daedalus and Icarus
Daedalus crafted the labyrinth of inescapable complexity for King Minos
To escape from Minos, Daedalus built wings of feather and wax for his son Icarus and himself
Don’t fly too high, lest sun melt the wax and you plummet to your doom
Dangers of innovation/invention, hubris,
Importance of knowing your limits, listening to your wiser elders
But most of us forget the rest of that story…

4. Student transition to college
Community colleges rely nearly entirely on standardized assessment
Most CC students placed below college-level
Significant barrier (Bailey, Jeong, and Cho, 2010)
First interaction is to tell students they don’t belong
Imply that most students are not ready for college and are likely to fail
Convinces many, including our students
Substantial majority of community colleges rely heavily if not nearly entirely on standardized assessment
Most community college students placed into coursework below college-level
Significant barrier to completion (Bailey, Jeong, and Cho, 2010)
First interaction with most students is to tell them they’re not ready for college
Implies that most students are not ready for college and are likely to fail
Convinces many
Including our students
Very first interaction with student is to tell them they’re not ready for college
“You’re not good enough”
You don’t belong
You’re not really college material
Again, and again.
Internalized by faculty, staff, administration, counselors, public
Eventually, tragically, many of the students
Placement into remediation is a significant barrier to completion
Two enormous systemic changes:
More, longer, slower remediation
Reduction of access (offerings at bottom of sequences first to get restricted)

5. Conventional Wisdom Explaining Assessment Results
It is a problem with today’s students
Students are simply, vastly unprepared for college
Kids these days ….
It is a problem with public education
Public education is failing to prepare students
Teachers these days…
Not my students, not my teachers.. Teachers in general, students in general, sometimes other problematic adjectives added like urban.

6. What If the Conventional Wisdom is Wrong?
Substantial, long-term increase in IQ: bit.ly/FlynnEffectIQ
18-24 with HS degree: 90% - highest ever: bit.ly/HS18-24
National Assessment of Educational Progress: at all-time highs in virtually every demographic category:bit.ly/NAEPInfo
Research increasingly questions effectiveness of standardized assessment for placement
Little relation to college course outcomes. (e.g., Belfield & Crosta, 2012; Edgescombe, 2011; Scott-Clayton, 2012; Scott-Clayton & Rodriguez, 2012): bit.ly/CCRCAssess
NAGB, 2012: Incredible variability in cutscores; 2-year colleges often use HIGHER cutscores than 4-year bit.ly/NAGB2012
Hiss & Franks, 2014: Underestimates capability of students of color, women, first gen college students, low SES: bit.ly/DefiningPromise
the National Assessment of Educational Progress from the NCES National Center for Educational Statistics
National Assessment Governing Board
Austin Peay – 50% remediation, poor completion – eliminate remediation, less prepared students enroll in college-level with 2 hours/week in learning lab required. Pre-53% pass developmental math, 30% college level w/in 2 years. Post – 67% complete college-level. English – 54% to 76%:

7. Big questions
What if the problem is not with our students, but with how we have assessed their capabilities? OR
What if one of the barriers to our students’ successful transition to college is one that we fully control?

8. LBCC Research
Five longitudinal cohorts tracking more than 7,000 HS grads who attend LBCC directly after high school
built with help of Cal-PASS
Examined predictive utility of wide range of high school achievement data
most notably 11th grade California Standards Test (CST) scores and high school grades
For predicting:
How students are assessed and placed into developmental skills sequences
How students perform in those classes
(and for understanding alignment between them)
Highly simplified presentation of unstandardized ordinal and logistic regression
Lots more too it and RP has posted a lot of it

9. * p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10
Alignment in English
(unstandardized ordinal and logistic regression coefficients)
High school, course you took, A-G classes, more. What’s more important than 11th or 12th grade English – Math CST score – almost as strong as overall GPA.
Most important predictor of performance – what course you took – after that – it’s your overall GPA.
* p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10

10. * p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10
Alignment in Math
Math much more complicated than this – the last course you took/which CST matters a lot in performance as does the difficulty level of the which course you take.
* p <.05 **, p <.01, *** p<.001, x = p< 1 x 10-10

11. Key Takeaways
Assessment should predict how students will perform at our colleges
Instead:
Standardized tests best predict standardized tests
Classroom performance best predicts classroom performance
More information tells us more about students than less information
Replicated statewide by STEPS project, conceptually replicated by CCRC work
Significant opportunities exist to improve placement, student achievement, and students’ college experience.

12. Multiple Measures Placement: Transfer-level Placement Rates F2012
Attending a community college
>75% students of color, >50% Hispanic
Largely eligible for the BOGG fee waiver and Pell grants
Two major communities of English as a Second Language students
Mostly first person in family to go to college
Many first or second generation immigrants

13. F2012 Promise Pathways vs. Fall 2011 2-year rates of achievement
Compares first year rates of achievement to most recent 6-year cohort. Pathways is ahead, in many cases quite significantly.

14. Success rates in transfer-level courses Fall 2012
Neither of these differences approach significance, p >.30

15. Equity impact: F2011 Baseline Equity Gaps for 2-year rates of achievement
Our equity gaps are typical of equity gaps seen at most California Community Colleges…

16. Equity impact: F2012 Pathways 2-year rates of achievement
Our equity gaps are typical of equity gaps seen at most California Community Colleges…

17. How might this change how we understand college readiness?
Notes: unduplicated by student but counting bottom category first each time. Majority of the LBCC placement is the alternative placements.

18. What was gained through evidence based approach to transition to college
Dramatic increases in students attaining early educational milestones, & shorter times to do so
New discussion of research and instructional pedagogy, kick-starting experimentation and innovation
Strong challenges to conventional wisdom and perceptions of students by administration, staff, faculty, and students themselves
Saved thousands of units of unnecessary remediation
Reminder of the forgotten second instruction of Daedalus
We keep on using these tests. I do not think they mean what we think they mean…
Just as important not to fly too low.
Concrete achievable steps that any college can take to dramatically improve all of our students’ futures.
Can your college do this research?
Can this research help your students too?
Can your college afford to implement something like this?
Making the primary changes to assessment and placement are not only virtually cost-free – it can save your college money and direct its resources more effectively and improve outcomes
Low cost, outcome accelerating, potential long-term savings – can your college afford not to try?
We’ve built a model, made it available through the work of the RP Group, and have already been providing support to over 20 California Community College

19. Contact Information
Research questions/data requests
John Hetts, Educational Results Partnership
ext. 208
cell
General questions about Promise Pathways or Long Beach College Promise
Alicia Kruizenga, Director of School Relations and International Education:
(562)

20. Additional Resources Background research
Achieving the Dream/Jobs for the Future summary of alternative assessment
bit.ly/AlternativeAssessment
CCRC research on Assessment, Placement, and Progression in Developmental Education
bit.ly/CCRCAssess and bit.ly/CCRCDevEd
RP Group’s Student Transcript-Enhanced Placement (STEPS) Project
bit.ly/RPSTEPS
Step by Step process for replication: bit.ly/RPSTEPS2
More information about our research
bit.ly/PathwaysResearch
Similar CCC research and implementation:
Peralta CCD: bit.ly/LaneySTEPS2, bit.ly/PeraltaSTEPS
Grossmont-Cuyamaca: bit.ly/Grossmont
State-level redesigns of assessment and placement
North Carolina – 2.6 GPA & 4 years of math transfer-level
Florida – developmental educational optional
Virginia – re-design of assessment in, modularization of math
Connecticut – dev. education cut to 1 semester, coreq only
Hawaii CCs- pilots using HSGPA in assessment
Colorado, Montana, Mississippi* - move to corequisite models
National level attention on assessment and placement
2013: US Governmental Accountability Office report on Developmental Education in Community Colleges
2014: Center for the Analysis of Postsecondary Readiness established by CCRC, MDRC