Let Icarus Fly : Multiple Measures, Assessment, and the Re- imagination of Student Capacity John J. Hetts, Ph.D. Senior Director of Data Science Educational.

Slides:



Advertisements
Similar presentations
East Bay Career Pathways Regional Convening #2 December 4, 2014
Advertisements

Promising pathways Understanding and predicting preparation and success and its potential as a model for change for the California Community Colleges and.
Early Assessment Program: A Valid Solution for CCC Math and English Placement? KC Greaney, Director Jeanne Fadelli, Research Analyst Santa Rosa Junior.
College Credit Plus KEY ADVISING MESSAGES & MATERIALS.
VALUE – ADDED 101 Ken Bernacki and Denise Brewster.
INVESTIGATING MILESTONES AND COMPLETIONS AT A LOCAL INSTITUTION BASED ON ADELMAN’S STUDY, “THE TOOLBOX REVISITED.” What’s in YOUR toolbox? DANIEL MARTINEZ,
Principles of Reform Bruce Vandal, Education Commission of the States April 12, 2012.
Remedial Education Reform Bruce Vandal, Education Commission of the States September 25, 2012.
GATEWAY COURSE SUCCESS Gateway not “gatekeeper” Principles of Reform Gateway not “gatekeeper” Principles of Reform.
Principles of Remedial Education Reform Bruce Vandal, Education Commission of the States October 24, 2011.
East Bay Career Pathways Consortium Student Transitions December 4, 2014 Chabot College.
The Differential Trajectories of High School Dropouts and Graduates By: Gregory P. Hickman, Ph.D. Mitchell Bartholomew Jennifer Mathwig Randy Heinrich,
Promise Pathways: Cohort Board of Trustees Retreat July 24, 2012 The Utility of High School Achievement Data for Placement in Foundational Skills.
LET THEM IN: ILLUMINATING THE POTENTIAL TO INCREASE ACCESS, COMPLETION AND EQUITY IN COLLEGE ENGLISH AND MATH THROUGH NATURAL EXPERIMENTS Leslie Henson,
Using Growth Models for Accountability Pete Goldschmidt, Ph.D. Assistant Professor California State University Northridge Senior Researcher National Center.
Why I-BEST In Washington state, over half of the students come to our community and technical college system with the goal of getting to work. SBCTC research.
1 Predicting Success in Math: The relationship between high school courses and remedial math in college Brandon Lagerquist Northshore School District Paul.
Multiple Measures Assessment Project Welcome Webinar Spring 2014.
COMPASS National and Local Norming Sandra Bolt, M.S., Director Student Assessment Services South Seattle Community College February 2010.
Let Them In: Increasing Access, Completion, and Equity in College English.
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda Eloy Ortiz Oakley Superintendent-President, Long Beach City College Terrence.
Propensity Score Matching
Kathy Booth, RP Group John Hetts, Long Beach City College Andrew LaManque, RP Group What Role Does Transcript Analysis Play in Multiple Measures Assessment?
Evaluating the California Acceleration Project Equity implications of increasing throughput via curricular redesign Craig Hayward Director of Research,
Evaluating the California Acceleration Project Equity implications of increasing throughput via curricular redesign Craig Hayward Director of Research,
High School Mathematics: Where Are We Headed? W. Gary Martin Auburn University.
ASSESSING DEVELOPMENT ASSESSMENT IN COMMUNITY COLLEGES: A REVIEW OF THE LITERATURE Katherine Hughes Community College Research Center Judith Scott-Clayton.
Terrence Willett, RP Group The Potential of Transcript- Based Placement CCCCSSAA 2013 Spring Institute, Costa Mesa March 13, 2013.
PRESENTATION TITLE IN HEADER / MONTH XX, COMMUNITY COLLEGE RESEARCH CENTER MONTH XX, 2012 Presenter name, title - set in 20pt Arial Bold Based on.
Evaluating the Vermont Mathematics Initiative (VMI) in a Value Added Context H. ‘Bud’ Meyers, Ph.D. College of Education and Social Services University.
Slide 1 Estimating Performance Below the National Level Applying Simulation Methods to TIMSS Fourth Annual IES Research Conference Dan Sherman, Ph.D. American.
WEST VIRGINIA UNIVERSITY Institutional Research WEST VIRGINIA ADVENTURE ASSESSMENT Created by Jessica Michael & Vicky Morris-Dueer.
Multiple Approaches to Multiple Measures: Four Three Approaches to Improving Student Achievement Through More Holistic, Evidence-Based Multiple Measures.
Terrence Willett, RP Group The Potential of Transcript- Based Placement CCCCIO 2013 Spring Conference, San Francisco April 4, 2013.
College Preparatory Course Certification Pilot May 5th,
Student Equity and the Re-imagination of Student Capacity Building a bridge to the future of the California Community Colleges.
Increasing Gateway Course Completion by Scaling Successful Statewide Initiatives Saundra King Assistant Vice President of Remediation and Innovation April.
Sierra College ERIK COOPER. A Brief History  The RP Conference 2013  STEPS Project  Sierra College  Paul Neal & Craig Kelly  Working with the English.
The Education Trust – West Educational Opportunity Audit Report of Findings Oakland Unified School District March 25, 2009 Linda Murray Tami Pearson.
Multiple Measures Assessment Project: Math & English Models from Phase II Multiple Measures Pilot College Convening - North Cañada College Terrence Willett.
Research review and resources to help to make your case Northern California Convening of MMAP Pilot Colleges October 28, 2015.
Enhanced Multiple Measures for Placement: The Multiple Measures Assessment Project (MMAP)
English Assessment Validity February 2012 Michael Orkin, Jo Ann Phillips, Hui Zhang, Sheryl Queen Office of Institutional Research Peralta Community College.
Examining the Enrollment and Persistence of Students with Discrepant High School Grades and Standardized Test Scores Anne Edmunds, Ed.D. Higher Education.
IMPROVING SUCCESS AMONG UNDERREPRESENTED COMMUNITY COLLEGE STUDENTS* California Community Colleges Chancellor's Office Equal Employment Opportunity and.
Math Placement and Diagnostic Feedback Together! Bruce Arnold CMC-South Annual Conference November 6, 2015.
College of the Canyons’ Math PAL: Accelerating Students to Completion Presenters: Kathy Kubo, Math Faculty Pola Pardon, Student Denee Pescarmona, Dean.
MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY February 16, 2016 Craig Hayward Director of Research, Planning and Accreditation.
Likelihood of Completion Depends on Level of Unpreparedness Community College Pre-College Level Remedial Math Sequence.
B ASIC S KILLS W ORKSHOP, F EBRUARY 16, 2016 Wendy Smith Professor of English Curriculum Development Specialist San Diego Mesa College Corequisite Remediation.
Measuring Turnaround Success October 29 th, 2015 Jeanette P. Cornier, Ph.D.
Making it Happen (MIH) Strategic Partners in Student Success MIH FALL 2015: 1,600 STUDENT COHORT.
Admission and Transfer Policy Review Task Force 1.
The Hitchhiker’s Guide to Guided Pathways Redesigning Community Colleges Bakersfield College Craig Hayward John Hetts Terrence Willett
MULTIPLE MEASURES, ASSESSMENT, AND THE RE-IMAGINATION OF STUDENT CAPACITY March 2, 2016 Loris Fagioli, PhD Research and Planning Analyst, Irvine Valley.
Friday, March 18 2:30 PM – 3:45 PM Breakout Session Block II Exemplary Program Winner 2015 – Bakersfield College MIH (Making it Happen) Program Bakersfield.
Strategy 2: Corequisite Remediation — English 1A + 1 unit co-req Josh Scott, English Instructor, BSI Coordinator.
Changes in placement in the California Community Colleges John J. Hetts Senior Director of Data Science, CalPASS Plus/Educational Results Partnership Former.
The California Acceleration Project A Faculty-Led Professional Development Network Supporting the State’s 113 Community Colleges To Transform Remediation.
Acceleration: A Powerful Lever for Increasing Completion and Equity
CMC3 South Conference October 15, 2016 Loris Fagioli, PhD
Strengthening Student Success Conference October 8, 2015
Multiple Measures Pilot College Convening - North
Positioning Students with the Multiple Measures Assessment Project
Board of Trustees Review
Propensity Score Matching Makes Program Evaluation Easy
What we know about ab 705 Cheryl Aschenbach, North Representative
Student Equity Planning August 28, rd Meeting
What we know about ab 705 Cheryl Aschenbach, North Representative
Student Equity and Achievement Metrics and Student Equity Plans
Presentation transcript:

Let Icarus Fly : Multiple Measures, Assessment, and the Re- imagination of Student Capacity John J. Hetts, Ph.D. Senior Director of Data Science Educational Results #LetIcarusFly Oregon Placement Work Group October 23, 2015

Overview Standardized assessment has led us to systematically and substantially underestimate student capacity Particularly for students of color, low income students, first generation college students, women Evidence-based, multiple measures is one of four key cornerstones on which to rebuild the foundations of community college education Demonstrates fundamental capacity of far more of our students to succeed if given the chance Powerful completion, equity, and real world implications Based powerfully both on basic principles of assessment and measurement as well as strong

First, Daedalus and Icarus Daedalus crafted the labyrinth of inescapable complexity for King Minos To escape from Minos, Daedalus built wings of feather and wax for his son Icarus and himself Don’t fly too high, lest sun melt the wax and you plummet to your doom Dangers of innovation/invention, hubris, Importance of knowing your limits, listening to your wiser elders But most of us forget the rest of that story…

Transition to College: Assessment and Placement Community colleges are open enrollment institutions Requires assessing and planning for educational needs of students. Goal Effectively place student at most appropriate level for their skill – where challenge matches skill level Zone of proximal development Optimal performance, flow If you think you can catch the bus, you will run for it.” Lee Peng Yee, Singapore National Institute of Education Mathematician

Community college student transition Community colleges rely nearly entirely on standardized assessment (WestEd, 2011) Majority of students placed below college-level Significant barrier to completion (Bailey, Jeong, &Cho, 2010) 50-60% of equity gap in completions (Stoup, 2015) What does this mean? First interaction is to tell students they don’t belong Imply most students not ready for college and likely to fail Convinced nearly everyone Including our students

Conventional Wisdom Explaining Assessment Results It is a problem with today’s students Students are simply, vastly unprepared for college Kids these days ….

That seems awfully familiar

Too familiar (Bye Bye Birdie – 1963)

What If that Conventional Wisdom is Wrong? Substantial, long-term increase in IQ: bit.ly/FlynnEffectIQ bit.ly/FlynnEffectIQ with HS degree: 91% - highest ever: bit.ly/2014HS18-24bit.ly/2014HS18-24 National Assessment of Educational Progress: at all-time highs in virtually every demographic category: bit.ly/NAEPInfo bit.ly/NAEPInfo

NAEP Math and Reading Assessments

What If the Conventional Wisdom is Wrong? Research increasingly questions effectiveness of standardized assessment for understanding student capacity Little relation to college course outcomes (e.g., Belfield & Crosta, 2012; Edgescombe, 2011; Scott-Clayton, 2012; Scott-Clayton & Rodriguez, 2012): bit.ly/CCRCAssess bit.ly/CCRCAssess Incredible variability in cutscores and 2-year colleges often use HIGHER cutscores than 4-year bit.ly/NAGB2012 Underestimates capability of students of color, women, first generation college students, low SES Hiss & Franks, 2014; bit.ly/DefiningPromisebit.ly/DefiningPromise

What if? What if the problem is not primarily with our students but with limitations in how we have assessed and understood their capacity to do college-level work?

It gets worse What if this flawed method of understanding and “remediating” student capacity has actually had the opposite effect? Imagine yourself arriving at college as a community college student…

But there’s good news… What if one of the key barriers to our students’ successful transition to and success in college is one that we fully control?

Cornerstone 1: Improving assessment through evidence- based multiple measures Resources/references:

LBCC Multiple Measures Research Five cohorts tracking more than 7,000 HS grads who matriculate to LBCC directly Examined predictive utility of wide range of high school achievement data For predicting: How students are assessed and placed How students perform in those classes (and alignment between them)

Alignment in English * p <.05 **, p <.01, *** p<.001, x = p< 1 x

Alignment in Math * p <.05 **, p <.01, *** p<.001, x = p< 1 x

Key Takeaways Assessment should predict how students will perform at our colleges Instead: Current standardized tests predict standardized tests Classroom performance predicts classroom performance More info tells us more about student capacity than less info

Re-imagined student capacity Starting in Fall 2012, students from LBUSD (now 6 districts covering >30 high schools and growing) were provided an alternative assessment Reverse engineered the analysis to place students using: Last high school course in discipline Grade in last course in discipline Overall HSGPA Last standardized test in discipline (and level) Placed students in highest course where projected success rate higher than average success rate for that course.

Implementing Multiple Measures Placement: Transfer-level Placement Rates F2012

But …

… didn’t that just flood transfer- level courses with unqualified students?

Comparison against students in advancing through sequence: Success rates in transfer-level courses Neither of these differences approach significance, p >.30

Cohort 1 English 1 Success Rates by Placement (vs. 6 year completion)

Comparison against traditional sequence: Success rates in transfer-level courses English difference, p <.001

Overall Success Rate in Transfer Level English by Method of Qualification (among students with high school data available)

Success Rate by Method of Qualification in Transfer Level English

… what about grade inflation in HS?

HS GPA Distribution from MMAP dataset

ACT research on grade inflation Little evidence for grade inflation over last decade Earlier observations of grade inflation may have been partly artifactual adjustments to GPA for AP/IB/Honors Zhang & Sanchez, 2014:

Concerns about grade inflation and social promotion do not fit evidence Suggests that there should be little to no relation between HS grades and college grades because HS grades unrelated to performance Everyone gets As and Bs would mean no variation to predict outcomes Yet, predictive utility strongly observed Stronger than standardized tests

Westrick & Allen, 2014: ACT COMPASS Validation Median Logistic R (Table 4) CourseCompass TestCompassHSGPA HSGPA + Compass English 1Writing Skills ArithmeticPre-Algebra AlgebraPre-Algebra Intermediate AlgebraAlgebra College AlgebraAlgebra College Algebra

Compass Score (30 extremely low to 90 extremely high) High School GPA Westrick & Allen, 2014: Conditional Success Rates (Table 6) Compass Score (30 extremely low to 90 extremely high) High School GPA %28%32% %49%55% %70%75%

… didn’t that work only because Long Beach is special/has special relationship between LBCC and LBUSD?

Not just Long Beach LBCC now includes multiple additional districts including ABC Unified and Los Alamitos Unified Long thread of research in the CCCs alone 2008: Willett, Hayward, & Dahlstrom 11 th grade HS variables as early alert mechanism for discipline assessment 2011: Martinez self-reported HS variables as more powerful predictors of college completion 2014: Willett & Karanjeff replication of LBCC research with 12 additional colleges (STEPS) Replication of implementation Bakersfield College and Sierra College began similar implementation in 2014 CCRC research (Belfield & Crosta, 2012; Scott-Clayton, 2012) MMAP Statewide Research & local replications: bit.ly/MMAP2015 bit.ly/MMAP2015

CA Multiple Measures Assessment Project (MMAP) Examination of high school achievement data for predictors of successful completion of English & math in the CCCs Focus on predictive validity (success in course) and improving student completion of foundational skills Statewide support Research base, predictive analytics, decision tree models Pilot colleges and faculty/staff engagement Webinars, convenings/summits, professional development K-12 outreach and data population Data warehouse and tool development

High school variables that predict college success English Cumulative HS GPA Grade in last HS English C+ or better in AP English class Score on English CST Non-remedial status in HS English Math Cumulative HS GPA Enrollment and grades in Geometry, Algebra II, Trigonometry, Pre-calculus, Statistics, Calculus Taking a more challenging CST Score on math CST Delay*

Sample Transfer-level decision rules English 11 th Grade High School GPA ≥ 2.6 Math (Statistics) 11 th Grade High School GPA ≥ 3.0 & Algebra I C or better OR 11 th Grade High School GPA ≥ 2.3 & Pre-calculus C or better

MMAP: Projected impact on course success rates

SDCCD MMAP F2015 Pilot (N = ~1000)

Sierra College F2014 Transfer-Level English Success Rates

… we’re happy with our placement. Why should we change?

Powerful reasons for change: 1) Basic assessment theory and methods Self-reported satisfaction with assessment by instructors and students is most common measure Has grave methodological flaws: Selection bias Confirmation bias Effort justification System justification Self-fulfilling prophecy effects

Powerful reasons for change: 1) Basic assessment theory and methods Principles of assessment/measurement theory: Measurement = True score + systematic error + random error Methodological gold standard of assessment- triangulation to true score via assessment across: methods content domains evaluators Time GPA assesses capacity assessment methods, content domains, dozens of instructors, and across time Combined with other available evidence/assessments increases quality further

Powerful reasons for change: 2) It’s poorly assessing students Substantial evidence of systemic & severe underplacement placing students in developmental education who could get a B or better in the transfer level course 36% percent of students placed into development English 25% of students placed into developmental Math Using multiple measures reduces error, has potential to increase success rates and sequence completion

Powerful reasons for change: 3) Transformational impacts for students Potential for dramatic increases in rates and time to completion of Transfer-level course in discipline Subsequent courses in discipline Other early educational milestones.

F2012 Promise Pathways vs. Fall year rates of achievement

Equity impact LBCC: F2011 Baseline Equity Gaps for 2-year rates of achievement

Equity impact LBCC: F year rates of achievement

… what about students for whom high school transcript data aren’t available/easy to get?

Self-reported HSGPA appears to be reliable alternative College of the Canyons Research (Gribbons, 2014) Self-report of last course and grade in Fall term very accurate Errors that do occur in part because of timing of assessment University of California admissions Uses self-report HSGPA but verifies after admission 2008: 9 campuses, students. No campus had more than 5 discrepancies b/w reported grades and student transcripts: Much of the ACT research uses self-report GPA and finds it to be a more powerful predictor than students actual scores on the standardized tests ACT, 2013: r(1978) =.84

ACT, 2013: Actual HSGPA Level N Mean HSGPA Mean diff. Accuracy Actual Student- reported % within 0.25 % within – –0.0487%98% 3.00– –0.0160%90% 2.50– %82% 2.00– %73% 1.50– %55% 0.00– %35% Total1, %83%

… what about non-traditional students?

Multiple measures continues to have utility for delayed matriculants Delay in matriculation is a main effect i.e., likelihood of success decreases somewhat Does not interact with HSGPA or course grades “A” students at a delay definitely better than “B students” at a delay “A” students at a delay remain better than “B” Likely grade drops but still very likely to be successful HSGPA continues to be predictively useful up to the point where we have data we can meaningfully connect(delay of ~10 years).

How long is High School GPA good for?

Westrick & Allen, 2014: ACT COMPASS Validation Standardized Logistic Regression Coefficients(Table 5) Course Compass TestStudent TypeCompassHSGPADiff English 1Writing SkillsTraditional Nontraditional ArithmeticPre-AlgebraTraditional Nontraditional AlgebraPre-AlgebraTraditional Nontraditional Int. AlgebraAlgebraTraditional Nontraditional Coll. AlgebraAlgebraTraditional Nontraditional Coll. Algebra Traditional Nontraditional

… that doesn’t seem like enough evidence that I need to rethink student capacity. What else have you got?

Cornerstone 2: Reconsider cut scores Resources/references: (Henson & Hern, 2014) (Kalamkarian, Raufman, & Edgecombe, 2015) (Rodriguez, 2014)

Natural experiment at Butte College In 2011, switched from one placement test to another Old test/cut scores: 23% of incoming students “college ready” in English New test/cut scores: 48% of incoming students “college ready” in English

Butte College: Assessment of first-year students

Butte College: Completion of Transfer-Level English in 1 st Year

Developmental Math Reform – Virginia Community College System Introduced new assessment instrument Intentionally increased percentage assigned to college-level math

Additional bodies of work showing higher student capacity Acceleration (e.g., Hayward & Willett, 2014) Corequisite developmental education (e.g., Coleman, 2015) 2-4X transfer-level course completion Comparable or higher success rates Works across demographic group Reduces equity gaps substantially

Evaluation of pilot year of California Acceleration Project Summary of Findings (Hayward & Willett, 2014) Large and robust effects of acceleration that work for Students of all backgrounds Students at all placement levels Not a function of selection/cherry-picking Examples from Math

Regression Adjusted Effects – Math

Completion of transfer-level math for traditional and accelerated pathways by ethnicity

Replication of CCBC ALP model ALP model involves: Enrollment directly in college-level English (mainstreamed) Concurrent enrollment in just-in-time companion developmental English course taught by same instructor Four early implementers at or near scale

Completion of College-Level English (of those who take one-level below course)

Combining cutscore revision and corequisite expansion in English - VCCS

Completion of College English - VCCS

Summary of impacts of approaches that re-imagine student capacity Transfer-level success rates (if taken) Developmental Success Rates Transfer-level completion (by entire cohort) Meaningful equity impacts Upfront Development of Curriculum Multiple Measures No change to higher Lower overall (but no change for students that remain) Much higher Substantial Low Acceleration No change to higher Much Higher Substantial High Moderate Corequisite models Higher Much Higher Substantial High Moderate Cutscore revision Slightly lower No change to slightly lower Much Higher Substantial Low

What might this mean for students? Bakersfield College has saved over 3000 semesters in first two years LBCC saved students over 10,000 semesters (5000 years!) of unneeded remediation in first three years. $250 per course for student (plus books!), $750 per course for state Dramatic opportunity costs of college reduced Median 2012 salary of “some college” is ~$30,000/year Don’t lose their first year or median salary though, they lose their last year.

What might this mean for all of us? These evidence-based cornerstones save students 1-2 semesters of developmental education that: Evidence predicted and research demonstrated that they did not need By law (in some places), fairness, and basic educational practice should not have been required to take

The Great Recession of 2008 took ,000 people of Oregon out of the workforce of Oregon Failure to correctly understand the capacity of Oregon’s community college students may be costing those140, ,000 students a year or more of additional time out of the workforce. Sense of the scale

What can be gained through assessment and placement reform A clarion call to better understand the true capacity of our students. The ability to transform student outcomes Powerful levers to address student equity gaps Renewed opportunities to collaborate with our K-12 educational colleagues The chance to stop meeting students at front door to tell them they don’t belong A far better future for our students and for us all A reminder of Daedalus’ second instruction to Icarus. It’s just as important not to fly to low.

Thank you! Contact information John Hetts Educational Results Partnership ext cell