2. Promise Pathways: 2012-2013 Cohort Board of Trustees Retreat July 24, 2012 The Utility of High School Achievement Data for Placement in Foundational Skills Courses John Hetts Mark Taylor Andrew Fuenmayor Karen Rothstein Long Beach City College Daniel Lamoree Mt. San Antonio College RP Group Annual Conference, April 1, 2013 1

3. Five goals 1.Initiate reflection on student transition to college, assessment, & placement. 2.Describe our research. 3.Introduce its implementation: F2012 LBCC Promise Pathways. 4.Briefly showcase tools Andrew, Dan, and Karen have built to help you. 5.Help collectively unlock the potential to better understand our students and improve our colleges. 2

4. Transition to College: Assessment and Placement CCCs are open enrollment Requires assessing and planning for students’ educational needs. Most colleges rely almost entirely on standardized assessment Ease of administration Transparency of administration Impartiality of administration Pre-approval 3

5. Consequences for student transition to college Majority of students placed into basic skills Placement is significant barrier to completion Colleges’ first interaction with students is “You are not ready for college” Courses take first few weeks to convince students Substantial long-term systemic changes: More, longer, slower remediation sequences Reduction of access, especially at bottom of sequences 4

6. Conventional Wisdom Explaining Assessment Results It is a problem with today’s students – Students are simply, vastly unprepared for college – Kids these days …. It is a problem with public education – Public education is failing to prepare high school students – Teachers these days… 5

7. What If the Conventional Wisdom is Wrong? Flynn Effect – substantial, long-term increase in IQ 18-24 year olds with HS degree: ~90% NAEP: Scores are at or near highs in virtually every age category for every demographic group Growing body of research questions effectiveness of standardized assessment for placement – Little relation to college course outcomes. (e.g.,. Belfield & Crosta, 2012; Edgescombe, 2011; Scott-Clayton, 2012; Scott-Clayton & Rodriguez, 2012; Xu, forthcoming) – NAGB, 2012: Highly variable cutscores, with 2-year colleges often using HIGHER cutscores than 4-year 6

8. We sought local answers to three questions: What predicts how students assess and place into developmental courses? What predicts how students perform in those courses? How well are placement and performance aligned? 7

9. Our Research Five longitudinal cohorts tracking more than 30,000 students built with help of Cal-PASS Focusing on 7,000 HS grads who attend LBCC directly after high school Examined predictive utility of wide range of high school achievement data most notably 11th grade California Standards Test ( CST ) scores and high school grades Assessment and placement into developmental skills courses Performance in those classes 8

10. Alignment in English * p <.05 **, p <.01, *** p<.001, x = p< 1 x 10 -10 9

11. Alignment in Math * p <.05 **, p <.01, *** p<.001, x = p< 1 x 10 -10 10

12. The Key Takeaways We want to use assessments that predict how students will perform at our college – Standardized tests predict standardized tests – Classroom performance predicts classroom performance – More information tells us more about our students than less information Significant opportunities exist to improve placement, student achievement, and students’ college experience 11

13. Reimagining the Transition to College: Fall 2012 Promise Pathways Predictive placement model in English & Math using multi-method, evidence-based assessment – English : A or B in 12 th Grade English (as proxy) – Math : Predicted rate of success using all variables ≥ average success rate in course Prescriptive, full-time course load via pre- populated first-semester success plan – Emphasis on English, Reading, Math Required group Counseling workshop Student success course Additional Experimental pilots (Contextualized Reading, academic achievement coaches) 12

14. Alternative placement: Transfer-level Placement Rates 13

15. First-term early educational milestone achievement 14

16. Success rates in transfer-level courses Fall 2012 Neither of these differences approach significance, p >.30 15

17. How can it affect time to college- level work? 1270 years of student time saved, or approximately 2.6 years per student 16

18. Can it be done? Can your college do this research? Could this model apply at your college too? Can your college afford to implement something like this? 17

19. If you’re interested in doing this research too: When you start this research: Step 1: Acquiring Cal-PASS Data Step 1 Log in at mycalpass.org and click on the data menu. 18

20. Raw Cal-PASS Data 19 Step 1

21. Downloading and Running the STEPS 2.0 Module 20 Step 2

22. How is STEPS 2.0 different? Interface – Redesigned the initial STEPS Database using VBA, SQL, and Shell scripts to fully automate data management – Forms to facilitate data validation High School Course Id (CBEDS code) Prior to College Level (MIS: CB21) To previous Spring to Fall direct matriculants, dataset adds students: – enrolled in summer term immediately after graduation – enrolled for first time in subsequent terms – not enrolling in MATH or ENGL in their first term at your institution *Note: You may restrict the dataset to fully mirror the original methodology! 21 Step 2

23. Who is in the dataset? 1 Students from your local feeder high schools 2 Their most recent high school academic year (MATH or ENGL) 3 Their most recent high school term (MATH or ENGL) 4 Their highest high school course within subject (CBEDS) 5 Their highest grade 6 Their most recent term at your institution 7 Their lowest level (CB21) course within ToP code (MATH or ENGL) 8 Their highest grade 22 Step 2

24. Final Analysis: Placement 23 Step 3

25. Predicting Placements 24 Step 3

26. Final Analysis continued: Performance 25 Step 3

27. Predicting Success in English 26 Step 3

28. Using the Model to Develop Alternative Placements 27 Step 3 To compute this measure we capture all the coefficients from our Predicting Success model.

29. Using the Model to Develop Alternative Placements (continued) 28 Step 3.6.6.6.6.4

30. So, can it be done? Can your college do this research? – YES! (Still skeptical? Come see Dan demo his even newer and more improved tool at 1:45 tomorrow in Mountain Vista ) Could this model apply at your college too? – YES! (Not sure? Join us for a panel discussing similar research and two statewide pilots, 10:15 AM Campus Vista ) Can your college afford to implement something like this? – Can your college afford not to? But wait, there’s more… – Tools to help understand effects of other interventions – Broad new understandings of our students and education 29

31. Improved tools to understand student success interventions (Come find out more at 9AM, Valley Vista tomorrow ) 30

32. Powerful start but self-selection questions remain. However, if supplemented by data generated by these tools… Bp Passed target course (not controlling for high school) 0.1240.039 (significant) Passed target course (controlling for high school GPA) 0.0370.550 (not significant) Outcome: Success in courses in the following term 31

33. Possible effects on how we understand college readiness 32

34. What did LBCC gain through prescriptive, evidence based approach to transition to college – Dramatic increases in students attaining early educational outcomes, & shorter times to do so – New discussion of research and instructional pedagogy, kick-starting experimentation and innovation – Strong challenge to conventional wisdom and perceptions of our students by administration, staff, faculty, and students themselves – Most importantly, concrete achievable steps that any college can take to dramatically improve all of our students’ futures 33

35. Can you get in touch with us? General questions – Mark Taylor, mtaylor@lbcc.edu or (562)938-4206mtaylor@lbcc.edu Research questions – John Hetts, jhetts@lbcc.edu or (562) 938-4052jhetts@lbcc.edu STEPS Participation – Terrence Willett, twillett@rpgroup.org or (510) 527-8500, ext. 109twillett@rpgroup.org STEPS syntax/output questions – Andrew Fuenmayor, afuenmayor@lbcc.edu, or (562) 938-4133afuenmayor@lbcc.edu STEPS 2.0/Illuminous questions? – Dan Lamoree, DLamoree@MtSAC.edu or (909)274-4708DLamoree@MtSAC.edu 34

36. Resources More information about our research – http://bit.ly/PathwaysResearch http://bit.ly/PathwaysResearch Achieving the Dream/Jobs for the Future summary of alternative assessment – http://bit.ly/AlternativeAssessment http://bit.ly/AlternativeAssessment CCRC research on Assessment, Placement, and Progression in Developmental Education – http://bit.ly/CCRCAssess http://bit.ly/CCRCAssess More information about the RP Group’s Student Transcript-Enhanced Placement (STEPS) Project – http://bit.ly/RPSTEPS http://bit.ly/RPSTEPS Step by Step process for replication – http://bit.ly/RPSTEPS2 http://bit.ly/RPSTEPS2 35

37. 36

38. Completion Rates: English

39. 38 Completion Rates: Math