1. Kathy Booth, RP Group John Hetts, Long Beach City College Andrew LaManque, RP Group What Role Does Transcript Analysis Play in Multiple Measures Assessment? Regional Research Groups October 19, 2012

2. Session Outcomes 1. Clarify the current status of common assessment efforts in the state 2. Explain how one college used an analysis of transcript data to change its placement practices 3. Share preliminary findings from a study replicating this analysis at 20 colleges What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012 2

3. STATEWIDE EFFORTS ON COMMON ASSESSMENT Building common systems for sharing assessment data 3 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

4. 4 SSTF Recommendation 2.1 Community colleges will develop and implement a common centralized assessment for English reading and writing, mathematics, and ESL that can provide diagnostic information to inform curriculum development and student placement and that, over time, will be aligned with the K-12 Common Core State Standards (CCSS) and assessments. What’s Completion Got to Do with It? – April 2012 Statewide Efforts on Common Assessment: the Task Force

5. Statewide Efforts on Common Assessment: Some History 5 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012 AB743 Centralized Assessment Bill signed by the Governor in 2011 but a permanent funding source has yet to be identified. 2012 TTAC Recommendations, April 2012 Develop a universally available assessment system that provides access to assessment results, as well as K-12 and CCC transcript data, to inform local placement. CCC Assessment Committee, June 2012 Bring together an ad hoc workgroup to flesh out the details of the data warehouse and bring back the results to this committee. This should include the crosswalk as well.

6. Statewide Efforts on Common Assessment: TTAC 6 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012 The topic of centralized assessment has highlighted the importance of utilizing a variety of data points in the process of student assessment and predictive behaviors. Since an ongoing funding source has yet to be identified, the Centralized Assessment Advisory Committee approved the plan to pursue the development of a centralized data warehouse The idea of predictive analytics offers additional possibilities for the area of student placement and support services.

7. Statewide Efforts on Common Assessment: TTAC 7 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012 The Data Warehouse Work Group proposes to link data in phases. Phase 1 will focus on data that is accessible to the project team, Cal-PASS, EAP, CCCApply and MIS. A key challenge with these data sets will be to link student records. The project team recognizes the value of transcript information among community colleges and will work to develop a plan for collecting college transcript information. K-12 transcript information is dependent upon K-12 institutions implementing an electronic transcript system; currently no K- 12 institution has a system in place.

8. 8 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012 Source: Tim Calhoon Statewide Efforts on Common Assessment

9. Q & A What questions do you have about the statewide work on common assessment? Use the chat box on the left side of your screen to type in questions. What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

10. HOW THINGS GOT STARTED Looking closer at assessment and placement 10 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

11. LBCC’s Research Sought local answers to 3 questions: What predicts how students assess and place into developmental courses? What predicts how students perform in those courses? How well are placement and performance aligned? 11 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

12. Our Research Five longitudinal cohorts tracking more than 7,000 high school graduates who attended LBCC from a large local unified school district Simplified presentation of utility of 11th grade California Standards Test (CST) scores and high school grades for predicting: How students are assessed and placed into the levels of our developmental skills sequences How students perform in those classes 12 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

13. Alignment in English * p <.05 **, p <.01, *** p<.001, x = p< 1 x 10 -10 13

14. Alignment in Math * p <.05 **, p <.01, *** p<.001, x = p< 1 x 10 -10 14

15. The Take-Aways Standardized tests best predict standardized tests Classroom performance best predicts classroom performance More information tells you more about a student than less information. 15 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

16. Evidence-Based Change: Fall 2012, Promise Pathways 966 students who completed matriculation by May 5 received Pilots predictive placement model in English & math using multi-method, evidence-based assessment of students’ preparation and readiness Prescriptive, full-time course load  Emphasis on early completion of foundational skills Taken together, what does this mean for our students? Elephant in the subway slides 16 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

17. English: Full Implementation of Placement and Prescriptive Enrollment Alternative vs. Traditional Placement 17 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

18. Math: Partial Implementation of Placement (but only partial prescriptive enrollment) 18 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

19. Placement is Not Enough: Full-Time Students 19 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

20. What LBCC Gained By Using Multiple Measures Gives students earlier but appropriate access to transfer level courses Increases the number of students that attain meaningful educational outcomes & decreases time to do so Challenges conventional wisdom about students’ readiness for college Opens new avenues of discussion about research and kick- starts opportunities for experimentation and innovation Provides concrete steps that any college—like yours—could take to dramatically improve all students’ futures 20 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

21. Q & A What questions do you have about Long Beach City College’s research and placement changes? Use the chat box on the left side of your screen to type in questions. What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

22. A BRIEF OVERVIEW The Student Transcript- Enhanced Placement Project (STEPS) 22 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

23. What Are We Trying to Do? Examine the value of using transcripts as part of the assessment process Create predictive models using Cal-PASS data to study students who had already taken community college courses The models analyze how well transcript data predicts the first English & math courses students take and how well they do in them Recruit colleges to do local analyses to generate campus-specific insights and trigger CCC/K-12 conversations 23 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

24. Related Work Brown & Conely. (2007). Comparing state high school assessments to standards for success in entry-level university courses. Brown & Niemi. (2007). Investigating the alignment of high school and community college assessments in California. Belfield & Crosta. (2012). Predicting Success in College: The Importance of Placement Tests and High School Transcripts. Willett, Hayward, & Dahlstrom. (2007). Leveraging the CST’s. Wurtz. (2007). Relations among high school math performance and college placements and achievement. Fuenmayor, Hetts, & Rothstein. (2011). Assessing Assessment: Evaluating Models of Assessment and Placement. English Curriculum Alignment Project (ECAP) between Grossmont College and Grossmont UHSD. 24 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

25. What Is Happening Now? Missing data weakened the predictive models (this highlights the value of strengthening data sharing among segments) Most colleges are still working on their analyses Bottom line: So far, high school performance is partially predictive of college performance 25 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

26. In English tests predict tests, grades and courses matter but vary by college 26

27. 27 Cox & Snell pseudo R-square ~ 0.35 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

28. In math, the specific classes taken in high school predict placement most strongly 28

29. 29 Cox & Snell pseudo R-square ~ 0.50 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

30. In English, grades predict grades 30

31. 31 Cox & Snell pseudo R-square ~ 0.20 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

32. In math, success predictors vary by college 32

33. 33 Cox & Snell pseudo R-square ~ 0.20 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

34. What Happens Next? Due to challenges with missing data, additional techniques will be applied with participating colleges We are launching a larger study using Cal-PASS data that will help answer additional research questions about using transcripts for assessment 34 What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

35. What Can You Do? Replicate this research at your campus http://www.rpgroup.org/projects/STEPS or http://bit.ly/RPSTEPS http://www.rpgroup.org/projects/STEPShttp://bit.ly/RPSTEPS Learn more about research on placement exams “Where to Begin? The Evolving Role of Placement Exams for Students Starting College” – Jobs for the Future/Achieving the Dream http://bit.ly/AlternativeAssessment “Where to Begin? The Evolving Role of Placement Exams for Students Starting Collegehttp://bit.ly/AlternativeAssessment Find out more about Long Beach’s efforts http://www.rpgroup.org/resources/promising-pathways Find out more about the previous Cal-PASS study http://www.rpgroup.org/content/2009-rp-statewide-research- award-winner-using-california-standards-test-identify-remediation What Role Does Transcript Analysis Play in Multiple Measures Assessment? | October 2012

36. Find Out More Terrence Willett, RP Group twillett@rpgroup.org John Hetts, Long Beach City College jhetts@lbcc.edu 36