1. The Most At-Risk First Time in College Students: Using Assessment Techniques to Identify and Assist Them Prepared for the International Assessment and Retention Conference, Assessment Reconsidered: Improving Learning, Student Success, and Transparency, 11–15 June 2008. Scottsdale, Arizona. Bernadette M.E. Jungblut, Ph.D. – Assistant Director Daniel Suleski – Coordinator of Statistical Research Brittany L. Resmann – Coordinator of Assessment and Research Office of Assessment and Planning Student Development and Enrollment Services University of Central Florida Orlando, FL 32816-0160

2. Workshop Goals Discuss the methods used to predict which First Time in College (FTIC) students are the most likely to be at risk for retention failure in their first year at UCF Describe a targeted intervention, the Knight Success Program, designed to assist these students and increase the likelihood of their engagement and persistence Provide session participants with several opportunities to:  Discuss the applicability of these methods at your institutions  Discuss the generalizability of this type of intervention to your institutions  Examine the efficacy of this approach in your specific institutional contexts

3. Workshop Agenda Introduction Short activity to assess participants’ needs and primary interests UCF Project Overview Discussion of applicability to participants’ institutions UCF Project Findings and Next Steps Discussion of participants’ expectations about at-risk students UCF Program Components Discussion of program’s generalizability to other institutions

4. Short Activity: Questions to Consider Are you currently identifying at-risk incoming students? If so, what techniques are you using to identify those students? Who are your at-risk incoming students? If not, do you plan to do so in the future – and what factors do you expect to be “likely suspects” to help you identify these at-risk students? Do you currently have targeted intervention programs in place for at-risk incoming students? If so, choose one (1) to focus on for this activity.  How is this program structured? What are the program’s primary components?  How is this program staffed?  How are you recruiting students for this program?  How are you connecting and communicating with these students? If you do not have targeted intervention programs in place for at-risk incoming students, are you planning to design and implement such programs?  What components would you include in such a program?

5. Facts about UCF Largest undergraduate enrollment and second largest total enrollment in Florida Sixth largest university in the country by total enrollment; second largest in the country in terms of undergraduate enrollment Fall 2007 Enrollment: 48,669 2007-2008 FTIC Applications: 26,986 2007-2008 FTIC Applications Accepted: 13,406 (49.7%) 2007-2008 New FTICs Enrolled: 6,717 Sources: UCF Office of Undergraduate Admissions US Department of Education Office of Postsecondary Education

6. Enrollment Trends Enrollment of first-time college students at UCF continues to increase while the number of traditional first-time students beginning in Fall is capped Enrollment increasingly accommodated through Summer admission

7. Quality of Incoming Students Average SAT score of incoming first-time students continues to increase Slight dip in 2005 coincides with introduction of the SAT writing section in March 2005 Average High School GPA has also increased over time Dip in 2005 coincides with a State of Florida reduction in the weighting of AP and dual enrollment coursework

8. Knight Success Program – Motivation Improvement in first-year retention stagnated despite improving admissions profile. What can be done to improve first-year retention when simply admitting “better” students no longer seems to work?

9. Knight Success Program – Development Hypothesis: The university can enhance its first-year retention rate by engaging students who are most at-risk for dropping out after their first year. Action Plan: Develop a program to engage our most at-risk students. Problem: How do we know which students are the most at-risk before they begin their college career?

10. Knight Success Program – Development Solution: Use what we already know about our incoming students to formulate a predictor of first-year retention. Resources available: Information new students provide to admissions Information current and former students have provided to admissions, along with their retention outcomes

11. Knight Success Program – Data What does the university know about students before they arrive on campus? Standardized test scores High school grades and coursework attempted Class rank and percentile Financial aid On-campus housing requests Personal demographics (age, gender, ethnicity, etc.) Residency (in-state, out-of-state, international) Name and type of high school (public/private/home school/etc.) College credits earned (Advanced Placement, Dual Enrollment, International Baccalaureate, etc.) Major student is pursuing (or undeclared status)

12. Factors to Examine at Your Institution Which of these factors are relevant to your institution? Which are not relevant or applicable? What other factors would you need or want to include in your models?

13. Knight Success Program – Data Analysis Model building: Use information from past admissions and outcomes to construct, test, and validate predictive models. Model selection: Choose the model with the highest hit rate. Hit rate: Total correctly predicted non-retained students divided by the total number of students not retained

14. Knight Success Program – Modeling Entropy split criterion found to be the best model

15. Knight Success Program – Decision Rules Decision Rule 1: - High school GPA less than 3.25 - About 30% of 1,000 most at-risk students Decision Rule 2: - Total SAT score is greater than or equal to 1240 - High school GPA is between 3.25 and 4.15 - Student is a National Merit Scholar Finalist - About 10% of 1,000 most at-risk students Decision Rule 3: - Total SAT score is greater than or equal to 1000 - SAT math score is greater than or equal to 460 - High school social sciences unit GPA is greater than 4.05 - High school English unit GPA is greater than 4.75 - About 60% of 1,000 most at-risk students

16. Knight Success Program – Results Retention Outcomes KSP-selected students have a lower retention rate than non- selected students. The KSP model is an effective predictor of at-risk students. KSP participants have a higher retention rate than non- participants. KSP participation is effective in improving first-year retention.

17. Other Student Outcomes of KSP Academic Probation Students selected for KSP are typically more likely to be placed on academic probation than non-selected students. Participants in the KSP program are less likely to be placed on academic probation than non-participants.

18. Other Student Outcomes of KSP Term GPA Students selected for KSP typically have lower term GPAs than non-selected students. Participants in the KSP program typically have higher term GPAs than non- participants.

19. Next Steps Comparison of data mining models with deductively derived logistic regression models More in-depth examination of additional factors  FCAT scores (Florida’s state-wide assessment of high school students)  First generation in college status  Socioeconomic factors including financial aid status  AP, IB, and Dual Enrollment credits earned in high school  High school curricula and credits earned by subject area CSRDE NSSR Follow-up Presentation: 29 September–1 October

20. KSP Components Years 1 and 2 Year 1 (AY 2005-2006)  August 2005 start (for September 2005 program launch)  Identified 1000 students; recruited 200 active participants  Incentives provided (bookstore coupons)  Distributed among all 12 first-year advisors  Skills assessments for 200 active participants  Engagement events for all 1,000 students invited to participate Year 2 (AY 2006-2007)  May 2006 start (for summer and fall program infusion)  Identified 1000 students; recruited 200 active participants  No incentives provided except opportunity for skills assessment  Assigned to one of two dedicated KSP Advisors  Infused into approximately 16 Orientations (May through August)  Skills assessment = Strength Quest (for 200 students)  Engagement events  E-mail contact  SLS 1501 – Strategies for College Success course

21. KSP Components Years 3 and 4 Year 3 (AY 2007-2008)  Very similar to Year 2  Same two KSP Advisors  Added First Year Advising and Exploration (FYAE) Peer Advisors  Special Orientation sessions for KSP selected students (logistical problems) Year 4 (AY 2008-2009)  Somewhat similar to Year 3  Same two KSP Advisors  Focus on engagement and academic success strategies  Enhanced communication and contact (e-mail plus more face-to-face)  Orientation sessions revised Day 1 = KSP Cohort special sessions Day 2 = College schedule planning (to address Year 3 logistical problems)  KSP Advisors design 20- to 30-student intensive pilot program

22. Program Components Which of these KSP components are relevant to your institution? Which are not relevant or applicable? What other components would you need or want to include?

23. Questions? Bernadette M.E. Jungblut, Ph.D. bjungblu@mail.ucf.edu Daniel Suleski dsuleski@mail.ucf.edu Brittany L. Resmann bresmann@mail.ucf.edu bresmann@mail.ucf.edu

24. Research Collaborators Special thanks to our co-researchers: - Dr. Ronald H. Atwell, Director of Student Development and Enrollment Services Office of Assessment and Planning - Dr. Morgan C. Wang, Professor of Statistics and Actuarial Science and Director of the University of Central Florida Data Mining Program - For more information about this program, please visit: http://dms.stat.ucf.edu/ http://dms.stat.ucf.edu/