1. Identifying At-Risk Students With Two- Phased Regression Models Jing Wang-Dahlback, Director of Institutional Research Jonathan Shiveley, Research Analyst Office of Institutional Research Sacramento State

2. About This Study This study focuses on 1-year and 2-year retention of first-time freshman because on average 28% of the student population will drop out within the first two years. Create early and final regression models to predict 1 and 2- year retention based on data availability. Use the regression models to calculate individual student risk scores. Use results to initiate action through retention outreach efforts.

3. Trends of 1-Year and 2-Year Retention 3

4. 1-Year and 2-Year Retention by College 4

5. 1-Year Retention: Profile Table 1. 1-Year Retention Persisted after 1 yearWithdrew after 1 year GapTotal Count Statistical Significance Count%/MeanCountRate Demographic Characteristics Gender Male2,91680.2%71919.8% -3.1% 3,635 Yes Female4,30183.4%85916.6%5,160 Race/Ethnicity URM2,53781.4%57818.6% -0.9% 3,115 No Non-URM4,68082.4%1,00017.6%5,680 First Generation College Student Yes 2,41081.5%54818.5% -1.0% 2,958 No 4,44082.5%94217.5%5,382 Low Income (Pell Grant Eligible) Yes3,86482.0%85018.0% -0.2% 4,714 No 3,35382.2%72817.8%4,081 Commuter Status Living on Campus2,22381.9%49218.1% -0.3% 2,715 No Commuter4,99482.1%108617.9%6,080 Distance to School 7,21726.01,57829.8-3.88,795Yes College Readiness Need Remediation4,20479.6%107920.4% -6.2% 5,283 Yes No Remediation3,01385.8%49914.2%3,512 Remediation Type English (E) 1,33983.1%27216.9% 1,611 Yes, E > B & M Math (M)94979.3%24820.7% 1,197 Both (B)1,91677.4%55922.6% 2,475 Yes, N > B & M None (N)3,01385.8%49914.2% 3,512 Test Scores HS GPA 7,195 3.26 1572 3.120.148,767Yes SAT Verbal 6,663 471 1442 460118,105Yes SAT Math 6,663 490 1442 476148,105Yes EPT 4,647 142 1110 14015,757Yes ELM 7,217 29 1578 31-28,795Yes AP Units 729 8.6 116 8.9-0.3845No * T-test or Chi-Square Test, p<.001, higher value is highlighted in yellow; p<.01, higher value is highlighted in green; p<.05, higher value is highlighted in blue. 5

6. 2-Year Retention: Profile Table 2. 2-Year Retention Persisted after 2 yearsWithdrew after 2 Years GapTotal Count Statistical Significance Count%/MeanCountRate Demographic Characteristics Gender Male2,55770.3%1,07829.7% -2.8% 3,635 Yes Female3,77373.1%1,38726.9%5,160 Race/Ethnicity URM2,20070.6%91529.4% -2.1% 3115 Yes Non-URM4,13072.7%1,55027.3%5,680 First Generation of College Student Yes 2,11371.4%84528.6% -1.0% 2,958 No 3,89772.4%1,48527.6%5382 Low Income (Pell Grant Eligible) Yes3,38471.8%1,33028.2% -0.4% 4,714 No 2,94672.2%1,13527.8%4081 Commuting Status Living on Campus1,95171.9%76428.1% -0.2% 2,715 No Commuter4,37972.0%170128.0%6,080 Distance to School6,33025.9246528.9-3.08,795Yes College Readiness Need Remediation3,63268.7%165131.3% -8.1% 5,283 Yes No Remediation2,69876.8%81423.2%3,512 Remediation Type English (E) 1,20474.7%40725.3% 1,611 Yes, E > B & M Math (M)81668.2%38131.8% 1,197 Both (B)1,61265.1%86334.9% 2,475 Yes, N > B & M None (N)2,69876.8%81423.2% 3,512 Test Scores HS GPA 6,312 3.28 2,455 3.130.158,767Yes SAT Verbal 5,844 473 2,261 461128,105Yes SAT Math 5,844 492 2,261 476168,105Yes EPT 4,031 142 1,726 14115,757Yes ELM 6,330 29 2,465 31-28,795Yes AP Unit 638 8.4 207 9.3-0.8845No 6

7. Highlights of 1-Year & 2-Year Retention Profiles Demographic Characteristics College Readiness Among the selected 6 factors, only 2 or 3 factors had a significant impact on 1- year and 2-year retention rates. Those factors were: gender, underrepresented minorities, and distance to school. All factors but AP units had a significant impact on 1-year or 2-year retention. Remediation is a key factor: The proportion withdrawals in need of remediation were 6% to 8% higher than those who persisted.

8. 1-Year Retention: Academic Performance Table 3. Academic Performance (By the end of first year) Persisted after 1 yearWithdrew after 1 year GapTotal Count Statistical Significance Count%/MeanCountRate Term 1 GPA7,2172.971,5781.961.018,795Yes Term 2 GPA7,1572.911,2031.791.128,360Yes Pass Rate (Overall GPA>=2.0) Pass6,75191.0%6669.0% 7,417 Yes Not Pass46633.8%91266.2%57.2%1,378 Dean's List (Overall GPA>=3.0) Yes3,46192.7%2747.3% 18.5% 3,735 Yes No3,75674.2%1,30425.8%5,060 STEM Major Yes1,65282.6%34917.4% 0.6% 2,001 No 5,56581.9%1,22918.1%6,794 Major Status Declared Major3,44781.9%76118.1% 4,208 No Pre-Major2,75782.6%58117.4% 3,338 Undecided1,01381.1%23618.9% 1,249 Changed Major Changed65961.0%42239.0% -24.1% 1,081 Yes No Change6,55885.0%1,15615.0%7,714 Repeating Courses Yes47062.9%27737.1% -20.9% 747 Yes No6,74783.8%1,30116.2%8,048 Unit Completion Units Attempted71572712032618,360Yes Units per term 14 131 Units Completed71572612031888,360Yes Units per term 13 94 Overall Units71572612031798,360Yes Units per term 13 85 * T-test, Chi-Square Test or ANOVA, p<.001, higher value is highlighted in yellow; p<.01, higher value is highlighted in green; p<.05, higher value is highlighted in blue. Note: STEM majors and declared majors/pre-majors/undecided were based on status at the second semester. Major change refers to changes which occurred between the first and second semester. 8

9. 2-Year Retention: Academic Performance Table 4. Academic Performance (By the end of second year) Persisted after 2 yearsWithdrew after 2 Years GapTotal Count Statistical Significance Count%/MeanCountRate Term 3 GPA6,2432.939742.310.627,217Yes Term 4 GPA6,2242.927142.320.596,938Yes Pass Rate (Overall GPA>=2.0) Pass6,13383.7%1,19316.3% 70.3% 7,326 Yes Not Pass19713.4%1,27286.6%1,469 Dean's List (Overall GPA>=3.0) Yes2,80286.0%45714.0% 22.3% 3,259 Yes No3,52863.7%2,00836.3%5,536 STEM Major Yes1,39872.0%54328.0% 0.0% 1,941 No 4,93272.0%1,92228.0%6,854 Major Status Declared Major3,10472.9%1,15227.1% 4,256 Yes, Major & Pre > Undecided Pre-Major2,38172.2%91927.8% 3,300 Undecided84568.2%39431.8% 1,239 Changed Major Changed00.0%0 0 No No Change00.0%0 0 Repeating Courses Yes1,69872.1%65727.9% 0.2% 2,355 No 4,63271.9%1,80828.1%6,440 Unit Completion Units Attempted6224547145136,938Yes Units per term 13 1 Units Completed62245171436156,938Yes Units per term 13 94 Overall Units62245271441116,938Yes Units per term 13 103 * T-test, Chi-Square Test or ANOVA, p<.001, higher value is highlighted in yellow; p<.01, higher value is highlighted in green; p<.05, higher value is highlighted in blue. Note: STEM majors and declared majors/pre-majors/undecided were based on status at the second semester. Major change refers to changes which occurred between the first and second semester. 9

10. 1-Year Retention: Intervention Intervention: 1-Year Retention Rate of Participants and Non-Participants 10

11. 2-Year Retention: Intervention 11

12. The Development of Regression Models 12 Literature review Data availability Identify variables (up to 36) Correlation Collinearity Missing values Select variables (18-19) Early models Final models Trim Outliers Develop regression models

13. Early Model—1-Year Retention 13 Table 5 Regression Model: 1-Year Retention (Early model) Predict VariablesBS.E.WalddfSig.Exp(B) Odds Ratio (Recalculated)Rank High School GPA.260.1323.8671.0491.30 4 Fulltime (first term) -.478.1896.3861.0120.621.612 Overall GPA1.534.066546.7391.0004.6364.641 Overall Units.041.01212.1541.0001.0421.04 1st year repeater.439.1399.9241.0021.5521.553 Constant-3.044.49038.5681.000.048 Model Indicators Baseline P*82.1%Chi-Square (df)1632.985 (17) Model N5,577Pseudo R 2.254 -.442 -2log L3124.886% Correctly predicted83.3% * Refers to 1-year retention rate.

14. Final Model—1-Year Retention 14 Table 6 Regression Model: 1-Year Retention (Final model) Predict VariablesBS.E.WalddfSig.Exp(B) Odds Ratio (recalculated.)Rank Underrepresented Minority-.306.1494.1951.041.7371.36 5 Need remediation-.426.1706.2381.013.6531.53 4 High School GPA-.455.1925.6141.018.6341.58 2 Distance to the University-.002.0014.8131.028.9981.00 Learning Community-.453.2283.9311.047.6361.57 3 Overall GPA3.504.156503.5351.00033.25233.25 1 Overall Units.036.0138.4141.0041.0371.04 Constant-4.323.74833.4061.000.013 Model Indicators Baseline P*82.1%Chi-Square (df)2239.184 (18) Model N5,293Pseudo R 2.345 -.675 -2log L1551.824% Correctly predicted91.3% * Refers to 1-year retention rate.

15. Preliminary and Predicted 1-Year Retention Rate (2014 Cohort) 15

16. Prediction Results (2014 Cohort) Preliminary and Predicted 1-Year Retention Rate Early Model Preliminary Total WithdrawPersist PredictedWithdraw 241149390 Persist 44128643305 Total 68230133695 Preliminary 18.5%81.5% Predict 10.6%89.4% Differ 7.9%-7.9% Overall Correctly predicted:84% Preliminary and Predicted 1-Year Retention Rate Final Model Preliminary Total WithdrawPersist PredictedWithdraw 402180582 Persist 28028333113 Total 68230133366 Preliminary18.5%81.5% Predicted15.8%84.2% Differ2.7%-2.7% Overall Correctly predicted:87.6% 16

17. 1-Year Retention: The Differences Between The Early and Final Model 1.The Early Model can be used for early intervention purposes during mid- Spring semester. The Final Models can be used to contact at-risk students during the Summer before second academic year. 2.The Early Model doesn’t contain any missing values. 3.The Final Model is more accurate compared to the Early Model. The gap between the predicted retention rate and actual retention rate was 2.7% vs. 7.9%, and overall 87.6% vs. 84% of the data was predicted correctly. 4.Eighteen (18) students did not have risk scores by using final model due to missing variables ( i.e. the commuters did not have a home address and thus the distance to school was unavailable). However, they were included as “persisted” based on their overall GPA. 17

18. Early Model—2-Year Retention 18 Table 7 Regression Model: 2-Year Retention (Early Model) Predict VariablesBS.E.WalddfSig.Exp(B) Odds Ratio (recalculated)Rank Remediation -.475.1539.6691.0020.621.615 High School GPA -.403.1785.1561.0230.671.506 Equity Programs.342.1723.9291.0471.4071.417 Overall GPA3.338.172377.1231.00028.16428.161 Overall units.057.00937.3241.0001.0591.06 Repeaters (two years)-.606.13520.0901.000.5451.834 Changed major (4 th term)-1.034.3727.7461.005.3552.823 Undeclared (4 th term)-1.059.3887.4561.006.3472.882 Constant-5.222.74449.2481.000.005 Model Indicators Baseline P*72.0%Chi-Square (df)1248.613 (18) Model N4,568Pseudo R 2.239-.491 -2log L1800.528% Correctly predicted91.0% * Refers to 2-year retention rate.

19. Final Model—2-Year Retention 19 Table 8 Regression Model: 2-Year Retention (Final Model) Predict VariablesBS.E.WalddfSig.Exp(B) Odds Ratio (recalculated) Rank Gender-.838.22114.3541.000.4332.314 Remediation-1.027.24917.0181.000.3582.793 High School GPA-.681.2865.6891.017.5061.985 Equity Programs.570.2834.0651.0441.7681.776 Overall GPA5.615.388208.9581.000274.483274.481 Overall units.104.01548.2401.0001.1101.117 Repeaters (two years)-1.048.22521.5991.000.3512.852 Constant-10.0061.28860.3531.000 Model Indicators Baseline P*72.0%Chi-Square (df)1173.137 (18) Model N4,265Pseudo R 2.240 -.682 -2log L 679.960 % Correctly predicted96.3% * Refers to 2-year retention rate.

20. Preliminary and Predicted 2-Year Retention Rate (2013 Cohort) 20

21. Prediction Results (2013 Cohort ) Preliminary and Predicted 2-Year Retention Rate Early Model Preliminary Total WithdrawPersist PredictedWithdraw 682118800 Persist 22223442566 Total 90424623366 Preliminary 26.9%73.1% Predict 23.8%76.2% Differ 3.1%-3.1% Overall Correctly predicted:89.9% Preliminary and Predicted 2-Year Retention Rate Final Model Preliminary Total WithdrawPersist PredictedWithdraw 69496790 Persist 21023662576 Total 90424623366 Preliminary26.9%73.1% Predicted23.5%76.5% Differ3.4%-3.4% Overall Correctly predicted:90.9% 21

22. 2 Year Retention: The Differences Between The Early and Final Model 1.Early Models can be used for early intervention purposes during mid-spring semester of the second year. The Final Models can be used to contact at- risk students during the summer before the third academic year. 2.The accuracy of Early Models and Final Models are at similar levels. The gap between predicted retention and actual retention rate is 3.1% vs. 3.4%, and overall correctly predicted was 89.9% vs. 90.9%, respectively. 3.Two-year retention models are more accurate than one-year retention models because the actual withdrawals from previous semesters have been included as the part of the prediction.

23. Calculating the Risk Score for Each Student One year calculation:  Early Model: 1-Year Retention Risk Score = -3.044 + 0.260*HSGPA - 0.478*Fulltimefirstterm + 1.534*Term1_GPA + 0.041*Term1_UNO + 0.439*Repeat1  Final Model: 1-Year Retention Risk Score = -4.323 - 0.306*URM - 0.426*Remed_ind - 0.455*HSGPA - 0.002*Distance - 0.453*UNIVLCommunity + 3.504*Term2_GPA + 0.036*Term2_UNO. Two year calculation:  Early Model: 2-Year Retention Risk Score = -5.222 - 0.475*Remed_ind - 0.403*HSGPA + 0.342*Equity all + 3.338*Term3_GPA +0.057*Term3_UNO - 0.606*Repeat2 -1.034*MajorChange3 - 1.059*Major_und4.  Final Model: 2-Year Retention Risk Score = -10.006 - 0.838*Gender1 - 1.027*Remed_ind - 0.681*HSGPA + 0.570*Equity all +5.615*Term4_GPA + 0.104*Term4_UNO - 1.048*Repeat2.

24. Identify Student at Risk by Using the Final Models 24 1. 582 students were subsequently identified as being at-risk and may not return in Fall 2015, including 223 actual withdrawals before Spring 2015. 2. After checking the current registration status (as of 6/29), those who had registered for Fall 2015 were included in the contact list. 2014 Cohort: 1-Year Retention (N= 3,695) 1. 790 students were subsequently identified as being at-risk and may not return in Fall 2015, including141 actual withdrawals before Spring 2015. 2. After checking the current registration status (as of 6/29), those who had registered for Fall 2015 were included in the contact list. 2013 Cohort: 2-Year Retention (N=3,366)

25. Intervention During Summer 2015 25 First Group of Students Enrolled in Spring 2015 with a high risk score May or may not register for Fall 2015 Need to encourage them to register for all 2015 Second Group of Students Withdrew or stopped out during Spring 2015 Have not registered for Fall 2015 Need to recruit them back in Fall 2015 Third Group of Students Withdrew or stopped out at least a year ago Must reapply for this University if they plan to come back Need to provide guidelines outlining the admission procedure

26. Contact Lists for Intervention Term4 Dept. Registe red Term4 ENR Term3 ENR Term2 ENR Ret2_ Score Ret2_Pr e Term4 _GPA Term4_ UNO ART0111-1.3901.729 COMS1111-2.7801.629 COMS1111-1.9201.8318 COMS1111-1.0801.7827 COMS0111-0.9601.9429 DOD0111-0.0302.121 HIST0111-1.2201.5536 PHIL1111-2.3501.5227 THEA1111-0.5601.4830 BUS0111-3.9501.4416 Term2 Dept. Register ed Term2 ENR Ret1 Score Ret1 Pre Term2 GPA Term2 UNO ART01-2.1101.336 ART11-1.3701.4612 COMS01-6.71000 COMS01-6.53000 COMS01-5.47000 COMS01-4.7200.386 COMS01-3.5200.69 COMS01-1.5101.1815 COMS01-1.501.2212 COMS01-1.4801.1414 26 2013 Cohort2014 Cohort

27. The Quality of Prediction Models for Retention High percent of overall correctly predicted: i.Early Model: 84% correctly predicted for 1-year retention when using to predict the retention rates for the 2014 cohort. ii.Final Model: 88% correctly predicted for 1-year retention when using to predict the retention rates for the 2014 cohort. iii.Early Model: 90% correctly predicted for 2-year retention when using to predict the retention rates for the 2013 cohort. iv.Final Model: 91% correctly predicted for 2-year retention when using to predict the retention rates for the 2013 cohort. v.All the results will need to be re-checked by using the Fall 2015 census files (currently not available). 27

28. Discussion: Unsolved Issues The following issues with the Regression Models need to be addressed or resolved: 1.Negative correlation between high school GPA and 1-year retention or 2- year retention with three of the four models. 2.Negative correlation between the Learning Community and 1-year retention rates with the Final Model. 3.Overall unit completion has a low odds ratio compared to other predictors even though it is still a powerful predictor for retention. 4.When using regression models to predict the retention for different cohorts, the accuracy has decreased slightly by year. For example, 1-year retention models had 1% to 2% higher accuracy of prediction for the 2013 cohort than for the 2014 cohort. 5.It is difficult to predict if or when the students will return after they have stopped out one or more semesters due to of lack of information. 28

29. Questions? Contact Information: Jing Wang-Dahlback Director of Research Office of Institutional Research California State University, Sacramento Email: Jwang@csus.eduJwang@csus.edu Sacramento State OIR Website: http://www.csus.edu/oir/