1. Early Alert System (EAS) in Online Education: Student Demographic Profile
Helen Zaikina-Montgomery, Ph.D.
Scott Burrus, Ph.D.
Meryl Epstein, Ed.D.
Elizabeth Young, Ed.D.
Roger Gung, Ph.D.

2. Overview and Purpose of Study
Examine the demographic characteristics of students who receive Early Alerts (EAs) in courses
Compare demographics of students who receive EAs to students who do not receive EAs
Existing research on course intervention strategies, such as EA identifies a need to better understand demographics of students who struggle in course work
Build statistical models to evaluate the impact of demographic characteristics to receive Early Alerts
Further evaluate the impact of Early Alerts on students’ probability of passing course
[Ortagus, 2017]

3. EAS in the Present Study Context
Implemented in 2007 with the goal of increasing course completion by alerting the student’s Academic Counselor to contact the student & discuss concerns.
Updated January 2016 to automatically alert the student’s Academic Counselor (AC) if the student did not submit an assignment or participate in the online discussions in the course.
Current EAS Process
Faculty files EA through classroom or EA issued by LMS
Academic Counselor is notified of issue
Academic Counselor reaches out to student

4. Supporting Research Overview
Student retention and graduation rates are a topic of institutional concern and academic examination
Due to a difference in modality of delivery, online courses are structured differently than traditional on-ground or blended courses
Online courses require students to possess more intrinsic motivation and higher levels of organizational and self-management skills
Through a mutually collaborative ecosystem,
[Allen, Seaman, Poulin, & Taylor, 2016; Braxton, 2002; Eaton, 2011; McElroy & Lubich, 2013]

5. Supporting Research Overview
National Postsecondary Student Aid Study (NPSAS) shows that being married, being a parent, and a full-time employee were positively associated with online course enrollment.
NPSAS data showed that minority students were less likely to engage in online education than their non-minority peers
Effective interventions for students in online courses need to be well-matched to the online learning environment and to the demographic characteristics of those who are most likely to struggle in their course work
[Allen, et al., 2016; Donnelly, 2010; McElroy & Lubich, 2013; NPSAS, 2012; Ortagus, 2017]

6. Questions guiding this Research
What is the demographic profile of students who received an Early Alert, and how do they differ demographically from those who did not receive EA?
What is the demographic profile of students who pass the courses, and can we measure the impact of EA on passing course?

7. Data Source
Collected from the university Office of Business Analytics and Operations Research.
Included records from students who were issues an early alert either by the course instructor or the learning management system (LMS) sometime during their course
Included courses with start dates between November 24, 2015 and January 26, A total of 26,573 student records were accessed and used in the study of which 2.4% (n = 640) were students who received an Early Alert

8. Early alert Profiling results

9. What is the demographic profile of students who received an Early Alert?
Demographic Characteristic
EAS Students (N = 640)
Non-EAS Students (N = 25,933)
Age in years M (SD)
33.9 (8.59)
35.1 (9.15)
Number of people in family M (SD)
2.69 (1.41)
2.83 (1.50)
GPA M (SD)
2.57 (0.59)
3.10 (0.62)
Total transfer credits into the university M (SD)
14.17 (19.38)
15.8 (20.44)
Gender N (%)
Male
173 (27.4)
8,045 (31.3)
Female
458 (72.6)
17,677 (68.7)
Marital status N (%)
Single
379 (63.5)
12,935 (54.7)
Married
145 (24.3)
7,686 (32.5)
Separated
36 (6.0)
1,193 (5.0)
Divorced
37 (6.2)
1,819 (7.7)

10. Demographic Characteristic
EAS Students (N = 640)
Non-EAS Students (N = 25,933)
Have dependents N (%)
Yes
86 (14.5)
3,475 (14.8) No
507 (85.5)
19,944 (85.2)
Military status N (%)
Military (past or current)
140 (21.9)
5,446 (21.0)
Not Military
500 (78.1)
20,487 (79.0)
Passed course N (%)
268 (41.9)
21,351 (82.3)
372 (58.1)
4,582 (17.7)
Course grade N (%) A
10 (1.6)
5,890 (22.7) A-
4,150 (16.0) B+
14 (2.2)
2,052 (7.9) B
18 (2.8)
2,011 (7.8) B-
28 (4.4)
2,196 (8.5) C+
26 (4.1)
1,297 (5.0) C
31 (4.8)
1,331 (5.1) C-
67 (10.5)
1,243 (4.8) D+
22 (3.4)
614 (2.4) D
42 (6.6)
554 (2.1) D-
30 (4.7)
546 (2.1) F
108 (16.9)
1,240 (4.8) W
234 (36.6)
2,800 (10.8)

11. Demographic Characteristic
EAS Students (N = 640)
Non-EAS Students (N = 25,933)
Form of payment N (%)
Student Direct Pay
27 (11.2)
2,313 (8.9)
Financial Aid
558 (87.2)
22,638 (87.3)
Military
10 (1.6)
797 (3.1)
Scholarship (non-EAS only)
0 (0.0)
185 (< 1.0)
Program level N (%)
Associate
172 (27.1)
6,546 (25.5)
Undergraduate
383 (60.4)
16,262 (63.3)
Graduate
79 (12.5)
2,875 (11.2)

12. Female students more likely to receive an EA than male students
How students who received an Early Alert differ demographically from students who did not receive an Early Alert?
Female students more likely to receive an EA than male students
Younger students and students who were single were more likely to have an early alert
Students who received an EA were less likely to pass the course in which early alert was filed, three times more likely to withdraw from the course, and four times more likely to fail the course

13. Cramer’s V ɸc (p-value)
How students who received an Early Alert differ demographically from students who did not receive an Early Alert?
Chi square tests applied to EAS status related to categorical variables
Variable
Chi-square df
p-value
Phi ɸ (p-value)
Cramer’s V ɸc (p-value)
Gender
4.276* 1
.039
-.031 (.039)
Marital status
23.047** 3
.000
.031 (.000)
Dependents
.052
.820
-.001 (.820)
Military status
.288
.592
.003 (.592)
Passed course **
.159 (.000)
Course grade ** 13
.187 (.000)
Form of payment
4.362 2
.076
.088 (.076)
Program level
2.365
.307
.009 (.307)
* = Values significant at the .05 level
** = Value significant at the level

14. Early alert Modeling

15. Modeling Approach

16. Early Alert Variables Exploration

17. Early Alert Variables Exploration

18. Early Alert Variables Coefficient
Factors
Estimate
Std Error
P-Value
Intercept
-1.521
0.250
0.000
GPA: 2 < · <= 2.67
-0.406
0.121
0.001
GPA: 2.67 < · <= 3.14
-0.950
0.135
GPA: > 3.14
-2.147
0.149
AGI: <= 10647
0.254
0.200
0.204
AGI: < · <= 40205
0.104
0.192
0.587
AGI: > 40205
-0.061
0.205
0.767
Gender: M
-0.119
0.100
0.232
Gender: O
0.623
0.353
0.078
Payment Option: DEF
-1.814
0.719
0.012
Payment Option: MIL
-0.771
0.374
0.039
Payment Option: OTPO
-0.501
0.134
Program Level: Graduate
1.166
0.153
Program Level: Undergraduate
0.044
0.096
0.648
AC Tenure: 5
-0.048
0.279
0.864
AC Tenure: 5 < · <= 10
-0.320
0.101
0.002
AC Tenure: > 10
-0.031
0.128
0.809
CCC Flag: Non-CCC
-0.650

19. Odds Ratio
Note: Odds Ratio is the odds of the positive outcome at one level of Xi relative to the odds of the positive outcome at reference level of Xi

20. Early Alert Model Performance
The early alert model (7 variables) has a C statistic of 0.736, which can be considered a good model performance

21. COURSE-PASS Modeling

22. Course-Pass Variables Exploration

23. Course-Pass Variables Exploration

24. Course-Pass Variables Coefficient
Factors
Estimate
Std Error
P-Value
Intercept
-3.782
0.184
0.000
EAS Flag: 0
1.485
0.094
Age: 33 < · <= 48.4
0.086
0.042
Age: > 48.4
0.133
0.074
Age: Missing
0.414
0.393
0.292
GPA: 2 < · <= 2.67
1.010
0.066
GPA: 2.67 < · <= 3.14
1.688
0.070
GPA: > 3.14
2.643
0.071
AGI <= 10647
0.006
0.120
0.958
AGI: < · <= 40205
0.135
0.123
0.272
AGI: > 40205
0.285
0.128
0.026
Payment Option: DEF
0.206
0.188
0.274
Payment Option: MIL
0.499
0.161
0.002
Payment Option: OTPO
-0.139
0.068
AC Tenure: 5
0.096
0.141
0.497
AC Tenure: 5 < · <= 10
-0.083
0.049
AC Tenure: > 10
-0.140
0.063
0.027
CCC Flag: Non-CCC
2.115
0.081
Num Family: 3
-0.050
0.051
0.332
Num Family: 3 < · <= 6
-0.007
0.050
0.885
Num Family: 6
-0.057
0.166
0.731
Num Family: Missing
0.339
0.079
Course Level: 2
-0.117
0.017
Course Level: 3
0.453
Course Level: 4
0.592
Course Level: 5
0.306
0.001

25. Odds Ratio
Note: Odds Ratio is the odds of the positive outcome at one level of Xi relative to the odds of the positive outcome at reference level of Xi

26. Course-Pass Model Performance
The early alert model (9 variables) has a C statistic of 0.798, which can be considered a good model performance

27. Conclusions & Future research
Students who receive EA tend to be more academically “at risk” than non-EA students, and have 41% lower passing rate.
Some of the expected factors (those that add more responsibility, such as being married and having dependents) were not associated with EA status
Additional data and further analysis, including hierarchical linear modeling to account for potential nested effects should be conducted
Universities can continue to develop interventions for “at risk” students keeping in mind factors from this study
Students’ GPA, collection status, form of payment, income have strong impact on students’ receiving EA
For a non-EA student, the odds of passing a course is 4.4 times as the odds of an EA student, due to the inherent risk

28. Questions & Discussion

29. Campus Collection Center (CCC) Flag
Variables List
Variable
Attribute
Description
Early Alert
Categorical
0: No Early Alert
1: Has Early Alert
Passed Course
Passing Grade is C- for graduate students,
D- for bachelor and associate students
GPA
<=2; ; ; >=3.14
Campus Collection Center (CCC) Flag
If the student’s risk code is CCC, then “CCC”; Otherwise “Non-CCC”
Age
<=33; ; >=48
AGI
Annual Gross Income
<= $10,647;
$10,647 - $40,205;
Missing (Missing may indicate cash students)
Program Level
Associate, Undergraduate, Graduate
Form of Payment
Payment Option
DBEB: Direct Bill or Employee Bill;
MIL: Military Payment;
DEF: Deferred or Delayed payment
OTPO: Other Payment Options

30. Variables List (Cont’d)
Attribute
Description
Total Transfer Credit
Categorical
<=20.67 credits;
credits;
>=55 credits
Military Flag
Y, N
Gender
F, M, O
High School Diploma
Y,N
Academic Counselor Tenure
Academic Counselor’s tenure with the company.
<=4, 4-5, 5-10, >=10
Course Level
1, 2, 3, 4, 5
Number of People in the Family
<=2, 3, 3-6, >6
Marital Status
Divorced; Married; Separated; Single

31. Variable Rank Order Based on Information Value (IV)
Information Value
Predictive Power
 >0.5
Suspicious or too good to be true
0.3 to 0.5
Strong predictor
0.1 to 0.3
Medium predictor
0.02 to 0.1
Weak predictor
< 0.02
useless for prediction