Early identification of online students at risk of failing Presented by Naomi Noguchi School of Public Health, University of Sydney In this study, we identified some predictors of failing in online students so we can intervene early in the semester.

Early identifiers so we can intervene early Background It is a challenge to identify students who are at risk of failing with no face-to-face contact Early identifiers so we can intervene early ?Access to the LMS in the first week ?Consistent access to the LMS ?Late enrolment Aim To identify predictors of failing the final assignment A difficulty we have faced running an online subject is that of identifying students at risk of failing. The lack of face to face contact makes it difficult to find out which students are struggling. It would be ideal if we can identify at-risk students early in the semester and offer increased support in a timely manner. Previous studies have identified a number of predictors of student performance such as -access to the learning management site in the first week of the semester and -consistent access to the LMS. We have also observed that late enrollers frequently have poorer performance. Our aim was to identify the early predictors of failing the final assignment in online students.

Methods: Study sample 131 online postgraduate students undertaking Introduction to Clinical Epidemiology at Usyd in S2 2017 Most students are medical doctors who study part-time Our study sample was online postgraduate students undertaking Introduction to Clinical Epidemiology at the University of Sydney. Most students were medical doctors who study part-time. *114/131 (87%) were M/Grad Dip/Grad Cert of Med/Surg students who are medical doctors.

Methods: Assessment tasks Weekly multiple-choice quizzes (15%) Submitting answers for weekly tutorial questions (10%) Mid-term assignment (15%) Final assignment (60%) Assessment tasks for this unit of study consisted of -15% for weekly multiple-choice quizzes, -10% participation marks for submitting answers for weekly tutorials, -15% for the mid-term assignment and -60% for the final assignment

Methods: Predictive factors Gender Past degrees Repeating the unit Enrolling late Baseline knowledge (Pre-test quiz) Weekly quiz marks Weekly tutorial participation Mid-term assignment mark Potential predictive factors that we examined included -gender, -past degrees, -repeating the unit, -enrolling late, -baseline knowledge based on the pre-test quiz that students complete before starting the unit -quiz marks -tutorial participation and -the mid-term assignment mark. The fail rate in the final assignment was compared between different categories of each variable. Numbers in each section were too small for statistical testing. We focused on characteristics where more than 15% failed which is twice the total fail rate.

Result – Overall Assignment 2 grades Overall, Fail and Absent Fail were 7% in total. *122 (93%) students passed 5 (4%) failed 4 (3%) did not submit (absent fail).

Result – Repeating the UoS PA FA AF This is failing the subject in the past. 1 of the 3 repeating students failed. * AF fail pass total AF+FA Repeating 1 0 2 3 33% 1 AF had an AF before. Of the 2 PS, 1 had an AF and the other had a FA before. Not repeating 3 5 120 128 6% FA+AF=33% FA+AF=6%

Result – Enrolling late PA FA AF FA+AF=5% FA+AF=20% FA+AF=33% FA+AF=0% This is enrolling late. 1 in 5 students who enrolled in week 1 failed and 1 in 3 students who enrolled in week 2 failed. Late enrollers might be disadvantaged by the delay in the beginning of the semester. Or enrolling late may simply be a marker of disorganisation. * AF fail pass total AF+FA Before Week 1 2 4 113 119 5% On Week 1 1 0 4 5 20% On Week 2 1 1 4 6 33% On Week 3 0 0 1 1 0% (Students enrol automatically until the end of week 2. Later enrolments need approval.)

Result – First 5 weekly tutorials PA FA AF FA+AF=40% FA+AF=27% FA+AF=2% This is participation in the first 5 tutorials. In students who missed 3 or more of the first 5 tutorials, 40% failed. In those who missed 2 of the first 5 tutorials, 27% failed. Early engagement seems to be important and Absent Fails did not have any engagement right from the beginning whereas some Fail students started off okay and later run into trouble once the tutorials get onto harder topics involving critical appraisal which is a key skill for being able to pass the assignment. * AF fail pass total AF+FA 0-2 weeks 3 1 6 10 40% 3 weeks 1 2 8 11 27% 4-5 weeks 0 2 108 110 2%

Result – First 5 weekly quiz PA FA AF This is the first 5 weekly multiple choice quiz marks. Only the lowest quartile had higher fail rate – this includes students who missed a few of the first quizzes as well as those receiving low marks. Again, early engagement seems to be important. * AF fail pass total AF+FA <5.5 1st quartile 3 3 26 32 19% -7.5 2nd-4th quartile 1 2 96 99 3% FA+AF=19% FA+AF=3%

Result – Mid-term assignment PA FA AF FA+AF=3% FA+AF=15% FA+AF=30% This is the mid-term assignment. Not submitting or failing the mid-term assignment predicted failing the final assignment. * AF fail pass total FA+AF Passed 1 2 98 101 3% Failed 1 2 17 20 15% Did not submit 2 1 7 10 30%

Variables that showed no major difference Gender Previous degrees Baseline knowledge There was no major difference for -gender -previous degrees and -baseline knowledge.

Summary Conclusion Failing the UoS previously Enrolling late Lack of engagement early in the semester predicted receiving either a fail or absent fail grade in the final assignment Conclusion In summary, -Failing the subject previously -enrolling late -lack of early engagement early in the semester predicted receiving either a fail or absent fail grade in the final assignment. We can provide increased support to these at-risk students such as -giving them a warning and -providing tips to pass the unit of study. These findings may apply to other online units that require regular participation early in the semester. Increased support can be provided to these at-risk students early

Acknowledgement Dr Fiona Stanaway: Unit Coordinator

Result – Gender FA+AF=4% FA+AF=10% PA FA AF AF fail pass total AF+F Male 1 2 66 69 4% Female 3 3 56 62 10% FA+AF=4% FA+AF=10%

Result – Previous degrees PA FA AF Most are medical doctors, but some may be PhD doctors. AF fail pass total AF+FA Dr 3 4 103 110 6% Not Dr 1 1 19 21 10% FA+AF=6% FA+AF=10%

Result – Baseline knowledge (Pre-Test) PA FA AF FA+AF=4% FA+AF=6% FA+AF=5% FA+AF=8% This is the baseline knowledge test marks in quartiles. AF fail pass total AF+FA <8 0 1 26 27 4% <10 0 2 32 34 6% <11 1 0 18 19 5% -15 2 2 46 50 8% quartiles

Result – First 5 weekly quiz PA FA AF FA+AF=19% FA+AF=0% FA+AF=6% FA+AF=3% This is the first 5 weekly multiple choice quiz marks in quartiles. Only the lowest quartile had higher fail rate – this includes students who missed a few of the first quizzes rather than receiving low marks. Again, early engagement seems to be important. AF fail pass total AF+FA <5.5 1st quartile 3 3 26 32 19% <6.25 2nd quartile 0 0 27 27 0% <7.0 3rd quartile 1 1 34 36 6% -7.5 4th quartile 0 1 35 36 3%

Result – All tutorials PA FA AF FA+AF=33% FA+AF=17% FA+AF=1% These are not early predictors, but I have included these in the abstract. I will bring the slides to the conference just in case someone ask about these factors. AF fail pass total AF+FA 0-4 weeks 3 2 10 15 33% 5-7 weeks 1 2 15 18 17% 8-10 weeks 0 1 97 98 1%

Result – Total quiz marks PA FA AF FA+AF=18% FA+AF=6% FA+AF=3% These are not early predictors, but I have included these in the abstract. I will bring the slides to the conference just in case someone ask about these factors. AF fail pass total AF+FA <10.25 1st quartile 3 2 23 28 18% <12.25 2nd quartile 1 1 29 31 6% <13.5 3rd quartile 0 1 37 38 3% -15 4th quartile 0 1 33 34 3%

Result – Number of pages viewed on LMS FA AF FA+AF=18% FA+AF=6% FA+AF=3% These are not early predictors, but I have included these in the abstract. I will bring the slides to the conference just in case someone ask about these factors. AF fail pass total AF+FA <418 1st quartile 4 1 27 32 16% <559 2nd quartile 0 0 33 33 0% <726 3rd quartile 0 4 28 32 13% -4000 4th quartile 0 0 33 33 0% The number ranged from 0 to about 4000.