Webinar Series This session is part of e-Assessment Scotland Aug 2013: Learning analytics: a bottom-up approach to enhancing and evaluating students' online learning Josie Fisher, Fredy-Roberto Valenzuela and Sue Whale (University of New England, Australia) Your hosts Professor Geoff Crisp, Dean Learning and Teaching, RMIT University Dr Mathew Hillier, Teaching and Educational Development Institute, University of Queensland

Learning Analytics: A Bottom-up Approach to Enhancing and Evaluating Students’ Online Learning Josie Fisher Fredy-Roberto Valenzuela Sue Whale

Structure of the Presentation 1.Literature Review 2.Overview of the Research Projects 3.Stage One of the Project 4.Stage Two of the Project 5.Final Conclusions and Remarks

1.a. Understanding What Analytics means Analytics refers to the application of business intelligence principles and tools to academia Academic analytics takes an institution level focus Learning analytics focuses on the learning process including the relationship between learner, content, institution and educator

1.b. Literature Increased enrolment numbers and study options, diversity of student population has also increased Students who successfully complete a subject are more likely to reenrol Correlation between learners engagement in effective learning practices and their academic success (Campbell et al 2010)

Literature (Cont’) Perception of psychological presence of teaching staff, peers and the institution has a significant impact on student learning, satisfaction and motivation. Effectiveness of coaching – supported by Bettinger and Baker (2011) and Nelson et al (2009)

2. Overview of the Research Projects Undertaken Using Learning Analytics One subject Case Study Interventions Various Management Subjects OLT Grant

3. Stage One of the Project Case study - used learning analytics to inform strategies aimed at enhancing student outcomes at the subject level Although we took a subject level focus institutional benefits were anticipated if these strategies were successful

Subject information Compulsory professional ethics subject in Graduate Certificate and Masters Full fee Offered online through LMS Typical enrolment of around 140 Two years’ relevant work experience required

Student demographics 70% between Gender balance even Most working full-time All studying part-time 73% had no previous university studies High % ESL

Most of these student characteristics can contribute to the likelihood of attrition and reduced success rates (Gabb, Milne & Cao, 2006) and this provided the motivation for developing an intervention program

Research questions 1.What inferences can be drawn from an analysis of students’ outcomes in the first two (online) assessment tasks that could inform subsequent offerings of the subject? 2.To what extent is participation in the LMS a predictor of overall achievement in the subject?

Methods Historical data from 2011 were analysed in two ways: 1.Participation and achievement in the first two assessment tasks and final results were compared 2.Content access and usage patterns of students were analysed for the five highest and lowest achievers

Analysis of student outcomes Students who completed the Activity were more likely to successfully complete the subject (91% compared to 72%) Students who completed the Activity achieved an average of 79% in the Test compared to 73% for those who did not Students who completed the Activity achieved an average final mark of 66% compared to 58% for those who did not

Cont. Of the 9 students who failed the Test, only 3 successfully completed the subject with marks of 50, 52, 52 The 30 students who achieved Test marks less than 65% achieved an average final mark of 54.5%

Inferences Students who completed the optional Activity were more likely to successfully complete the subject than those who didn’t Students who completed the Activity achieved better marks in the Test than those who did not complete it Test results were an indicator of future success in the subject

Proposed interventions with students who do not complete the Activity Provide private discussion space for these students to discuss content related to the activity Personal contact from lecturer to identify issues which may be alleviated

Proposed interventions with students who fail the Test Host synchronous online forum/s to reiterate subject matter covered in the Test Create private discussion space for students where questions can be asked and further explanations provided by lecturer Provide additional resources to enhance understanding of key concepts

Extent to which participation in the LMS predicts overall success Sample of 5 students whose final marks were 80%+ These students accessed learning materials regularly and in sequence Four accessed the weekly podcasts They all completed the optional Activity They accessed the topic forums regularly

Cont. Sample of 5 students whose final marks were 40-49% These students demonstrated random access to the LMS Three students did not access the weekly podcasts at all while the other two only accessed two of the 12 podcasts Some did not access the LMS until after week three

Inferences Students were more likely to succeed if they: Accessed all materials in sequence and regularly Accessed additional resources (video clips and podcasts) Accessed and participated in forums Completed the optional Activity

Proposed interventions Monitor consistency and regularity of access and highlight potential risks of not maintaining study pattern Contact students who have not accessed materials by end of week two Emphasise the importance of podcasts

4. Stage Two: The OLT project Explored use of simple real time learning analytics by individual educators as part of their learning and teaching activities in order to inform interventions which were implemented within the teaching period Offered opportunity to explore potential impact of these interventions on student engagement and satisfaction across a number of subjects offered by distance mode (online) in UNEBS.

Aim Critical Evaluation of the use of LA and associated interventions to increase student engagement, satisfaction across subjects offered by distance mode (online).

Design Implemented by individual educators Using student data readily available in the LMS

Approach Tracking activities Implementation of Interventions Evaluation of Results

Analysis of behaviours Utilised results of case study to establish patterns of behaviour and resulting success and satisfaction for students. Behaviours identified as limiting success of students: –Limited access to LMS in early weeks of teaching period –Poor results in early assessment –Limited access to materials relating to major assessment tasks –Inconsistent access across teaching period

These ‘triggers’ were used to inform interventions which were implemented and evaluated during the teaching period – providing opportunities to explore potential impact on students learning experience.

Process and results Timing of each intervention was based on individual behaviours of students Tailored to utilise range of contact methods – phone calls, , broad messaging (preferred method – personalised)

Intervention 1 Limited access to LMS and learning materials in first 2-3 weeks of teaching period 43 students contacted personally through phone calls Questions asked: –Have you looked at unit materials? –Have you experienced problems with access or working through materials? –Do you have the prescribed text? –If an indication was given that the student was considering withdrawing – why? Could we assist with getting started and continuing?

Intervention 1 (continued) Those unable to be contacted by phone were approached via personal responses followed up within 24 hours Student feedback ‘thank you very much for your …thank you for your support’ ‘thank you very much for getting my focus back’ ‘sincere thanks for your – I appreciate your interest’ (Student comments, 2013)

Intervention 2 Poor results or non-participation in early assessment items was an obvious indicator of students who may be struggling with subject content. Students were contacted via ‘Dear XXX, The Online Test marked the ‘official’ end of Module One. The content, however, will be applied throughout Module Two, so an understanding of the main theories and concepts introduced in Topics 1.1 – 1.5 will be required. If you did not do as well as you had hoped in the Online Test, I suggest you carefully review the questions you answered wrongly. If you have difficulty in understanding where you went wrong, or you have any other questions, please contact me.’

Intervention 2 (continued) This intervention focused on early assessment tasks and those which were not weighted heavily. Since performance in these early assessment tasks provides a strong indicator of success in the remainder of the unit, ensuring students engaged with the materials and understood the key concepts was vital.

Intervention 3 The third intervention was aimed at student who had not accessed the LMS for more than 7 days prior to due date of the major assessment task. Students were contacted personally by of these (36) students subsequently submitted the assignment.

Online Survey (35% response rate) Subject Initial prompts were perceived as positive by students Prompts encouraged engagement with materials Prompts assisted in preparation for assessment Prompts enhanced learning experience/ increased satisfaction MM GSB GSB Table 2: Targeted Students Survey Results (out of 5)

Online Survey (continued) Students responses Prompts encouraged them to engage with materials Considered prompts assisted in assignment preparation Prompts enhanced learning experience

Online Survey (continued) Test group: 57 students not targeted by interventions compared to 16 students targeted Results Questions Students who were Targeted by the Interventions Students who were NOT Targeted by the Interventions Overall Learning Experience Table 3: Results of End of Trimester Survey (out of 5)

5. Final Conclusions and Remarks Tracking students activities and the timely implementation of interventions has the potential to influence students’ behaviours and improve chances of success and hence to enhance students online learning experience Behaviours identified which may impact student results in online learning Consideration given to timing and format of interventions – emphasis on personal contact

Final Conclusions and Remarks (cont’) Student responses revealed that interventions were highly appreciated as they improved learning experience Challenges – workload Next steps – Interviews Future Research – Ethics vs Analytics