1. Flexible and Online Learning and Student Progress in First Year: CDU Interim Report Bill Tyler Charles Webb Nicola Rolls Sharon Bridgeman Malcolm Flack

2. Overall Study Aims
To explore the pedagogic impact of flexible and online learning approaches on students’ retention and success through analysing:
The effect of engagement in Learnline and collaborate on grade
Student perceptions of different flexible delivery approaches as assessed from customised survey instruments
The effect of different online pedagogies on student engagement

3. Summary of Previous Findings I
Decadal trends in first year intake composition and student learning situation have profound effects on patterns of retention and success.
Overall CU & first year trends for success are very positive , external withdrawals are predictably higher given wider cast of the net.
The effects of student profile and situation on rates of retention and success need further investigation.
Students are drawn to tools that keep them connected to each other, are offered in lots of forms (including hard copy) and are relevant to assessment and accessible.
Learnline is helping students learning and students find tools that inform, connect and motivate them very useful for learning – collaborate & discussion and communication from teaching staff.
Transitional period for Core Units offerings:
12 out of 19 units sampled introduced in 2013
4 units phased out/ not offered during the period
3 only with all five year availability
Trending “gaps” in Pass rates all units, CUs and Core:
Continuing gaps in External/Internal modes (CU’s 5%, Core 8%)
Narrowing gaps in “TER below 50”, Male, Full-time
Rebound in core unit External mode in 2013 (new units)
Comparing Trends: Core and Common Units (Passes & Withd’ls):
Common Units Passes trending up-Core Units Passes trending down
Significant remaining differences :(16% for Core, 10% for CUs) in Withdrawals for External/Internal in both unit types

4. Perceived Effectiveness
Checking the effect of students’ learning Motivation on Perceived Effectiveness of LL and Engagement – online survey study
Motivation X
Perceived Effectiveness
Usage
Would motivated students engage regardless of learnings tools provided?
Correlations between students learning motivation and frequency of access and ratings of Learnline effectiveness checked
No significant correlations
Therefore – learner motivation levels are not a significant factor for frequency of access and engagement in LL !!!

5. Summary of Previous Findings II
Initial analysis showed some association of student success with an increasing extent of overall Learnline use and use of the collaborate tool.

6. Introducing analytics
Learnline engagement data now made available through analytics
Data more accessible and more specifically describes individual student’s engagement with Learnline content by total:
Number of times a unit is accessed
Total Minutes spent in unit on LL
Total Clicks within the LL site

7. Preliminary findings of engagement & success using analytics metrics

8. Preliminary findings of engagement & success using analytics metrics ctd

9. Preliminary findings of engagement & success using analytics ctd

10. Collaborate use and Success

11. Collaborate use and Success ctd

12. Multivariate Modelling of Learnline Engagement and Grades
Employed a multivariate model to emulate a randomised control trial type study.
The control group assumes “low engagement”. Comparisons were made between the low group and different levels of higher engagement (as assessed by number of visits to LL).
Data were collected from 6 units: S2, (BCO106, CML100, CUC100, CUC106, CUC107, LWZ108, SBI105).
Modelling takes into account background factors that may bias results (e.g. age, BOA, TER etc).

13. Testing the Model on Learnline Engagement % Grade improvement from a PC or Pass to Credit or Above
Online activity has the potential to increase PC/Pass to Credit or higher. Interestingly, by increasing engagement a relatively small amount has the largest effect on grade. %
The second quartile (50 and 63 accesses) yields the highest gain.

14. Testing the Model on S.2 2013 collaborate accesses
Results indicated positive effect of participation in Collaborate on Grade Awarded
Results indicated statistically significant positive effect of participation in the Collaborate program on Grade Awarded for a continuing enrolment sample of all Common Units in Sem 1, 2013.
The effect of Collaborate participation was estimated as an average improvement in Grade Awarded from a low Credit to mid-Distinction level for both minimal (21%) and higher (37%) levels of attendance at available Collaborated sessions.
In terms of participation: External Mode of Attendance was uniformly the most important positive influence on self-selection to participate.
Grade point average

15. Summary of Previous Findings III
External students using collaborate achieve higher grades than other students and consideration might be given to promoting collaborate to ensure more students engage with this tool.

16. Collaborate engagement by Sub-Cohort

17. Summary of Previous Findings IV
Overall qualitative student commentary on Learnline experience is positive but some possible priority areas for improvement have been identified including aspects of platform useability, assignment relevance and staff accessibility.

18. Updated BA/NI analysis

19. Possible improvement areas
Useability
Not perceived as user friendly, needs a more attractive interface that is easier to navigate, issues with different sites looking and being set up differently, and problems with links not working
Discussion boards and interactive tools
Often difficult to use and to easily keep track of discussion threads. Wikis not viewed very favourably. Need for more consistent use across units of interactive tools facilitating student:student and student: teacher interaction
Recorded Lectures and online tools
Concern that units did not consistently provide recorded classes or if provided often not loaded in a timely fashion. Concern with the quality of recorded sessions students often finding it hard to hear information. Students would like more consistent access to the collaborate tool across all their units with fewer technical glitches that impact on access.
Assessment
Concerns with the process of assignment submission including clarity on due dates and better acknowledgment of safe receipt of submitted assessment items. Students would like more guidance with requirements of assessment tasks

20. BA/NI by Sub-cohort

21. BA/NI by unit type

22. Audit for best practice online pedagogy
Criteria for audit of units based on:
Blackboard exemplar
Students views from survey about what they think is important to include in Learnline
Standards based on Blackboard categories:
Exemplary (E)
Accomplished (A)
Promising (P)
Incomplete (I)
Wanted to explore what is happening across units that have been set up differently
One was of doing this was to use the rubric from the blackboard exemplary course program, this was customised taking into account what students found important
Used tool to assign each unit into a category.

23. Audit – Summary of findings
Summary of category %/semester
Semester
Category
Exemplary
Accomplished
Promising
Incomplete
S2.2013 9
63.6
27.4
S1.2014 25 75
S2.2014
8.3
16.7
S1.2015
62.5
12.5
Used the same unit sample for the quantitative analysis and the surveys, namely the common units and other large 1st year units.
Need to be careful when interpreting the results as not all units are offered in both semesters.
Looking at this sample - overall the ratings are quite high, no units fall into the incomplete category Since 2014, over 80% are scoring in category A or higher. 

24. The Final Phase
Complete analysis of data on Learnline/collaborate collected in s1 2015
Further investigation and analysis from online surveys of student attitudes towards, and perceptions of, Learnline
Application of multivariate causal analysis to the larger data sets to test the impact of online engagement on student success
Exploring relationships between unit audit types and patterns of student engagement and satisfaction with Learnline
Completing the longitudinal analysis of student retention and success in common units and other first year offerings
Developing recommendations for policy and practice to further enhance student online engagement across different student cohorts