1. Flexible and Online Learning Modes and Student Progress in First Year – FINAL PRESENTATION for the Common Unit Committee and Office of Learning and Teaching Research Fellow Menzies Bill Tyler Professor Charles Webb Dr. Nicola Rolls Sharon Bridgeman Dr. Malcolm Flack

2. Project Aim & Background
To investigate the impact of flexible modes of learning and attendance in the first year of university :
By exploring student responses to fleximode pedagogies and the effect of these on their academic progress
Using data from student surveys, Analytics records of usage, collaborate use records and student grades
Data gathered and analysed over three years for common units and 16 course-specific, core First Year units with large enrolments: Law, Business, Nursing, Science, Education
Data set: Learnline 14881, Collaborate 4983, Surveys 17% of population

3. Previous Findings for 2013-2014 data
Association of student success with use of both Learnline and collaborate
Multivariate analysis and counterfactual techniques show these associations are statistically sound and significant
Students drawn to tools that are accessible, keep them connected with other students and staff and are relevant to assessment including collaborate
Learner motivation levels per se not a significant factor influencing student engagement with Learnline
Overall qualitative student commentary on the Learnline experience positive but some possible priority areas for improvement
Indications of student sub-cohort differences between levels of engagement and satisfaction
Outcomes of an audit of units (based on BlackBoard categories) for best practice on-line pedagogy showed 80% in accomplished or exemplary category

4. This Presentation: outcomes based on full data set
Student success and engagement with online learning tools
Assessing the strength of association and testing possible causality
Further work on the impact of motivation, intentions to use these tools and perceived effectiveness and ease of use
Studies of qualitative commentary on student experience of learnline
Relationship between subcohort engagement with on-learning and student satisfaction
Recommendation for policy and practice directions for fleximode teaching
Next steps for understanding the whole student engagement universe

5. Learnline use and success at a glance
This slide shows the relationship between extent of learnline engagement and academic outcome (grade scale) based on data from nearly 15,000 students studying in 43 unit instances included in the study and offered across the four main semesters from semester to semester Learnline engagement has been assessed using a use algorithm derived from learnline analytics data for students in different units based on numbers of unit accesses, keyboard clicks and total minutes spent visting the unit learnline site, normalised by comparing data for individual students with unit average data. There is a clear upward trend with students achieving higher grades progressively being more engaged with learnline as measured by the use algorithm
While there is an obvious association between the extent of use of learnline and the academic success of students the relationship needs to be tested for statistical significance taking account of potential confounding affects resulting from different mixes of students in grade scale categories for characteristics such as gender, age, home language, mode of study. This has been addressed through the use of multivariate statistical and counterfactual modelling.

6. Multivariate test: association between LL use and grade holds up (n=8,970 enrolments, Sem Sem1, 2015)
This figure plots the association between Learnline activity (briefly define how this was measured and what 1, 2, 3 equate too if possible) and average grade expressed as GPA (how grade is recorded in Callista).
Importantly, the scores here are adjusted to take into account the influences of background variables we have access too such as age, gender, mode of study … This provides a more robust comparison than simply comparing average score across levels of usage
So on average, those students who engaged at “level 1” achieved a GPA of 4.5 (in the CUC units I think). In contrast, those who engaged to the most (10) achieved an average grade just under 5.5, which is almost a grade category higher. The band either side of the main line is the confidence interval, which is an estimate of where average scores would fall if the same study was ran again under exactly the same conditions - 95% sure the values would fall between these lines.
Adjusted for variables: study mode, age, gender, language background, part-time or full time, BOA

7. Applying rigorous testing: the counterfactual Model
There are many complicating and confounding variables including: study mode, age, gender, language background, part-time or full time, BOA plus a range of other factors we cant measure or perhaps don’t know about. A randomised clinical trial where Collaborate or learnline exposure was the “treatment” would account for these complicating and confounding factors in the population tested Since we cannot do this, we can still satisfy the assumptions of a randomised trial. This is called the potential outcomes or counterfactual framework. This is done by applying two statistical tricks that can cancel out the hidden biases (the things you haven’t or cant measure) that can lead to misleading or inconsistent findings. In this model we try to cancel out the biases on both the grade outcome and on the students self-selection to participate in learnline. We then subtract the potential mean for the group exposed to learnline from the potential mean for the unexposed group to obtain an average effect of exposure to learnline.

8. The relationship between Learnline activity and grade holds under the Counterfactual model
The blue line on this graph now depicts the same information as shown on the earlier slide with a line fitted to represent the general relationship (regression line).Although the blue plot takes account of the background factors we have been able to measure, there are potentially other factors that influence students’ engagement that we have not measured or even know about. How this issue is typically solved in clinical medical trials is to use randomised control studies where participants are randomly allocated to different groups (conditions).
While this is not an option with research like this, there is a process which allows estimate what a random control study would look like called the counterfactual model. The results of this modelling are shown here on the red line. As can be seen here the general relationship still holds, giving us greater confidence in causal association between use (engagement) and grades.
Learnline activity in deciles

9. Collaborate use = success
This chart shows the relationship between academic outcome and extent of use of the collaborate tool based on data from nearly 5,000 students studying in 35 unit instances included in the study that utilised collaborate Use of collaborate has been assessed by the percentage of maximum live and recorded collaborate sessions a student accessed in a unit which accounts for variations between units in the number of collaborate sessions available. Again there is a clear trend that among students using collaborate higher passing grades scores are associated with increased use of the collaborate tools.

10. Covariate confirmation: Increased levels of Collaborate activity have a positive effect on Grade Awarded in the Unit*
The blue line on this graph shows linear relationship (regression line) between the use of collaborate tools and GPA, and is adjusted to take in account the background factors we have been able to measure.
So on average, those students who engaged at the lowest level with collaborate achieved a GPA of just over 5. In contrast, those who engaged at the highest level received an average grade 6.3, which is more than a grade category higher. The band either side of the main line is the confidence interval, which is an estimate of where average scores would fall if the same study was ran again under exactly the same conditions - 95% sure the values would fall between these lines.
Adjusted for Part-time Status, NESB, Indigeneity, BoA Higher Ed. Crse,BOA Sec. Education

11. The Gains of Collaborate Activity holds against the most rigorous test
Mean Numeric Grade Score
This slide illustrates a comparison between the application of the previous illustrated multivariate regression approach correcting for confounding factors (upper blue line) and the application of the counterfactual model to the relationship between collaborate usage and grade outcome (the lower red lines) The slopes of both lines illustrate a significant positive relationship and although application of the counterfactoral model shows a less strong relationship than the multivariate regression the general relationship still holds, giving us confidence that increasing Collaborate use has a causal impact on increasing grade success. (May need to explain Collab it is helpful but not the silver bullet))
Percentage of Collaborate Activity

12. Is collaborate an optional extra or integral
Is collaborate an optional extra or integral? Collaborate Effect by Unit Type and Mode of Attendance
External & Core
External & Common
Internal & Common
Internal & Core
This graph simply beaks down data from the previous slide to explore whether the relationship between collaborate use and grade holds across different cohorts and modes.
As can be seen here, the relationship between collaborate use does hold in most cases with the exception of Internal and common units. External showed most benefit but internal core units also benefited . But CU internal –strong pedagogy in active learning spaces! .

13. Perceived Effectiveness
Does the Motivation to Study Account for the Relationship between Perceived Effectiveness of LL and Use
Motivation X X
Perceived Effectiveness
Usage
The results of sophisticated statistical analysis undertaken indicate that there is a statistically significant and likely positive causal relationship between level of use of learnline and the collaborate tool and academic success of students. It is therefore important to try and understand what factors might influence student’s use of online learning tools and resources.
A survey conducted In Semester 2, 2014 to assess CUC students perceptions of the effectiveness of various Learnline tools, indicated that, consistent with theory and previous research there was a strong relationship between perceived effectiveness (usefulness) of Learnline tools and the likelihood of use
An obvious question is are simply the more motivated students engaging with the online learning tools and hold more positive views about the effectiveness of these tools? Assessments of three different types of academic motivation:
intrinsic goal orientation – where participation in a task is an end in itself for reasons such as challenge, curiosity and mastery
extrinsic goal orientation where participation in a task is means to an end such as grades, others approval)
task value which refers to a student’s evaluation of how useful, interesting and important the task is
showed that none of these three types of academic motivation were helpful in explaining either perceived usefulness or usage of learnline technologies.

14. Perceptions that influence usage
Psychological Resources
Ease of Use
Sense of Social Presence
Usage
(intention)
Usefulness
In this study, which was carried out by an Honours student and was based on a survey of CDU students in semesters 1 ad 2 of 2015, focused on ease of use and perceptions of usefulness (the two ovals in the middle of diagram) as the two primary predictors of Use.
The intention of students to use Learnline was measured through the survey instrument using a scale with items such as ‘I will frequently use Learnline in the future’.
Perceived Ease of Use was measured using statements related to how free from effort the participant believed CDU’s online learning system, Learnline, to be. and Perceived Usefulness was measured by how useful participants perceived Learnline to be, relevant to their course,
As illustrated by the arrows in the diagram, Ease of use and Usefulness both related to intention to use learning technologies. The study proceeded to investigate three factors possibly impacting on perceptions of ease of use and usefulness of learnline:
User Resources a construct that covers the psychological resources a student felt they possessed in relation to their use of Learnline.
Social Presence measuring the individual’s sense of social presence in the Learnline environment with a scale designed to capture feelings of cohesion, belonging, trust and social community.
Classroom Community measuring students’ sense of community in the Learnline environment.
As illustrated in the diagram user resources and social presence influenced perceptions of ease of use while social presence and sense of community impacted on student perceptions of the usefulness of Learnline.
These results provide some insights into the areas that may be fostered to increase students potential to engage in online learning – a unit designed to build social presence, sense of community, and tools that are not too onerous to use all contribute to the likelihood of more favourable views, and, in turn, more use of learning technologies.
Sense of Classroom Community

15. Student commentary on Learnline
Insights into what might be promoting or inhibiting student engagement with online learning systems and tools can come from directly surveying students about their experience of using learnline and its tools and the next slide reports the outcomes of a survey carried out over the study period of students’ qualitative commentary on learnline and its tools, more specifically free form answers to two questions:
What are the best aspects (BA) of learnline and its tools
What aspects of learnline and its tools are most in need of improvement (NI)
Comments collected were assigned to a number of categories and sub-categories based on those used in the national Course Experience Questionnaire tool
A sum of the number of BA and NI comments gives an indication of how important the area is to students, while the ratio of BA/NI gives an indication of students’ satisfaction with an area of their experience.
While the overall BA/NI ratio of 1.54 is positive there are quite marked differences between different categories of experience. Looking at areas where there have been a reasonable number of total responses a BA/NI ratio of 2.73 is recorded for learning resources indicating that more than 70% of the students responding saw learning resources as a best aspect of their experience while less than 30% believed it was an area needing improvement.
At the other end of the spectrum the BA/NI ratio for the useability of learnline at 0.80 shows that more than half of the students commenting on this area identified it as needing improvement.
In between there are a number of other areas (identified with orange shading) which attract relatively high numbers of responses but still have more than 40% of the respondents identifying as in need of a improvement (i.e. a BA/NI ratio of less than 1.5) including aspects of staff engagement, assessment, interactive tools and online learning tools .

16. Possible improvement areas
Useability Not perceived as user friendly, needs a more attractive interface that is easier to navigate, issues with different sites looking and being set up differently, and problems with links not working
Discussion boards and interactive tools Often difficult to use and to easily keep track of discussion threads. Wikis not viewed very favourably. Need for more consistent use across units of interactive tools facilitating student:student and student: teacher interaction
Recorded Lectures and online tools Concern that units did not consistently provide recorded classes or if provided often not loaded in a timely fashion. Concern with the quality of recorded sessions students often finding it hard to hear information. Students would like more consistent access to the collaborate tool across all their units with fewer technical glitches that impact on access.
Assessment Concerns with the process of assignment submission including clarity on due dates and better acknowledgment of safe receipt of submitted assessment items. Students would like more guidance with requirements of assessment tasks
Directions in regard of areas of improvements that might be made which would increase student satisfaction and this slide illustrates common improvement suggestions from students that would enhance their experience of learnline and its tools in the areas of:
Useability
Discussion boards and interactive tools
Recorded lectures and other online tools
Assessment
These suggestion for improvement may represent low-hanging fruit that if addressed would see enhanced satisfaction with the effectiveness of learnline and its learning tools and hence increased engagement.

17. Cohort engagement and satisfaction: perceived use correlates with perceived effectiveness
LL engagement of student cohort pairwise comparison
Cohorts with significantly lower BA/NI ratios & areas for attention
Mode
external>internal*
internal: useability of Learnline
Gender
female >male *
male: recorded lectures and online learning tools
Age
mature age > under25 *
under 25: recorded lectures and online learning tools
Home Language
English > not English *
not English: Discussion boards and interactive tools
ATSIs
Non ATSI > ATSI *
ATSI: LR support (but strong on staff)
*statistically significant

18. Learnline engagement and satisfaction by unit type
In addition to investigating differential engagement and satisfaction among different student cohorts by student characteristics and study mode some studies were undertaken to test for any variation between student engagement and satisfaction in relation to the type of unit they were enrolled in. Learnline engagement of students in common units was statistically higher than in core first year units included in the study and this slide shows a comparison of BA/NI ratios across a range of areas for CUC and other core first year units.
As can be observed in term of total comments common units have a statistically higher BA/NI ratio than the non CUC units as well as having statistically higher BA/NI ratios for the comment categories of course design expectations and recorded lectures and online tools. The BA/NI ratio for none of the comment categories was statistically higher for core first year units than common units.
Before the CU Committee starts popping the champagne corks on a job well done it is worth pointing out that common unit BA/NI ratios for the domains of discussion boards and interactive tools, useability of learnline and staff indicate that over half of the comments received were in the category of needs improvement perhaps signalling areas for attention.
An audit for best practice online pedagogy of units included in the study based on Blackboard exemplars was carried out last year and classified instances of common units and some other first year core units as accomplished or exemplary compared with a number of core units as promising, the minimum classification applied across the unit sample. Students in units classified as accomplished and exemplary had a significantly higher level of learnline use than those in units classified as promising and suggest a wider review of unit learnline pedagogy and alignment with best practice exemplars might be a fruitful activity in stimulating learnline engagement

19. Policy and practice recommendations
Promulgate causal relationship between increasing engagement of students Learnline and Learnline tools with academic success to the student body and academic staff
Promote and instigate the broader use of collaborate across the university
Act on the priorities for improvements for Learnline useability, recorded lectures and online tools, discussion boards and interactive tools and assessment identified from qualitative student commentary obtained during this study
Sense of social presence and classroom community could be enhanced to improve perceptions of Learnline’s usefulness and useability
Encourage a review of all online units against best practice online pedagogy and make amendments to online instances as required

20. Student Engagement Universe
Towards A Comprehensive Framework
Pro Vice-Chancellor Academic
v3b, 19 April 2016

21. What do students want? Curriculum Design and Learner Engagement
Formal Learning that is:
personalised (recognises prior learning)
accessible (offered via flexible & dynamic modalities)
connected (to peer, professional & industry networks)
targeted (tailored learning packages)
supported (caring staff, connected peers, services)
And that is:
relevant (to industry, further study and context)
recognised (duly credentialed and transferrable)
value for money and effort
Quality Assurance

22. Main Site 2nd Site CHANGES IN STUDENT-TEACHER ENGAGEMENT METHODS Room
Learner
Teacher
Room
Video conference
Bb Collaborate
Mobile devices

23. LAYER 1: STUDENT ENGAGEMENT NEEDS
Access and move proprietary resources
Access and move OER resources
Create (and co-create) and move resources
CONTENT
Interact with student peers
Submit and access enrolment information
Interact with CDU staff
ESTABLISH AND DEVELOP PERSONAL PRESENCE
Access course and unit information
INTERACTION
MEMBERSHIP
Interact with non-CDU networks
Access broad information and support services
“What will be the Learning Environment” includes Student Support in the report.
ASSESSMENT
Apply for advanced standing
Check work for text matching
Submit assignments
Undertake proctored exams
Access grade information

24. LAYER 2: STUDENT-FACING SYSTEMS
Campuses
Specialist Teaching Facilities
WIL
Virtual
Classroom
Library & Bookshop
LMS (+Mobile)
“What will be the Learning Environment” includes Student Support in the report.
CDU Website
Academic Integrity (eProctoring, Text Matching etc.)
Virtual Reality
Social Learning
Media
ePortfolio

25. LAYER 3: BACKGROUND ARCHITECTURE
Integrated Support
Student
Portal
Universal Authentication
Common Technical Standards (Caliper?)
Identity, Roles & Permissions
Common User Device Standards (BYOD?)
Universal Analytics
“What will be the Learning Environment” includes Student Support in the report.
Asset storage and servers
Data Warehouse, Querying & Reporting