Using CMS Data as a Force for Good? Applying Academic Analytics to Teaching and Learning Leah P. Macfadyen Science Centre for Learning and Teaching, UBC, Canada Shane Dawson Queensland University of Technology/University of Wollongong, Australia

Project Foundations 1.Emergence of “Academic Analytics” 2.Increased use of ICTs in teaching and learning 3.Increasing availability and detail of Course Management System data 4.Increasinging interest in socio-constructivist learning theory (and its implications)

Teaching and Learning for Engagement Socio-constructivist theories of learning Importance of engagement and learning communities Increasing use of ICTs Questions i.Which web-based tools and activities can promote student engagement and community online? ii.How do engagement and sense of community correlate with student achievement?

CMS data CMS usage is now prevalent (US data 2006: 93% student adoption in average of 2.5 courses; UBC data: >25,000 student users of Bb Vista) CMS data is immediate (can be mined at any time) CMS data is non-intrusive (does not require faculty intervention) (Bart Collins, Purdue University, 2006)

Bb Vista Tracking Data

Data points available to instructors Date of first access# Chat sessions Date of last access# Assessments begun Total # sessions# Assessments finished Total time onlineTime on Assessments # Mail messages read# Assignments read # Mail messages sent# Assignments submitted # Artifacts createdTime on Assignments # Artifacts saved# Goals viewed # Discussion messages read# Weblinks viewed # Discussion messages posted# Content folders viewed # Viewed Calendar entries# Files viewed # Added Calendar entries# Media library entries viewed # Media library collections viewed

Project goals Develop a data interpretation and visualization tool to: aid faculty and students in the interpretation of the vast array of data currently captured by Bb Vista permit ongoing formative evaluation of student engagement in learning activities and allow early identification of at risk students provide administrators and institutions with benchmarks of activity, usage trends, disciplinary differences

Case Study: BIOL200

Descriptive Data BIOL 200 online, 2008 BIOL200, 2007 web-supported N (completers) Average final grade60%65% Average online sessions/ student Average hours online/stud ent Average discussion messages read/student Average messages posted722 Average # files viewed826423

Summary of correlations (bivariate) Variable BIOL200 online BIOL200 web-supported RR2R2 RR2R2 # Disc. messages posted # Files viewed # Online sessions Total time online # Assignments submitted n/a # web links viewed ns # Mail messages sent ns # Disc. messages read # Assessments finished

Bivariate Correlations Categories of variables: 1.Measures of effort time online, number of sessions online, time on specific activities 2.Engagement and community activities discussion forums, chat 3.Administrative activities mail, calendar, announcements, tracking, grades 4.Content-related activities files, folders, media 5.Assessment activities assignments, assessments

Predictive modelling BIOL200 online multiple regression (with variables for tools used) BIOL200 web-supported multiple regression (with variables for tools used): (Compare to: Morris, Finnegan & Wu (2005): R 2 =.310 for online courses) RR2R2 F changeSig. F change RR2R2 F changeSig. F change

Discussion participation….

Visualizing student engagement Instructor

Instructor Disconnected students

Institutional tool use 27 Aug Jan 2008 Percentage of total interactions

Lessons learned so far… Some (but not all) CMS data variables are useful predictors of eventual student achievement Several seem to support theoretical propositions regarding the importance of community in learning Correlation ≠ causality… Significance of variables depends on course design